MultiObject Detection and Tracking from a Moving Platform

1
Multi-Object Detection and Tracking from a Moving
Platform
2
Tracking from a Moving Platform

• 1-Analysis and detection
• Registration across video group of frames (VGoF)
• Detection and segmentation of motion blobs
  (background models, shadow)
• 2-Representation and tracking
• Video object representation (shape, color
  descriptors, geometric models)
• Object tracking (prediction, correspondence,
  occlusion resolution etc.)
• 3-Access and event modeling
• Efficient data structures for video queries in
  high-dimensional feature space
• High-level event representation

3
Multi-Object Tracking

• 1. Detect moving objects in stabilized frames.
• 2. Predict locations of the current set of
  objects.
• 3. Match predictions to actual measurements.
• 4. Update object trajectories.
• 5. Update image stabilized ref coord system.

Multi-object Detection and Tracking Unit
Tracking
VGoF Registration Into Common Coordinate System
Moving Object Detection Feature Extraction
Data Association (Correspondence)
Update Trajectories
Object States
Update Coord System
Prediction
Context
4
Dynamic State Estimation for Tracking
System state
State estimate
Measurements
Dynamic System
Measurement System
State Estimator
State uncertainties
System noise
Measurement noise

• System Errors
• Agile motion
• Distraction/clutter
• Occlusion
• Changes in lighting
• Changes in pose
• Shadow
• (Object or background models
• are often inadequate or inaccurate)

• Measurement Errors
• Camera noise
• Framegrabber noise
• Compression artifacts
• Perspective projection

• State Error
• Position
• Appearance
• Color
• Shape
• Texture etc.
• Support map

5
Motion Detection- 3D Spatiotemporal Volume
Spatio-temporal volume of hall monitor sequence
(a) Left entire volume, (b) Middle cut taken at
vertical position y0, (c) Right Cut taken at
vertical Position y1.
Gerald Kuhne, Motion-based segmentation and
classification of Video Objects Dissertation
Univ. of Mannheim, 2002
6
Motion Detection - Structure and Flux Tensor
Approach

• Typical Approach threshold trace(J)
• Problem trace(J) fails to capture the nature of
  gradient changes and results in ambiguities
  between stationary versus moving features
• Alternative Approach Analyze the eigenvalues
  and the associated eigenvectors of J
• Problem Eigen-decompositions at every pixel is
  computationally expensive for real time
  performance
• Proposed Solution Flux tensor ?? time derivative
  of J

7
Motion Detection Flux Tensor vs Gaussian Mixture
8
Multi-object Tracking Stages

• Probabilistic Bayesian framework
• Features Used in Data Association Proximity and
  Appearance-based
• Data Association Strategy Multi-hypothesis
  testing
• Gating Strategies Absolute and Relative
• Discontinuity Resolution Prediction (Kalman
  filter), or Appearance models
• Filtering Temporal consistency check and
  Spatio-temporal cluster check

9
Association Strategy

• Multi-hypothesis testing with delayed decision -
  Many matches are kept with evidence-based pruning
• Support for multiple interactions - one-to-one
  object matches, many-to-one, one-to-many,
  many-to-many, one-to-none, or none-to-one matches
  
• Corresponding low-level object tracking events
• Segmentation errors
• Group interactions (merge/split)
• Occlusion
• Fragmentation
• Entering object
• Exiting object

ObjectMatchGraph
10
Match Confidence Computation

• Match confidence quantifies correspondence
  goodness-of-fit
• Confidence value has two components
• Similarity confidence (Confsim)
• Separation confidence(Confsep)

?1,j is the closest competitor in terms of
distance
Conf(i,j)
Link
Nodei
Nodej

• bounding box
• support map
• centroid
• area etc.

• bounding box
• support map
• centroid
• area etc.

11
Trajectory Segment Generation

• Trace links in the ObjectMatchGraph to generate
  possible trajectory segments
• SegmentList - Linked list of inner nodes
  (objects/cells)
• Trajectory labeling - Extracted trajectory
  segments are labeled using a modified connected
  components labeling
• Trajectory linking - Trajectories are formed by
  linking unfiltered segments sharing the same
  label.

Trajectory
ObjectMatchGraph
12
Data Hierarchy
Node (Object-Region)
Segment
Macro segment
Trajectory
13
Need for Local Registration
14
Exp Results DARPA ET01 Video Frame 50
Registered Frame
Motion Detection Results
Foreground Mask
Tracking Results
15
Exp Results - NGA Crystal View HD Video Frame
787 in Coord. 740
UPS
c) Predictions
d) After occlusion handling
16
Future Work - Trajectory Matching and Filtering

• Establishing trajectory continuity (object ID
  matching) across moving coordinate systems
• Customizing trajectory analysis for airborne
  video tracking with misregistration error, large
  platform motion, zooming, etc
• Maintaining temporal consistency of trajectories
• Removing periodic clustered trajectories
• Resolving discontinuous trajectories
• Trajectory display and visualization video vs
  mosaic

17
Future Work Performance Optimization and Tuning

• Moving object detector filters
• Flux tensor fixed optimal threshold learning or
  continuous adaptive thresholding
• Morphological post processing filters
• Real-time versus offline MATLAB (approximate)
• Flux tensor detection 4sec/frame
• Object tracking 2sec/frame (for around 10
  objects)
• Excluding I/O time

18
Future Work - Near Term Performance Improvements

• Frame-to-frame registration accuracy difficult to
  maintain across a hundred frames or more (few
  seconds of video)
• Reducing false motion trajectories due to
  registration errors due to scene structure
• Maintaining a common coordinate system for
  registering long airborne video sequence
• Tracking through large platform motion
• Dealing with large camera field-of-view changes
• Platform motion jitter

19
Future Work - Longer Term Performance Improvements

• Filtering periodic motions produced by clutter,
  etc.
• Shadows (e.g. false detections, shape
  distortions, merges)
• Sudden illumination changes (e.g. due to cloud
  movements)
• Glare from specular surfaces (e.g. windshields,
  water surfaces)
• Perspective distortion (e.g. object size, shape
  and position)
• Trajectory gaps and distortion due to occlusion
• Poor video quality (e.g. low resolution, low
  color saturation)