Motion Segmentation at Any Speed

1
Motion Segmentation at Any Speed

• Shrinivas J. Pundlik
• Department of Electrical and Computer
  Engineering,
• Clemson University, Clemson, SC

2
Gestalt Theory and Visual Perception

• The human visual system
• Focus on well organized patterns rather than
  disparate parts
• Grouping - the idea behind visual perception
• Factors affecting the grouping process
  proximity, similarity, closure, smoothness,
  symmetry, common fate and so on.

3
Gestalt Laws of Grouping
Proximity
Continuity
Common Fate

• Basis of many image and video segmentation
  algorithms
• Work well in combinations. For example, proximity
  and similarity
• Motion segmentation grouping or aggregating
  entities with common fate

4
Applications of Motion Segmentation

• Object detection and tracking
• Surveillance
• Robot Motion
• Image and Video Compression
• Video Editing/Motion Magnification
• Shape Recovery

5
Existing Approaches
Extraction of Motion Layers Wang and Adelson
1994, Weiss 1996 Ayer and Sawhney 1995,Xiao and
Shah 2005 Ke and Kanade 2002
Detecting Motion Discontinuities Black and Fleet
1999, Birchfield 1998
Feature Point Grouping Beymer 1997, Fua
2003 Kanhere et. al. 2005
Normalized Cuts Shi and Malik 1998
6
Preview
7
Incremental Motion Segmentation

• Existing approaches
• Consider 2 frames or a spatio-temporal volume
• Threshold on velocities
• t gt (?x/ ?t)

• Proposed approach
• An incremental approach
• Threshold on position
• Waits till enough
• evidence accumulates before segmenting

8
Different Approaches To Segmentation
Segmentation/Grouping
Agglomerative
Divisive
Start with single point and grow the
group. Region growing or Region merging
Start with entire data and split into
clusters. Clustering or partitioning
last step
first step
last step
first step
seed 2
seed 1
9
Representation of Motion

• Why use feature points instead of optic flow?
• Reduced time and complexity of computation
• Reliable and repeatable
• Well suited for tracking over long sequences

10
Feature Tracking
Idea behind feature tracking minimize the
dissimilarity between two feature windows in the
successive frames

Good Features Small image regions having high
intensity variation in more than one direction
Affine Consistency Check
11
Feature Clustering

• Clustering Data Feature displacement over
  multiple frames
• K-means clustering by fitting lines
• Works better than clustering points

12
Results of Feature Clustering by K-means

• Limitation Clustering not accurate for more
  challenging sequences

13
Affine Partitioning

• Requires prior initialization and number of
  groups to be found
• Processing only on feature motion between two
  frames

14
Normalized Cuts

• Graph G(V,E)
• Partitions A,B
• Weight of an edge w
• Affinity Matrix Feature motion between two frames

15
Region Growing

• Process over two frames
• Select seed point
• Fit affine model to neighbors
• Repeat until the group does not change
• Discard all features except the one near the
  centroid
• Grow group by including neighboring features with
  similar motion till it grows no further
• Update the affine model

16
Finding Neighbors

• Traditional way Spatial window
• Makes the algorithm sensitive to the feature
  locations
• Alternative Delaunay Traingulation
• Simple and efficient technique

17
Finding Consistent groups

• Parameters affecting region growing grouping
  threshold, choice of frames and seed point
• Different choice of seed points produce different
  grouping results
• Features grouped together irrespective of the
  choice of seed points are consistent feature
  groups

18
Consistent Groups
19
Maintaining Groups Over Time

• Finding new feature groups
• Segmenting new objects entering the scene
• Splitting existing feature groups
• Split when configuration of a group changes over
  time
• Adding new features to existing feature groups
• Include new scene information over time

20
Results
Segmentation results for the statue sequence
21
Results
Frames 3, 4, 5, 6 of the statue sequence with
threshold 0.7
Frames 8, 64, 188, 395,6 of the fast statue
sequence (generated by dropping every alternate
frame)
22
Results
Segmentation results for different sequences
23
Conclusions

• Segmentation based on the availability of
  evidence
• An incremental approach able to handle long
  sequences