SpatioTemporal Segmentation of Video by Hierarchical Mean Shift Analysis

1
Spatio-Temporal Segmentation of Video by
Hierarchical Mean Shift Analysis

• Daniel DeMenthon, Rémi Megret
• Statistical Methods in Video Processing Workshop,
  Copenhagen, Denmark, June 1-2, 2002.
• Presented by Ryan Crabb
• November 1, 2007

2
Overview

• Input video sequence
• Performs color and motion segmentation of
  space-time volume (video stacks)
• Outputs Labeled regions over time, ie region
  tracking

3
Overview

• Pixels are mapped to 7D feature space
• Space is segmented using mean-shift segmentation
• Performed iteratively with increasing search
  window
• Resulting segments represent tracked regions

4
What is Spatio-Temporal Segmentation?
5
Define the Feature Space

• Pixels are categorized by
• Color
• Motion
• Location (wrt estimated motion)
• A subset of these features could be used

6
Feature Space - Color

• Various known color spaces can be applied
• RGB
• CIE Luv
• HSV/HSI

7
Feature Space - Motion

• Estimate motion with optical flow
• Motion vector can be characterized by u,v the
  respective motion in x,y directions
• Motion can also be characterized by motion angle
• Project motion vectors onto (t,x) and (t,y)
  planes, respectively

8
Feature Space Motion Angles

• Define
• Angles range from 0º to 180º
• Fast motion approaches 0º or 180º
• Still patches are 90º
• Faster motion is more easily clustered

9
Feature Space - Position

• For still patches x,y location remains constant
  over time, can be clustered
• Moving patches not easily clustered by x,y data
• Use distance of estimated trajectory from center
  of video stack

10
Feature Space - Position
11
Feature Space

• Define
• Summary
• L, u, v, ax, ay, Dx, Dy

12
Feature Space Mapping
13
Segmentation

• Segmentation done by Mean-Shift

Comaniciu, D. Meer, P., "Mean shift a robust
approach toward feature space analysis," Pattern
Analysis and Machine Intelligence, IEEE
Transactions on , vol.24, no.5, pp.603-619, May
2002
14
Segmentation

• Segmentation done by Mean-Shift
• h is search window radius
• g is derivative of kernel function

Comaniciu, D. Meer, P., "Mean shift a robust
approach toward feature space analysis," Pattern
Analysis and Machine Intelligence, IEEE
Transactions on , vol.24, no.5, pp.603-619, May
2002
15
Mean Shift Segmentation

• Kernel function can be simple
• Exponential function is common
• Epanechnikov has constant derivative
• No need to estimate number of segments
• For each pixel in volume
• Iterate until convergence upon a point
• Label pixel with cluster center
• Pixels with same labels are regions

16
Mean Shift Segmentation
17
Does it work?
18
Segmentation Results

• XY
• Over 12 frames

19
Segmentation Results

• XT
• Over 12 Y-slices

20
Mean Shift Segmentation

• How large a search radius?
• Small window ? more clusters
• Large window ? large search time
• Claimed that 1/5 span of feature space provides
  good results

21
Mean Shift Speed-Up

• Finding points within a range of a certain point
  in high dimension can be expensive
• Use fast and/or approximate search
• Binary tree search like ATRIA
• Fast Approximate Similarity Search in Extremely
  High-Dimensional Data Sets. ME Houle, J Sakuma -
  Data Engineering, 2005. ICDE 2005.

22
Hierarchical Mean Shift

• For most time-saving tree structures, advantage
  is lost for large search radii
• Most tree branches must be searched
• Cost for N points is O(N log N) for small radii
• Cost approaches O(N2) for large radii
• Perform hierarchy of mean-shift clusterings with
  increasing radius

23
Hierarchical Mean Shift

• Perform segmentation with small radius
• Original points have weight of 1
• Each cluster center becomes point with weight
  totaling sum of member points
• Increase window radius 25-50
• Repeat until desired radius is reached
• (or desired number of clusters)

24
Some Comparative Data
25
Some Comparative Data
26
Does Hierarchical M-S work?