Mean Shift Theory and Applications

1
Mean ShiftTheory and Applications
Yaron Ukrainitz Bernard Sarel
2
Agenda

• Mean Shift Theory
• What is Mean Shift ?
• Density Estimation Methods
• Deriving the Mean Shift
• Mean shift properties
• Applications
• Clustering
• Discontinuity Preserving Smoothing
• Object Contour Detection
• Segmentation
• Object Tracking

3
Mean Shift Theory
4
Intuitive Description
Region of interest
Center of mass
Mean Shift vector
Objective Find the densest region
Distribution of identical billiard balls
5
Intuitive Description
Region of interest
Center of mass
Mean Shift vector
Objective Find the densest region
Distribution of identical billiard balls
6
Intuitive Description
Region of interest
Center of mass
Mean Shift vector
Objective Find the densest region
Distribution of identical billiard balls
7
Intuitive Description
Region of interest
Center of mass
Mean Shift vector
Objective Find the densest region
Distribution of identical billiard balls
8
Intuitive Description
Region of interest
Center of mass
Mean Shift vector
Objective Find the densest region
Distribution of identical billiard balls
9
Intuitive Description
Region of interest
Center of mass
Mean Shift vector
Objective Find the densest region
Distribution of identical billiard balls
10
Intuitive Description
Region of interest
Center of mass
Objective Find the densest region
Distribution of identical billiard balls
11
What is Mean Shift ?
A tool for Finding modes in a set of data
samples, manifesting an underlying probability
density function (PDF) in RN

• PDF in feature space
• Color space
• Scale space
• Actually any feature space you can conceive

Non-parametric Density Estimation
Discrete PDF Representation
Non-parametric Density GRADIENT Estimation
(Mean Shift)
PDF Analysis
12
Non-Parametric Density Estimation
Assumption The data points are sampled from an
underlying PDF
Data point density implies PDF value !
Assumed Underlying PDF
Real Data Samples
13
Non-Parametric Density Estimation
Assumed Underlying PDF
Real Data Samples
14
Non-Parametric Density Estimation
?
Assumed Underlying PDF
Real Data Samples
15
Parametric Density Estimation
Assumption The data points are sampled from an
underlying PDF
Estimate
Assumed Underlying PDF
Real Data Samples
16
Kernel Density EstimationParzen Windows -
General Framework
A function of some finite number of data
points x1xn

• Kernel Properties
• Normalized
• Symmetric
• Exponential weight decay
• ???

17
Kernel Density Estimation Parzen Windows -
Function Forms
A function of some finite number of data
points x1xn
In practice one uses the forms
or
Same function on each dimension
Function of vector length only
18
Kernel Density EstimationVarious Kernels
A function of some finite number of data
points x1xn

• Examples
• Epanechnikov Kernel
• Uniform Kernel
• Normal Kernel

19
Kernel Density Estimation
Gradient
Give up estimating the PDF ! Estimate ONLY the
gradient
Using the Kernel form
We get
Size of window
20
Kernel Density Estimation
Computing The Mean Shift
Gradient
21
Computing The Mean Shift
Yet another Kernel density estimation !

• Simple Mean Shift procedure
• Compute mean shift vector
• Translate the Kernel window by m(x)

22
Mean Shift Mode Detection
What happens if we reach a saddle point ?
Perturb the mode position and check if we return
back

• Updated Mean Shift Procedure
• Find all modes using the Simple Mean Shift
  Procedure
• Prune modes by perturbing them (find saddle
  points and plateaus)
• Prune nearby take highest mode in the window

23
Mean Shift Properties

• Automatic convergence speed the mean shift
  vector size depends on the gradient itself.
• Near maxima, the steps are small and refined
• Convergence is guaranteed for infinitesimal
  steps only ? infinitely convergent, (therefore
  set a lower bound)
• For Uniform Kernel ( ), convergence is
  achieved in a finite number of steps
• Normal Kernel ( ) exhibits a smooth
  trajectory, but is slower than Uniform Kernel
  ( ).

Adaptive Gradient Ascent
24
Real Modality Analysis
Tessellate the space with windows
Run the procedure in parallel
25
Real Modality Analysis
The blue data points were traversed by the
windows towards the mode
26
Real Modality AnalysisAn example
Window tracks signify the steepest ascent
directions
27
Adaptive Mean Shift
28
Mean Shift Strengths Weaknesses

• Strengths
• Application independent tool
• Suitable for real data analysis
• Does not assume any prior shape (e.g.
  elliptical) on data clusters
• Can handle arbitrary feature spaces
• Only ONE parameter to choose
• h (window size) has a physical meaning,
  unlike K-Means

• Weaknesses
• The window size (bandwidth selection) is not
  trivial
• Inappropriate window size can cause modes to
  be merged, or generate additional shallow
  modes ? Use adaptive window size

29
Mean Shift Applications
30
Clustering
Cluster All data points in the attraction basin
of a mode
Attraction basin the region for which all
trajectories lead to the same mode
Mean Shift A robust Approach Toward Feature
Space Analysis, by Comaniciu, Meer
31
ClusteringSynthetic Examples
Simple Modal Structures
Complex Modal Structures
32
ClusteringReal Example
Feature space Luv representation
Initial window centers
Modes found
Modes after pruning
Final clusters
33
ClusteringReal Example
Luv space representation
34
ClusteringReal Example
2D (Lu) space representation
Final clusters
Not all trajectories in the attraction
basin reach the same mode
35
Discontinuity Preserving Smoothing
Feature space Joint domain spatial
coordinates color space
Meaning treat the image as data points in the
spatial and gray level domain
Image Data (slice)
Mean Shift vectors
Smoothing result
Mean Shift A robust Approach Toward Feature
Space Analysis, by Comaniciu, Meer
36
Discontinuity Preserving Smoothing
The image gray levels
can be viewed as data points in the x, y, z
space (joined spatial And color space)
37
Discontinuity Preserving Smoothing
Flat regions induce the modes !
38
Discontinuity Preserving Smoothing
The effect of window size in spatial and range
spaces
39
Discontinuity Preserving SmoothingExample
40
Discontinuity Preserving SmoothingExample
41
Object Contour DetectionRay Propagation
Accurately segment various objects (rounded in
nature) in medical images
Vessel Detection by Mean Shift Based Ray
Propagation, by Tek, Comaniciu, Williams
42
Object Contour DetectionRay Propagation
Use displacement data to guide ray propagation
Discontinuity preserving smoothing
Displacement vectors
Vessel Detection by Mean Shift Based Ray
Propagation, by Tek, Comaniciu, Williams
43
Object Contour DetectionRay Propagation
Speed function
Normal to the contour
Curvature
44
Object Contour Detection
Gray levels along red line
Gray levels after smoothing
Original image
Displacement vectors
Displacement vectors derivative
45
Object Contour DetectionExample
46
Object Contour DetectionExample
Importance of smoothing by curvature
47
Segmentation
Segment Cluster, or Cluster of Clusters

• Algorithm
• Run Filtering (discontinuity preserving
  smoothing)
• Cluster the clusters which are closer than
  window size

Image Data (slice)
Mean Shift vectors
Segmentation result
Smoothing result
Mean Shift A robust Approach Toward Feature
Space Analysis, by Comaniciu, Meer http//www.caip
.rutgers.edu/comanici
48
SegmentationExample
when feature space is only gray levels
49
SegmentationExample
50
SegmentationExample
51
SegmentationExample
52
SegmentationExample
53
SegmentationExample
54
SegmentationExample
55
Non-Rigid Object Tracking


56
Non-Rigid Object Tracking
Real-Time
Object-Based Video Compression
Surveillance
Driver Assistance
57
Mean-Shift Object TrackingGeneral Framework
Target Representation
58
Mean-Shift Object TrackingGeneral Framework
Target Localization
Start from the position of the model in the
current frame
Repeat the same process in the next pair of frames
59
Mean-Shift Object TrackingTarget Representation
Kernel Based Object Tracking, by Comaniniu,
Ramesh, Meer
60
Mean-Shift Object TrackingPDF Representation
Target Model (centered at 0)
Target Candidate (centered at y)
61
Mean-Shift Object TrackingSmoothness of
Similarity Function
62
Mean-Shift Object TrackingFinding the PDF of the
target model
Target pixel locations
63
Mean-Shift Object TrackingSimilarity Function
Target model
Target candidate
Similarity function
64
Mean-Shift Object TrackingTarget Localization
Algorithm
Start from the position of the model in the
current frame
65
Mean-Shift Object TrackingApproximating the
Similarity Function
Model location
Candidate location
Independent of y
Density estimate! (as a function of y)
66
Mean-Shift Object TrackingMaximizing the
Similarity Function
The mode of
sought maximum
67
Mean-Shift Object TrackingApplying Mean-Shift
The mode of
sought maximum
Original Mean-Shift
Find mode of
using
68
Mean-Shift Object TrackingAbout Kernels and
Profiles
69
Mean-Shift Object TrackingChoosing the Kernel
A special class of radially symmetric kernels
Epanechnikov kernel
Extended Mean-Shift
70
Mean-Shift Object TrackingAdaptive Scale
Problem
The scale of the target changes in time
The scale (h) of the kernel must be adapted
71
Mean-Shift Object TrackingResults
Feature space 16?16?16 quantized RGB Target
manually selected on 1st frame Average mean-shift
iterations 4
72
Mean-Shift Object TrackingResults
73
Mean-Shift Object TrackingResults
74
Mean-Shift Object TrackingResults
Feature space 128?128 quantized RG
75
Mean-Shift Object TrackingThe Scale Selection
Problem
Kernel too big
Kernel too small
76
Tracking Through Scale SpaceMotivation
Spatial localization for several scales
Simultaneous localization in space and scale
Previous method
This method
Mean-shift Blob Tracking through Scale Space, by
R. Collins
77
Lindebergs TheorySelecting the best scale for
describing image features
Scale-space representation
Differential operator applied
50 strongest responses
78
Lindebergs TheoryThe Laplacian operator for
selecting blob-like features
Best features are at (x,s) that maximize L
79
Lindebergs TheoryMulti-Scale Feature Selection
Process
Original Image
80
Tracking Through Scale SpaceApproximating LOG
using DOG
2D LOG filter with scale s
2D DOG filter with scale s

• Why DOG?
• Gaussian pyramids are created faster
• Gaussian can be used as a mean-shift kernel

81
Tracking Through Scale SpaceUsing Lindebergs
Theory
The likelihood that each candidate pixel belongs
to the target
82
Tracking Through Scale SpaceExample
Image of 3 blobs
A slice through the 3D scale-space representation
83
Tracking Through Scale SpaceApplying Mean-Shift
Use interleaved spatial/scale mean-shift
Spatial stage
Scale stage
Fix s and look for the best x
Fix x and look for the best s
84
Tracking Through Scale SpaceResults
Fixed-scale
85
Thank You