Analysis of Contour Motions

1
Analysis of Contour Motions
Neural Information Processing Systems 2006

• Ce Liu William T. Freeman Edward H. Adelson
• Computer Science and Artificial Intelligence
  Laboratory
• Massachusetts Institute of Technology

2
Visual Motion Analysis in Computer Vision

• Motion analysis is essential in
• Video processing
• Geometry reconstruction
• Object tracking, segmentation and recognition
• Graphics applications
• Is motion analysis solved?
• Do we have good representation for motion
  analysis?
• Is it computationally feasible to infer the
  representation from the raw video data?
• What is a good representation for motion?

3
Seemingly Simple Examples
Kanizsa square
From real video
4
Output from the State-of-the-Art Optical Flow
Algorithm
Kanizsa square
Optical flow field
T. Brox et al. High accuracy optical flow
estimation based on a theory for warping. ECCV
2004
5
Output from the State-of-the-Art Optical Flow
Algorithm
Dancer
Optical flow field
T. Brox et al. High accuracy optical flow
estimation based on a theory for warping. ECCV
2004
6
Optical flow representation aperture problem
Corners
Lines
Flat regions
Spurious junctions
Boundary ownership
Illusory boundaries
7
Optical Flow Representation
Corners
Lines
Flat regions
Spurious junctions
Boundary ownership
Illusory boundaries
8
Layer Representation

• Video is a composite of layers
• Layer segmentation assumes sufficient textures
  for each layer to represent motion
• A true success?

9
Layer Representation

• Video is a composite of layers
• Layer segmentation assumes sufficient textures
  for each layer to represent motion
• A true success?

10
Challenge Textureless Objects under Occlusion

• Corners are not always trustworthy (junctions)
• Flat regions do not always move smoothly
  (discontinuous at illusory boundaries)
• How about boundaries?
• Easy to detect and track for textureless objects
• Able to handle junctions with illusory boundaries

11
Analysis of Contour Motions

• Our approach simultaneous grouping and motion
  analysis
• Multi-level contour representation
• Junctions are appropriated handled
• Formulate graphical model that favors good
  contour and motion criteria
• Inference using importance sampling
• Contribution
• An important component in motion analysis toolbox
  for textureless objects under occlusion

12
Three Levels of Contour Representation

• Edgelets edge particles
• Boundary fragments a chain of edgelets with
  small curvatures
• Contours a chain of boundary fragments

Forming boundary fragments easy (for textureless
objects) Forming contours hard (the focus of
our work)
13
Overview of our system
1. Extract boundary fragments
2. Edgelet tracking with uncertainty.
3. Boundary grouping and illusory boundary
4. Motion estimation based on the grouping
14
Forming Boundary Fragments

• Boundary fragments extraction in frame 1
• Steerable filters to obtain edge energy for each
  orientation band
• Spatially trace boundary fragments
• Boundary fragments lines or curves with small
  curvature
• Temporal edgelet tracking with uncertainties

• Frame 1 edgelet (x, y, q)
• Frame 2 orientation energy of q
• A Gaussian pdf is fit with the weight of
  orientation energy
• 1D uncertainty of motion (even for T-junctions)

15
Forming Contours Boundary Fragments Grouping

• Grouping representation switch variables
  (attached to every end of the fragments)
• Exclusive one end connects to at most one other
  end
• Reversible if end (i,ti) connects to (j,tj),
  then (j,tj) connects to (i,ti)

1
Arbitrarily possible connection
A legal contour grouping
0
Reversibility
Another legal contour grouping
1
1
0
0
16
Local Spatial-Temporal Cues for Grouping
Illusory boundaries corresponding to the
groupings (generated by spline interpolation)
Motion stimulus
17
Local spatial-temporal cues for grouping (a)
Motion similarity
The grouping with higher motion similarity is
favored
KL( ) lt KL( )
Motion stimulus
18
Local spatial-temporal cues for grouping (b)
Curve smoothness
The grouping with smoother and shorter illusory
boundary is favored
Motion stimulus
19
Local spatial-temporal cues for grouping (c)
Contrast consistency
The grouping with consistent local contrast is
favored
Motion stimulus
20
The Graphical Model for Grouping

• Affinity metric terms
• (a) Motion similarity
• (b) Curve smoothness
• (c) Contrast consistency
• The graphical model for grouping

reversibility
affinity
no self-intersection
21
Motion estimation for grouped contours

• Gaussian MRF (GMRF) within a boundary fragment
• The motions of two end edgelets are similar if
  they are grouped together
• The graphical model of motion joint Gaussian
  given the grouping

This problem is solved in early work Y. Weiss,
Interpreting images by propagating Bayesian
beliefs, NIPS, 1997.
22
Inference

• Two-step inference
• Grouping (switch variables)
• Motion based on grouping (easy, least square)
• Grouping importance sampling to estimate the
  marginal of the switch variables
• Bidirectional proposal density
• Toss the sample if self-intersection is detected
• Obtain the optimal grouping from the marginal

23
Why bidirectional proposal in sampling?
24
Why bidirectional proposal in sampling?
Affinity metric of the switch variable (darker,
thicker means larger affinity)
b1?b2 0.39 b1?b3 0.01 b1?b4 0.60
b4?b1 0.20 b4?b2 0.05 b4?b3 0.85
b2?b1 0.50 b2?b3 0.45 b2?b4 0.05
b3?b1 0.01 b3?b2 0.45 b3?b4 0.54
b1?b2 0.1750 b1?b3 0.0001 b1?b4 0.1200
Bidirectional proposal
Normalized affinity metrics
25
Why bidirectional proposal in sampling?
Bidirectional proposal of the switch variable
(darker, thicker means larger affinity)
b1?b2 0.39 b1?b3 0.01 b1?b4 0.60
b4?b1 0.20 b4?b2 0.05 b4?b3 0.85
b2?b1 0.50 b2?b3 0.45 b2?b4 0.05
b3?b1 0.01 b3?b2 0.45 b3?b4 0.54
b1?b2 0.62 b1?b3 0.00 b1?b4 0.38
Bidirectional proposal (Normalized)
Normalized affinity metrics
26
Example of Sampling
Self intersection
27
Example of Sampling
A valid grouping
28
Example of Sampling
More valid groupings
29
Example of Sampling
More valid groupings
30
From Affinity to Marginals
Affinity metric of the switch variable (darker,
thicker means larger affinity)
Motion stimulus
31
From Affinity to Marginals
Marginal distribution of the switch variable
(darker, thicker means larger affinity)
Greedy algorithm to search for the best grouping
based on the marginals
Motion stimulus
32
Experiments

• All the results are generated using the same
  parameter settings
• Running time depends on the number of boundary
  fragments, varying from ten seconds to a few
  minutes in MATLAB

33
Two Moving Bars
Frame 1
34
Two Moving Bars
Frame 2
35
Two Moving Bars
Extracted boundary fragments. The green circles
are the boundary fragment end points.
36
Two Moving Bars
Optical flow from Lucas-Kanade algorithm. The
flow vectors are only plotted at the edgelets
37
Two Moving Bars
Estimated motion by our system after grouping
38
Two Moving Bars
Boundary grouping and illusory boundaries (frame
1). The fragments belonging to the same contour
are plotted in one color.
39
Two Moving Bars
Boundary grouping and illusory boundaries (frame
2). The fragments belonging to the same contour
are plotted in one color.
40
Kanizsa Square
41
Frame 1
42
Frame 2
43
Extracted boundary fragments
44
Optical flow from Lucas-Kanade algorithm
45
Estimated motion by our system, after grouping
46
Boundary grouping and illusory boundaries (frame
1)
47
Boundary grouping and illusory boundaries (frame
2)
48
Dancer
49
Frame 1
50
Frame 2
51
Extracted boundary fragments
52
Optical flow from Lucas-Kanade algorithm
53
Estimated motion by our system, after grouping
54
Lucas-Kanade flow field
Estimated motion by our system, after grouping
55
Boundary grouping and illusory boundaries (frame
1)
56
Boundary grouping and illusory boundaries (frame
2)
57
Rotating Chair
58
Frame 1
59
Frame 2
60
Extracted boundary fragments
61
Estimated flow field from Brox et al.
62
Estimated motion by our system, after grouping
63
Boundary grouping and illusory boundaries (frame
1)
64
Boundary grouping and illusory boundaries (frame
2)
65
Conclusion

• A contour-based representation to estimate motion
  for textureless objects under occlusion
• Motion ambiguities are preserved and resolved
  through appropriate contour grouping
• An important component in motion analysis toolbox
• To be combined with the classical motion
  estimation techniques to analyze complex scenes

66
Thanks!

• Analysis of Contour Motions
• Ce Liu William T. Freeman Edward H. Adelson
• Computer Science and Artificial Intelligence
  Laboratory
• Massachusetts Institute of Technology
• http//people.csail.mit.edu/celiu/contourmotions/

67
Backup Slides
68
Why bidirectional proposal in sampling?
69
Why bidirectional proposal in sampling?
Affinity metric of the switch variable (darker,
thicker means larger affinity)
b1?b2 0.39 b1?b3 0.01 b1?b4 0.60
b4?b1 0.20 b4?b2 0.05 b4?b3 0.85
b2?b1 0.50 b2?b3 0.45 b2?b4 0.05
b3?b1 0.01 b3?b2 0.45 b3?b4 0.54
b1?b2 0.1750 b1?b3 0.0001 b1?b4 0.1200
Bidirectional proposal
Normalized affinity metrics
70
Why bidirectional proposal in sampling?
Bidirectional proposal of the switch variable
(darker, thicker means larger affinity)
b1?b2 0.39 b1?b3 0.01 b1?b4 0.60
b4?b1 0.20 b4?b2 0.05 b4?b3 0.85
b2?b1 0.50 b2?b3 0.45 b2?b4 0.05
b3?b1 0.01 b3?b2 0.45 b3?b4 0.54
b1?b2 0.62 b1?b3 0.00 b1?b4 0.38
Bidirectional proposal (Normalized)
Normalized affinity metrics
71
Sampling Grouping (Switch Variables)
72
Lucas-Kanade flow field
Estimated motion by our system, after grouping