Motion estimation

1
Motion estimation

• Digital Visual Effects, Spring 2006
• Yung-Yu Chuang
• 2005/4/12

with slides by Michael Black and P. Anandan
2
Announcement

• The first part of project 2 (feature detection
  and matching) is due on Sunday, please send your
  source code and two images showing your results
  to TAs.

3
Outline

• Motion estimation
• Lucas-Kanade algorithm
• Tracking
• Optical flow

4
Motion estimation

• Parametric motion (image alignment)
• Tracking
• Optical flow

5
Parametric motion
direct method for image stitching
6
Tracking
7
Optical flow
8
Three assumptions

• Brightness consistency
• Spatial coherence
• Temporal persistence

9
Brightness consistency
10
Spatial coherence
11
Temporal persistence
12
Image registration

• Goal register a template image J(x) and an input
  image I(x), where x(x,y)T.
• Image alignment I(x) and J(x) are two images
• Tracking I(x) is the image at time t. J(x) is a
  small patch around the point p in the image at
  t1.
• Optical flow I(x) and J(x) are images of t and
  t1.

13
Simple approach

• Minimize brightness difference

14
Simple SSD algorithm

• For each offset (u, v)
• compute E(u,v)
• Choose (u, v) which minimizes E(u,v)
• Problems
• Not efficient
• No sub-pixel accuracy

15
Lucas-Kanade algorithm
16
Newtons method

• Root finding for f(x)0

17
Newtons method

• Root finding for f(x)0
• Taylors expansion

18
Newtons method

• Root finding for f(x)0

x2
x1
19
Newtons method

• pick up xx0
• iterate
• compute
• update x by x?x
• until converge
• Minimize g(x)
• ?find f(x)g(x)0

20
Lucas-Kanade algorithm
21
Lucas-Kanade algorithm
22
Lucas-Kanade algorithm

• iterate
• shift I(x,y) with (u,v)
• compute gradient image Ix, Iy
• compute error image J(x,y)-I(x,y)
• compute Hessian matrix
• solve the linear system
• (u,v)(u,v)(?u,?v)
• until converge

23
Parametric model
24
Parametric model
minimize
with respect to
25
Parametric model
26
Jacobian of the warp
For example, for affine
27
Parametric model
minimize
Hessian
28
Lucas-Kanade algorithm

• iterate
• warp I with W(xp)
• compute error image J(x,y)-I(W(x,p))
• compute gradient image with W(x,p)
• evaluate Jacobian at (xp)
• compute
• compute Hessian
• compute
• solve
• update p by p
• until converge

29
(No Transcript)
30
Coarse-to-fine strategy
J
I
J
Jw
I
refine
warp

J
I
Jw
pyramid construction
pyramid construction
refine
warp

J
I
Jw
refine
warp

31
Application of image alignment
32
Tracking
33
Tracking
I(x,y,t)
I(x,y,t1)
34
Tracking
brightness constancy
optical flow constraint equation
35
Optical flow constraint equation
36
Multiple constraints
37
Area-based method

• Assume spatial smoothness

38
Aperture problem
39
Aperture problem
40
Aperture problem
41
Demo for aperture problem

• http//www.sandlotscience.com/Distortions/Breathin
  g_Square.htm
• http//www.sandlotscience.com/Ambiguous/Barberpole
  _Illusion.htm

42
Aperture problem

• Larger window reduces ambiguity, but easily
  violates spatial smoothness assumption

43
Area-based method

• Assume spatial smoothness

44
Area-based method
must be invertible
45
Area-based method

• The eigenvalues tell us about the local image
  structure.
• They also tell us how well we can estimate the
  flow in both directions
• Link to Harris corner detector

46
Textured area
47
Edge
48
Homogenous area
49
KLT tracking

• Select feature by
• Monitor features by measuring dissimilarity

50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
KLT tracking
http//www.ces.clemson.edu/stb/klt/
54
KLT tracking
http//www.ces.clemson.edu/stb/klt/
55
SIFT tracking (matching actually)
Frame 10
?
Frame 0
56
SIFT tracking
Frame 100
?
Frame 0
57
SIFT tracking
Frame 200
?
Frame 0
58
KLT vs SIFT tracking

• KLT has larger accumulating error partly because
  our KLT implementation doesnt have affine
  transformation?
• SIFT is surprisingly robust
• Combination of SIFT and KLT (example)
• http//www.frc.ri.cmu.edu/projects/buzzard/smal
  ls/

59
Tracking for rotoscoping
60
Tracking for rotoscoping
61
Waking life
62
Optical flow
63
Single-motion assumption

• Violated by
• Motion discontinuity
• Shadows
• Transparency
• Specular reflection

64
Multiple motion
65
Multiple motion
66
Simple problem fit a line
67
Least-square fit
68
Least-square fit
69
Robust statistics

• Recover the best fit for the majority of the data
• Detect and reject outliers

70
Approach
71
Robust weighting
72
Robust estimation
73
(No Transcript)
74
(No Transcript)
75
(No Transcript)
76
Regularization and dense optical flow
77
(No Transcript)
78
(No Transcript)
79
(No Transcript)
80
(No Transcript)
81
(No Transcript)
82
(No Transcript)
83
(No Transcript)
84
(No Transcript)
85
(No Transcript)
86
(No Transcript)
87
Input for the NPR algorithm
88
Brushes
89
Edge clipping
90
Gradient
91
Smooth gradient
92
Textured brush
93
Edge clipping
94
Temporal artifacts

• Frame-by-frame application of the NPR algorithm

95
Temporal coherence
96
REVision
97
What dreams may come
98
Reference

• B.D. Lucas and T. Kanade, An Iterative Image
  Registration Technique with an Application to
  Stereo Vision, Proceedings of the 1981 DARPA
  Image Understanding Workshop, 1981, pp121-130.
• Bergen, J. R. and Anandan, P. and Hanna, K. J.
  and Hingorani, R., Hierarchical Model-Based
  Motion Estimation, ECCV 1992, pp237-252.
• J. Shi and C. Tomasi, Good Features to Track,
  CVPR 1994, pp593-600.
• Michael Black and P. Anandan, The Robust
  Estimation of Multiple Motions Parametric and
  Piecewise-Smooth Flow Fields, Computer Vision and
  Image Understanding 1996, pp75-104.
• S. Baker and I. Matthews, Lucas-Kanade 20 Years
  On A Unifying Framework, International Journal
  of Computer Vision, 56(3), 2004, pp221 - 255.
• Peter Litwinowicz, Processing Images and Video
  for An Impressionist Effects, SIGGRAPH 1997.
• Aseem Agarwala, Aaron Hertzman, David Salesin and
  Steven Seitz, Keyframe-Based Tracking for
  Rotoscoping and Animation, SIGGRAPH 2004,
  pp584-591.