Optical%20Flow

1
Optical Flow

• Marc Pollefeys
• COMP 256

Some slides and illustrations from L. Van Gool,
T. Darell, B. Horn, Y. Weiss, P. Anandan, M.
Black, K. Toyama
2
last week polar rectification
3
Last week polar rectification
4
Last week Stereo matching

• Constraints
• epipolar
• ordering
• uniqueness
• disparity limit
• disparity gradient limit
• Trade-off
• Matching cost (data)
• Discontinuities (prior)

(Cox et al. CVGIP96 Koch96 Falkenhagen97
Van Meerbergen,Vergauwen,Pollefeys,VanGool
IJCV02)
5
Questions for assignment?
6
Tentative class schedule
Aug 26/28 - Introduction
Sep 2/4 Cameras Radiometry
Sep 9/11 Sources Shadows Color
Sep 16/18 Linear filters edges (Isabel hurricane)
Sep 23/25 Pyramids Texture Multi-View Geometry
Sep30/Oct2 Stereo Project proposals
Oct 7/9 Tracking (Welch) Optical flow
Oct 14/16 - -
Oct 21/23 Silhouettes/carving (Fall break)
Oct 28/30 - Structure from motion
Nov 4/6 Project update Camera calibration
Nov 11/13 Segmentation Fitting
Nov 18/20 Prob. segm.fit. Matching templates
Nov 25/27 Matching relations (Thanksgiving)
Dec 2/4 Range data Final project
7
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

8
Optical FlowWhere do pixels move to?
9
(No Transcript)
10
(No Transcript)
11
Motion is a basic cue
Even impoverished motion data can elicit a strong
percept
12
Applications

• tracking
• structure from motion
• motion segmentation
• stabilization
• compression
• mosaicing

13
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

14
Definition of optical flow
OPTICAL FLOW apparent motion of
brightness patterns
Ideally, the optical flow is the projection of
the three-dimensional velocity vectors on the
image
?
15
Caution required !
Two examples
1. Uniform, rotating sphere ? O.F. 0
2. No motion, but changing lighting
? O.F. ? 0
?
16
Caution required !
17
Mathematical formulation
I (x,y,t) brightness at (x,y) at time t
Brightness constancy assumption
Optical flow constraint equation
?
18
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

19
The aperture problem
1 equation in 2 unknowns
?
20
The aperture problem
0
21
(No Transcript)
22
The aperture problem
23
Remarks
24
Apparently an aperture problem
?
25
What is Optic Flow, anyway?

• Estimate of observed projected motion field
• Not always well defined!
• Compare
• Motion Field (or Scene Flow)
• projection of 3-D motion field
• Normal Flow
• observed tangent motion
• Optic Flow
• apparent motion of the brightness pattern
• (hopefully equal to motion field)
• Consider Barber pole illusion

26
(No Transcript)
27
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

28
Horn Schunck algorithm
Additional smoothness constraint
besides OF constraint equation term
minimize es?ec
?
29
The calculus of variations
look for functions that extremize functionals
?
30
Calculus of variations
Suppose
1. f(x) is a solution
2. ? (x) is a test function with ? (x1) 0
and ? (x2) 0
for the optimum
?
31
Calculus of variations
regardless of ?(x), then
Euler-Lagrange equation
?
32
Calculus of variations
Generalizations
?
33
Calculus of variations
0
?
34
Calculus of variations
is the Euler-Lagrange equation
?
35
Horn Schunck
The Euler-Lagrange equations
In our case ,
?
36
Horn Schunck
Remarks
1. Coupled PDEs solved using iterative
methods and finite differences
2. More than two frames allow a better
estimation of It
3. Information spreads from corner-type
patterns
?
37
?
38
Horn Schunck, remarks
1. Errors at boundaries
2. Example of regularisation (selection
principle for the solution of illposed
problems)
?
39
Results of an enhanced system
40
Structure from motion with OF
41
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

42

43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
(No Transcript)
47
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

48
(No Transcript)
49
(No Transcript)
50
(No Transcript)
51
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
(No Transcript)
66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
(No Transcript)
70
(No Transcript)
71
(No Transcript)
72
(No Transcript)
73
(No Transcript)
74
(No Transcript)
75
(No Transcript)
76
(No Transcript)
77
(No Transcript)
78
(No Transcript)
79
(No Transcript)
80
(No Transcript)
81
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

82
(No Transcript)
83
(No Transcript)
84
(No Transcript)
85
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

86
(No Transcript)
87
(No Transcript)
88
(No Transcript)
89
(No Transcript)
90
(No Transcript)
91
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

92
(No Transcript)
93
(No Transcript)
94
(No Transcript)
95
(No Transcript)
96
(No Transcript)
97
(No Transcript)
98
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

99
(No Transcript)
100
Rhombus Displays
http//www.cs.huji.ac.il/yweiss/Rhombus/
101
(No Transcript)
102
(No Transcript)
103
(No Transcript)
104
(No Transcript)
105
(No Transcript)
106
(No Transcript)