Stanford CS223B Computer Vision, Winter 200809 Lecture 8 Optical Flow

1
Stanford CS223B Computer Vision, Winter
2008/09Lecture 8Optical Flow

• Professor Sebastian Thrun
• CAs Ethan Dreyfuss, Young Min Kim, Alex Teichman
  

2
Optical Flow Outline

• Examples
• Formal definition, 1D case
• From 1D to 2D Aperture Problem
• Course motion and pyramids

3
Stereo versus Optical Flow

• Stereo 2 or more images taken at same point in
  time, from different vantage points
• Optical Flow 2 or more images taken one after
  another
• Objects in scene may move
• Camera may move (but motion unknown)
• Goal of optical flow track features/objects

4
Optical Flow
Image tracking
3D computation
Image sequence (single camera)
Tracked sequence
3D structure 3D trajectory
Courtesy of Jean-Yves Bouguet
5
Optical Flow
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
6
Optical Flow
Image tracking
3D computation
Image sequence (single camera)
Tracked sequence
Courtesy of Jean-Yves Bouguet
7
Optical Flow (Structure From Motion)
3D structure 3D trajectory
Courtesy of Jean-Yves Bouguet
8
Optical Flow Break Down?
?
From Marc Pollefeys COMP 256 2003
9
(No Transcript)
10
Optical Flow Outline

• Examples
• Formal definition and assumptions
• Optical Flow 1D case
• From 1D to 2D Aperture Problem
• Course motion and pyramids

11
What is Optical Flow?
12
Optical Flow Assumptions

• Brightness Constancy
• Spatial Coherence
• Temporal Persistence

13
Optical Flow AssumptionsBrightness Constancy
Slide from Michael Black, CS143 2003
14
Optical Flow Assumptions
Slide from Michael Black, CS143 2003
15
Optical Flow Assumptions
Slide from Michael Black, CS143 2003
16
Optical Flow Outline

• Examples
• Formal definition and assumptions
• Optical Flow 1D case
• From 1D to 2D Aperture Problem
• Course motion and pyramids

17
Optical Flow 1D Case
Brightness Constancy Assumption
18
Tracking in the 1D case
19
Tracking in the 1D case
20
Tracking in the 1D case
Iterating helps refining the velocity vector
Converges in about 5 iterations
21
Algorithm for 1D tracking
22
Optical Flow Outline

• Examples
• Formal definition and assumptions
• Optical Flow 1D case
• From 1D to 2D Aperture Problem
• Course motion and pyramids

23
From 1D to 2D tracking
1D
Shoot! One equation, two velocity (u,v) unknowns
24
From 1D to 2D tracking
We get at most Normal Flow with one point we
can only detect movement perpendicular to the
brightness gradient. Solution is to take a patch
of pixels Around the pixel of interest.
Slide from Michael Black, CS143 2003
25
How does this show up visually?Known as the
Aperture Problem
26
Aperture Problem Exposed
Motion along just an edge is ambiguous
27
Aperture Problem Example
28
Aperture Problem in Real Life
29
Solving the aperture problem

• How to get more equations for a pixel?
• Basic idea impose additional constraints
• most common is to assume that the flow field is
  smooth locally
• one method pretend the pixels neighbors have
  the same (u,v)
• If we use a 5x5 window, that gives us 25
  equations per pixel!

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
30
Lukas-Kanade flow

• Prob we have more equations than unknowns

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
31
Conditions for solvability

• Optimal (u, v) satisfies Lucas-Kanade equation

• When is This Solvable?
• ATA should be invertible
• ATA should not be too small due to noise
• eigenvalues l1 and l2 of ATA should not be too
  small
• ATA should be well-conditioned
• l1/ l2 should not be too large (l1 larger
  eigenvalue)

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
32
Eigenvectors of ATA

• Suppose (x,y) is on an edge. What is ATA?

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
33
Edge

• large gradients, all the same
• large l1, small l2

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
34
Low texture region

• gradients have small magnitude
• small l1, small l2

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
35
High textured region

• gradients are different, large magnitudes
• large l1, large l2

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
36
Observation

• This is a two image problem BUT
• Can measure sensitivity by just looking at one of
  the images!
• This tells us which pixels are easy to track,
  which are hard
• One suggestion Track Harris Corners!

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
37
Optical Flow, Example
Harris Corners
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
38
Optical Flow, Example
Optical flow
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
39
Optical Flow, Example
Particle Filter
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
40
Optical Flow, Example
Final output
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
41
Optical Flow Outline

• Examples
• Formal definition and assumptions
• Optical Flow 1D case
• From 1D to 2D Aperture Problem
• Course motion and pyramids

42
Revisiting the small motion assumption

• Is this motion small enough?
• Probably notits much larger than one pixel (2nd
  order terms dominate)
• How might we solve this problem?

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
43
Reduce the resolution!
From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
44
Coarse-to-fine optical flow estimation
45
Coarse-to-fine optical flow estimation
run iterative L-K
46
Multi-resolution Lucas Kanade Algorithm
47
Optical Flow Results
From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
48
Optical Flow Results
From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
49
Summary Optical Flow

• Examples
• Formal definition and assumptions
• Optical Flow 1D case
• From 1D to 2D Aperture Problem
• Course motion and pyramids