Stanford CS223B Computer Vision, Winter 2006 Lecture 7 Optical Flow

1
Stanford CS223B Computer Vision, Winter
2006Lecture 7 Optical Flow

• Professor Sebastian Thrun
• CAs Dan Maynes-Aminzade, Mitul Saha, Greg
  Corrado
• Slides by Gary Bradski, Intel Research and
  Stanford SAIL

2
(No Transcript)
3
Optical Flow Outline

• Examples
• Formal definition, 1D case
• From 1D to 2D Aperture Problem
• Course motion and pyramids
• Flow Segmentation

4
Optical Flow
Picture from Khurram Hassan-Shafique CAP5415
Computer Vision 2003
5
Optical Flow, Example
6
Another Example in Robotics
2) Use optical flow for near-far association
3) Associate known driveability with visually
similar regions
1) Rich local sensor data determines drivability
7
Optical Flow, Example
Harris Corners
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
8
Optical Flow, Example
Optical flow
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
9
Optical Flow, Example
Reward
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
10
Optical Flow, Example
Particle Filter
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
11
Optical Flow, Example
Result
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
12
Conference Paper
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
13
Optical Flow Outline

• Examples
• Formal definition, 1D case
• From 1D to 2D Aperture Problem
• Course motion and pyramids
• Flow Segmentation

14
Optical Flow
Image tracking
3D computation
Image sequence (single camera)
Tracked sequence
3D structure 3D trajectory
15
What is Optical Flow?
16
Optical Flow Break Down?
?
From Marc Pollefeys COMP 256 2003
17
Optical Flow AssumptionsBrightness Constancy
Slide from Michael Black, CS143 2003
18
Optical Flow Assumptions
Slide from Michael Black, CS143 2003
19
Optical Flow Assumptions
Slide from Michael Black, CS143 2003
20
Optical Flow 1D Case
Brightness Constancy Assumption
21
Tracking in the 1D case
22
Tracking in the 1D case
23
Tracking in the 1D case
Iterating helps refining the velocity vector
Converges in about 5 iterations
24
Algorithm for 1D tracking
25
Optical Flow Outline

• Examples
• Formal definition, 1D case
• From 1D to 2D Aperture Problem
• Course motion and pyramids
• Flow Segmentation

26
From 1D to 2D tracking
1D
Shoot! One equation, two velocity (u,v) unknowns
27
From 1D to 2D tracking
We get at most Normal Flow with one point we
can only detect movement perpendicular to the
brightness gradient. Solution is to take a patch
of pixels Around the pixel of interest.
Slide from Michael Black, CS143 2003
28
How does this show up visually?Known as the
Aperture Problem
29
Aperture Problem Exposed
Motion along just an edge is ambiguous
30
Aperture Problem Example
31
Aperture Problem in Real Life
32
From 1D to 2D tracking
The Math is very similar
Window size here 5x5 or 11x11
33
More DetailSolving the aperture problem

• How to get more equations for a pixel?
• Basic idea impose additional constraints
• most common is to assume that the flow field is
  smooth locally
• one method pretend the pixels neighbors have
  the same (u,v)
• If we use a 5x5 window, that gives us 25
  equations per pixel!

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
34
Lukas-Kanade flow

• Prob we have more equations than unknowns

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
35
Conditions for solvability

• Optimal (u, v) satisfies Lucas-Kanade equation

• When is This Solvable?
• ATA should be invertible
• ATA should not be too small due to noise
• eigenvalues l1 and l2 of ATA should not be too
  small
• ATA should be well-conditioned
• l1/ l2 should not be too large (l1 larger
  eigenvalue)

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
36
Eigenvectors of ATA

• Suppose (x,y) is on an edge. What is ATA?

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
37
Edge

• large gradients, all the same
• large l1, small l2

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
38
Low texture region

• gradients have small magnitude
• small l1, small l2

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
39
High textured region

• gradients are different, large magnitudes
• large l1, large l2

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
40
Observation

• This is a two image problem BUT
• Can measure sensitivity by just looking at one of
  the images!
• This tells us which pixels are easy to track,
  which are hard
• very useful later on when we do feature
  tracking...
• Once suggestion Track Harris Corners!

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
41
Optical Flow, Example
Harris Corners
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
42
Optical Flow, Example
Optical flow
David Stavens, Andrew Lookingbill, David Lieb
(CS223B 2004)
43
Optical Flow Outline

• Examples
• Formal definition, 1D case
• From 1D to 2D Aperture Problem
• Course motion and pyramids
• Flow Segmentation

44
Revisiting the small motion assumption

• Is this motion small enough?
• Probably notits much larger than one pixel (2nd
  order terms dominate)
• How might we solve this problem?

From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
45
Reduce the resolution!
From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
46
Coarse-to-fine optical flow estimation
47
Coarse-to-fine optical flow estimation
run iterative L-K
48
Multi-resolution Lucas Kanade Algorithm
49
Optical Flow Results
From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
50
Optical Flow Results
From Khurram Hassan-Shafique CAP5415 Computer
Vision 2003
51
Optical Flow Outline

• Examples
• Formal definition, 1D case
• From 1D to 2D Aperture Problem
• Course motion and pyramids
• Flow Segmentation

52
Problem

• Multiple motion types in the image
• (e.g., moving cars on 280 or El Camino Real)

53
Motion Template Idea
54
Motion Segmentation Algorithm

• Stamp the current motion history template with
  the system time and overlay it on top of the
  others

55
Motion Segmentation Algorithm

• Measure gradients of the overlaid motion history
  templates

56
Motion Segmentation Algorithm

• Threshold large gradients to get rid of motion
  template edges resulting from too large of a time
  delay

57
Motion Segmentation Algorithm

• Find boundaries of most recent motions
• Walk around boundary
• If drop not too high, Flood fill downwards to
  segment motions

Segmented Motion
Segmented Motion
58
Motion Segmentation Algorithm

• Actually need a two-pass algorithm for labeling
  all motion segments
• Fill downwards At bottom, turn around and fill
  upwards.
• Keep the union of these fills as the segmented
  motion.

59
Motion Template for Motion Segmentation and
Gesture
Overlay silhouettes, take gradient for normal
optical flow. Flood fill to segment motions.
Motion Segmentation
Motion Segmentation
Pose Recognition
Gesture Recognition