Computer%20Vision

1
Computer Vision

• Spring 2006 15-385,-685
• Instructor S. Narasimhan
• Wean 5403
• T-R 300pm 420pm
• Lecture 16

2
Announcements

• Homework 4 due today.
• Homework 5 will be out tonight, due in two
  weeks.
• Use bboard frequently and visit us during OH.

3

• Optical Flow and Motion
• Lecture 16

4
Optical Flow and Motion

• We are interested in finding the movement of
• scene objects from time-varying images (videos).
• Lots of uses
• Track object behavior
• Correct for camera jitter (stabilization)
• Align images (mosaics)
• 3D shape reconstruction
• Special effects

5
Tracking Rigid Objects
(Simon Baker, CMU)
6
Tracking Non-rigid Objects
(Comaniciu et al, Siemens)
7
Face Tracking
(Simon Baker et al, CMU)
8
Applications of Face Tracking

• User Interfaces
• Mouse Replacement Head Pose and Gaze Estimation
• Automotive Windshield Displays, Smart Airbags,
  Driver Monitoring
• Face Recognition
• Pose Normalization
• Model-Based Face Recognition
• Lipreading/Audio-Visual Speech Recognition
• Expression Recognition and Deception Detection
• Rendering and Animation
• Expression Animations and Transfer
• Low-Bandwidth Video Conferencing
• Audio-Visual Speech Synthesis

(Simon Baker, CMU)
9
3D Structure from Motion
(David Nister, Kentucky)
10
3D Structure from Motion
(David Nister, Kentucky)
11
Behavior Analysis
Query
Result
(Michal Irani, Weizmann)
12
Motion Field

• Image velocity of a point moving in the scene

13
Optical Flow

• Motion of brightness pattern in the image
• Ideally Optical flow Motion field

14
Optical Flow Motion Field
Motion field exists but no optical flow
No motion field but shading changes
15
Problem Definition Optical Flow

• How to estimate pixel motion from image H to
  image I?

• Find pixel correspondences
• Given a pixel in H, look for nearby pixels of the
  same color in I

• Key assumptions
• color constancy a point in H looks the same
  in image I
• For grayscale images, this is brightness
  constancy
• small motion points do not move very far

16
Optical Flow Constraint Equation
Optical Flow Velocities
Displacement

• Assume brightness of patch remains same in both
  images

• Assume small motion (Taylor expansion of LHS
  upto first order)

17
Optical Flow Constraint Equation
Divide by and take the limit
Constraint Equation
NOTE must lie on a straight line We
can compute using gradient
operators! But, (u,v) cannot be found uniquely
with this constraint!
18
Finding Gradients in X-Y-T
y
time
j1
k1
j
k
x
i
i1
19
Optical Flow Constraint

• Intuitively, what does this constraint mean?

• The component of the flow in the gradient
  direction is determined
• The component of the flow parallel to an edge is
  unknown

20
Optical Flow Constraint
21
Aperture Problem
22
Aperture Problem
23
Computing Optical Flow

• Formulate Error in Optical Flow Constraint
• We need additional constraints!
• Smoothness Constraint (as in shape from shading
  and stereo)
• Usually motion field varies smoothly in the
  image.
• So, penalize departure from smoothness
• Find (u,v) at each image point that MINIMIZES

weighting factor
24
Discrete Optical Flow Algorithm

• Consider image pixel
• Departure from Smoothness Constraint
• Error in Optical Flow constraint equation
• We seek the set that
  minimize

NOTE show up in more than one term
25
Discrete Optical Flow Algorithm

• Differentiating w.r.t
  and setting to zero
• are averages of
  around pixel

Update Rule
26
Example
27
Optical Flow Result
28
Low Texture Region - Bad

• gradients have small magnitude

29
Edges so,so (aperture problem)

• large gradients, all the same

30
High Textured Region - Good

• gradients are different, large magnitudes

31
Focus of Expansion (FOE)

• Motion of object - (Motion of Sensor)
• For a given translatory motion and gaze
  direction, the world
• seems to flow out of one point (FOE).

After time t, the scene point moves to
32
Focus of Expansion (FOE)

• As t varies the image point moves along a
  straight line in the image
• Focus of Expansion Lets backtrack time or

33
Focus of Expansion (FOE) - Example
http//homepages.inf.ed.ac.uk/rbf/BOOKS/BANDB/LIB/
bandb7_12.pdf
34
Revisiting the Small Motion Assumption

• Is this motion small enough?
• Probably notits much larger than one pixel (2nd
  order terms dominate)
• How might we solve this problem?

35
Reduce the Resolution!
36
Coarse-to-fine Optical Flow Estimation
37
Coarse-to-fine Optical Flow Estimation
run iterative OF
38
Image Alignment

• Goal Estimate single (u,v) translation
  (transformation) for entire image

39
Mosaicing
(Michal Irani, Weizmann)
40
Mosaicing
(Michal Irani, Weizmann)
41
Next Class

• Structured Light and Range Imaging
• Reading ? Notes