Title: Optical Flow
1Optical Flow
2Optical Flow
- Distribution of apparent velocities of movement
of brightness pattern in an image
3Optical Flow
-
- Perspective projection (pinhole camera)
- Instantaneous 2D velocity
4Definitions
The optical flow is a velocity field in the image
which transforms one image into the next image in
a sequence HornSchunck
frame 2
frame 1
flow field
The motion field is the projection into the
image of three-dimensional motion vectors
HornSchunck
5Ambiguity of optical flow
Frame 1
6Determining Optical Flow
- Berthold K.P. Horn and Brian G. Rhunck
- Artificial Intelligence Laboratory, MIT.
7Optical Flow
- Optical flow field Estimate the 2D motion field,
which are the 2D velocities for all visible
points - Two key problems
- Determine what image property to track
- Determine how to track it
8Brightness Constancy
u
frame t1
v
frame t
9Brightness Constancy
- Track points of constant brightness, assuming
that surface radiance is constant over time - Brightness constancy is often violated by Mother
Nature, so the resulting optical flow field is
sometimes a very poor approximation to the 2D
motion field
10Optical Flow
- The optical flow cannot be computed at a point
independently of neighboring points without
introducing additional constraints
11Brightness Constancy Constraint
- The brightness of a particular point in the
pattern is constant - Cannot determine the flow velocity
- locally without additional
constraints
Constraint line
12Brightness Constancy Constraint
- Normal Velocity
- No constraint when the gradient magnitude is zero
13Optical Flow Assumptions
Slide from Michael Black, CS143 2003
14Optical Flow Assumptions
Slide from Michael Black, CS143 2003
15Smoothness Constraint
- Constraint Assumption
- Neighboring points on the objects have similar
velocities and - The velocity field of the brightness patterns in
the image varies smoothly almost everywhere - Discontinuities in flow can be expected where one
object occludes another - Algorithms based on smoothness constraint are
likely to have difficulties with occluding edges
16Smoothness Constraint
- Minimize the square of the gradient magnitude of
the optical flow velocity - Sum of the squares of Laplacians of x- and y-
components of the flow
17Estimate the Partial Derivatives
18Estimate the Laplacian of Flow Velocities
19Minimization
- Minimize the sum of errors in rate of change of
image brightness - And measure of departure from smoothness
- Total error to be minimized
20Minimization
21Difference of Flow from Local Average
- (u,v) lies in the direction towards the
constraint line along a line that perpendicular
to it - a2 plays significant role when Ex, Ey is small
roughly equal to the expected noise in the
estimate of
22Constrained Minimization
- Minimizing
- While maintaining
0 - The (u,v) here is just the intersection on the
constraint line
23Iterative Solution
24Optical Flow 1D Case
Brightness Constancy Assumption
25Tracking in the 1D case
26Tracking in the 1D case
27Tracking in the 1D case
Iterating helps refining the velocity vector
Converges in about 5 iterations
28Algorithm for 1D tracking
29Iterative Solution
- Very costly to solve these equations
simultaneously - Iterative algorithms, such as Gauss-Seidel method
It is interesting to note that the new estimates
at a particular point do not depend directly on
the previous estimate at the same point
30Fill in Uniform Region
- Uniform region brightness gradient is zero
- No local information to constrain the vecocity
- Fill in uniform region
- Given the values on region boudnary
- Solve the Laplace equation under Dirichlet
boundary condition
31Tightness of Constraint
- In general, the solution is most accurately
determined in the region where the brightness is
not too small and varies in direction from point
to point - Constraint information is available in a relative
small neighborhood
32Choice of Iterative Scheme
- How to interlace the iterations with the time
step? - Get stable values before going to next frame
- Only one iteration per time-step gives good
initial guess - Ability to deal with more images per unit time
- Better estimation in uniform regions because of
the noise in measurements of images will be
independent and tend to cancel out
33Experiment
- Very low resolution video, say, 3232
- A rotating ball with smoothly varying patterns,
but no shading
34Results I
- Simple linear translation of the entire pattern
- Estimate between two images
- Showing the velocity
- Few changes occur after 32 iterations when errors
10
35Results I
1 time step
4 time steps
36Results I
16 time steps
64 time steps
37Results II
- Only one iteration per time-step
- More rapid
- Few changes after 16 iteration when errors 7
38Results II
1 time steps
4 time steps
39Results II
16 time steps
64 time steps
40Results III
- Simple rotation and contraction of the brightness
pattern
41Result III
initial
32 time steps
42Result IV
- Rigid body motion
- Laplacian for one of the velocity components
becomes infinite on the occluding bound - Reasonable results are still obtained with finite
velocities
43Results IV
Rotating cylinder
Rotating sphere
44Results IV
Exact flow pattern for a rotating cylinder and
rotating sphere
45Summary
- A method for computing optical flow from a
sequence of images - Two components of flow velocity
- Additional constraint smoothness
- Iterative method
46Optical Flow Research Timeline
HornSchunck
LucasKanade
1981
1992
1998
now
BenchmarkGalvin et.al.
BenchmarkBarron et.al.
Seminal papers
A slow and not very consistent improvement in
results, but a lot of useful ingredients were
developed
47http//people.csail.mit.edu/celiu/ECCV2008/zz