Optical Flow Methods

1
Optical Flow Methods

• 2007/8/9

2
Outline

• Introduction to 2-D Motion
• The Optical Flow Equation
• The Solution of Optical Flow Equation
• Comparison of different methods
• Reference

3
The 2-D Motion

• The projection of 3-D motion into the image plane.

4
The 2-D Motion(2)

• A 2-D displacement field is a collection of 2-D
  displacement vectors.

5
Definition of Optical Flow

• Optical flow is a vector field of pixel
  velocities based on the observable variations
  form the time-varying image intensity patter.

6
Difference between Optical flow and 2-D
displacement(1)

• There must be sufficient gray-level variation for
  the actual motion to be observable.

7
Difference between Optical flow and 2-D
displacement(2)

• An observable optical flow may not always
  correspond to actual motion. For example changes
  in external illumination.

8
Outline

• Introduction to 2-D Motion
• The Optical Flow Equation
• The Solution of Optical Flow Equation
• Comparison of different methods
• Reference

9
The Optical Flow Equation(1)

• Let the image brightness at the point (x, y) in
  the image plane at time t be denoted by
• The brightness of a particular point in the
  pattern is constant, so that
• Using the chain rule for differentiation we see
  that,

10
The Optical Flow Equation(2)

• If we let and ,
  for the partial
  derivatives, we have a single linear equation in
  two unknowns u and v.
• Writing the equation in the two unknowns u and v,

11
The Optical Flow Equation(3)

• Writing the equation in another form,
• The component of the movement in the direction of
  the brightness gradient equals

12
The Optical Flow Equation(4)

• The velocity has to lie along a line
  perpendicular to the brightness gradient vector.

13
Outline

• Introduction to 2-D Motion
• The Optical Flow Equation
• The Solution of Optical Flow Equation
• Comparison of different methods
• Reference

14
Second-Order Differential Methods(1)

• Based on the conservation of the spatial image
  gradient.
• The flow field is given by

15
Second-Order Differential Methods(2)

• The deficiencies
• The constraint does not allow for some motion
  such as rotation and zooming.
• Second-order partials cannot always be estimated
  with sufficient accuracy.

16
Block Motion Model (1) (Lucas and Kanade Method)

• Based on the assumption that the motion vector
  remains unchanged over a particular block of
  pixels.
• for x,y inside block B

17
Block Motion Model (2)

• Computing the partials of error with respect to
  and , then setting them equal to zero, we
  have

18
Block Motion Model (3)

• Solving the equations, we have

19
Block Motion Model (4)

• It is possible to increase the influence of the
  constraints towards the center of the block by
  weighted summations.
• The accuracy of estimation depends on the
  accuracy of the estimated spatial and temporal
  partial derivatives.

20
Horn and Schunck Method(1)

• The additional constraint is to minimize the sum
  of the squares of the Laplacians of the optical
  flow velocity
• and

21
Horn and Schunck Method(2)

• The minimization of the sum of the errors in the
  equation for the rate of changes of image
  brightness.
• and the measure of smoothness in the velocity
  flow.

22
Horn and Schunck Method(3)

• Let the total error to be minimized be
• The minimization is to be accomplished by finding
  suitable values for optical flow velocity (u ,v).
• The solution can be found iteratively.

23
Horn and Schunck Method Directional-Smoothness
constraint

• The directional smoothness constraint
• W is a weight matrix depending on the spatial
  changes in gray level content of the video.
• The directional-smoothness method minimizes the
  criterion function

24
Gradient Estimation Using Finite Differences(1)

• To obtain the estimates of the partials, we can
  compute the average of the forward and backward
  finite differences.

25
Gradient Estimation Using Finite Differences(2)

• The three partial derivatives of images
  brightness at the center of the cube are
  estimated form the average of differences along
  four parallel edges of the cube.

26
Gradient Estimation by Local Polynomial Fitting(1)

• An approach to approximate E(x,y,t) locally by a
  linear combination of some low-order polynomials
  in x, y, and t that is,
• Set N equal to 9 and choose the following basis
  functions

27
Gradient Estimation by Local Polynomial Fitting(2)

• The coefficients are estimated by using the least
  squares method.
• The components of the gradient can be found by
  differentiation,

28
Estimating the Laplacian of the Flow Velocities(1)

• The approximation takes the following form
• and
• The local averages u and v are defined as

29
Estimating the Laplacian of the Flow Velocities(2)

• The Laplacian is estimated by subtracting the
  value at a point form a weighted average of the
  values at neighboring points.

30
Outline

• Introduction to 2-D Motion
• The Optical Flow Equation
• The Solution of Optical Flow Equation
• Comparison of different methods
• Reference

31
Comparison of different methods(1)

• Three different method to be compared
• Lucas-Kanade method based on block motion model.
  (11x11 blocks with no weighting)
• Horn-Schunck method imposing a global smoothness
  constraint.( , allowed for 20 to 150
  iterations)
• The directional-smoothness method of Nagel(
  with 20 iterations)

32
Comparison of different methods(2)

• These methods have been applied to the 7th and
  8th frames of a video sequence, known as the
  Mobile and Calendar.
• The gradients have been approximated by average
  finite differences and polynomial fitting.
• The images are spatially pre-smoothed by a 5x5
  Gaussian kernel with the variance 2.5 pixels.

33
Comparison of different methods(3)

• Comparison of the differential methods.

Method PSNR(dB) PSNR(dB) Entropy(bits) Entropy(bits)
Polynomial Difference Polynomial Difference
Frame-Difference Lucas-Kanade Horn-Schunck Nagel 23.45 30.89 28.14 29.08 - 32.09 30.71 31.84 - 6.44 4.22 5.83 - 6.82 5.04 5.95
34
Outline

• Introduction to 2-D Motion
• The Optical Flow Equation
• The Solution of Optical Flow Equation
• Comparison of different methods
• Reference

35
Reference

• A. M. Tekalp, Digital Video Processing. Englewood
  Cliffs, NJ Prentice-Hall, 1995.
• Horn, B.K.P. and Schunck, B.G. Determining
  optical flowA retrospective, Artificial
  Intelligence, vol. 17, 1981, pp.185-203.
• J.L. Barron, D.J. Fleet, and S.S. Beauchemin,
  Performance of Optical Flow Techniques, in
  International Journal of Computer Vision,
  February 1994, vol. 12(1), pp. 43-77.