Lucas-Kanade Image Alignment

1
Lucas-Kanade Image Alignment

• Iain Matthews

2
Paper Reading

• Simon Baker and Iain Matthews,
• Lucas-Kanade 20 years on A Unifying Framework,
  Part 1
• http//www.ri.cmu.edu/pub_files/pub3/baker_simon_2
  002_3/baker_simon_2002_3.pdf
• And Project 3 Description

3
Recall - Image Processing Lecture

• Some operations preserve the range but change the
  domain of f
• What kinds of operations can this perform?
• Still other operations operate on both the domain
  and the range of f .

4
Face Morphing
5
(No Transcript)
6
(No Transcript)
7
Applications of Image Alignment

• Ubiquitous computer vision technique
• Mosaicing
• Tracking
• Parametric and layered motion estimation
• Image registration and alignment
• Face coding / parameterization
• Super-resolution

8
Generative Model for an Image

• Parameterized model

shape
Parameters
Image
appearance
9
Fitting a Model to an Image

• What are the best model parameters to match an
  image?

shape
Parameters
Image
appearance

• Nonlinear optimization problem

10
Active Appearance Model

• Cootes, Edwards, Taylor, 1998

11
Image Alignment
12
Want to Minimize the Error

• Warp image to get compute

13
How to Minimize the Error

• Minimise SSD with respect to p,

Generally a nonlinear optimisation problem ?
14
Linearize

• Taylor series expansion, linearize function f
  about x0

15
Gradient Descent Solution

• Least squares problem, solve for ?p

16
Gradient Images

• Compute image gradient

W(xp)
W(xp)
I(W(xp))
17
Jacobian

• Compute Jacobian
• Mesh parameterization

4
1
4
1
Warp, W(xp)
Template, T(x)
Image, I(x)
3
2
3
2
Image coordinates x (x, y)T
Warp parameters, p (p1, p2, , pn)T (dx1,
dy1, , dxn, dyn)T
18
Lucas-Kanade Algorithm

1. Warp I with W(xp) ? I(W(xp))
2. Compute error image T(x) - I(W(xp))
3. Warp gradient of I to compute ?I
4. Evaluate Jacobian
5. Compute Hessian
6. Compute ?p
7. Update parameters p ? p ?p

-

?
19
Fast Gradient Descent?

• To reduce Hessian computation
• Make Jacobian simple (or constant)
• Avoid computing gradients on I

20
Shum-Szeliski Image Aligment

• Additive Image Alignment Lucas, Kanade

W(x0 ?p) W(x?p)
21
Compositional Image Alignment

• Minimise,

Jacobian is constant, evaluated at (x, 0) ?
simple.
22
Compositional Algorithm

1. Warp I with W(xp) ? I(W(xp))
2. Compute error image T(x) - I(W(xp))
3. Warp gradient of I to compute ?I
4. Evaluate Jacobian
5. Compute Hessian
6. Compute ?p
7. Update W(xp) ? W(xp) o W(x?p)

-

?
23
Inverse Compositional

• Why compute updates on I?
• Can we reverse the roles of the images?
• Yes!
• Baker, Matthews CMU-RI-TR-01-03 Proof that
  algorithms take the same steps (to first order)

24
Inverse Compositional

• Forwards compositional

• Inverse compositional

25
Inverse Compositional

• Minimise,
• Solution
• Update

?
26
Inverse Compositional

• Jacobian is constant - evaluated at (x, 0)
• Gradient of template is constant
• Hessian is constant

• Can pre-compute everything but error image!

27
Inverse Compositional Algorithm

1. Warp I with W(xp) ? I(W(xp))
2. Compute error image T(x) - I(W(xp))
3. Warp gradient of I to compute ?I
4. Evaluate Jacobian
5. Compute Hessian
6. Compute ?p
7. Update W(xp) ? W(xp) o W(x?p)-1

-

?
28
Framework

• Baker and Matthews 2003
• Formulated framework, proved equivalence

Algorithm Can be applied to Efficient? Authors
Forwards Additive Any No Lucas, Kanade
Forwards Compositional Any semi-group No Shum, Szeliski
Inverse Compositional Any group Yes Baker, Matthews
Inverse Additive Simple linear 2D Yes Hager, Belhumeur
29
Example