The basics

1
Computing motion between images

• The basics

2
Matching images

• We will deal with two similar tasks
• A. Given a point in one image, find its matching
  point in the other image - feature
  matching/optical flow.
• B. Find the transformation mapping one image to
  the other - Image alignment/ Image Registration.

Feature matching/ optical flow
Alignment Scale by 3/4
3
Why is it useful

• Camera stabilization
• Tracking moving objects
• Finding the camera motion and 3D reconstruction.
• Image processing by combining several images
  (mosaicing, super-resolution)
• Compression.
• ..

4
Matching by sliding window

• A basic method to find matching points between
  images. (correspondence)
• a neighborhood of the point (Window) in image I1
  is compared to several Windows in I2.
• The window with the best score wins.
• Various scores can be used.

5
Matching by sliding window
The winning window
The window in image 1
A sliding window in I2
6
Matching Criteria - Difference

• Common matching criteria
• SSD - Sum of Squared Differences
• k - the size of the window.
• p1 (x1, y1) is the center of the window in I1.
• p2 (x2, y2) is the center of the window in I2.
  
• SAD - Sum of Absolute Differences

7
Matching Criteria - Correlation

• Cross Correlation is similar to SSD, but can be
  implemented more efficiently

The window to search
The image to search (I2)
The maximum cross correlation score
The cross correlation score
8
Handling illumination changes

• SSD, SAD and Cross Correlation assume constant
  brightness.
• On order to handle illumination changes,
  Normalized cross-correlation can be used.

I1
I2
9
Normalized Cross Correlation

• When ordering the pixels in the windows in
  vectors v1, v2
• The Normalized Cross Correlation is

Squared Vector norm
Inner product
10
The aperture problem
?
?
?
11
The aperture problem (cont)
Easy to track in both directions.
Hard to track vertically.
12
The aperture problem (cont)
T junction
normal flow
real motion
13
And what about smooth areas ?

• Use bigger windows
• Less noise
• Smaller aperture problem
• Might include different motions
• Smooth the image !

?
?
14
Using a Pyramid for optical flow
Pyramid0

• Smoothing the images we get an estimation of the
  motion in uniform regions.
• Handling large motions Even if the motion in the
  original image is big, the motion in the small
  level is small.
• Once the solution was find in level k, there is
  only small motion to fix in the level k-1 The
  optical flow calculations become much faster !

Pyramid1
Pyramid2
15
Auto-Correlation (and how does the SSD handle a
combination of motions)
The AC meaning How similar does the image looks
to it selves in different shifts. It has a
strong connection with the aperture problem. The
SSD should find the best motion depending on
the auto-correlation function.
16
Deformations

• The window matching assumes a pure image
  translation in small regions.
• Possible solutions for deformations
• Invariant features
• Iterations of motion computations warping

17
Limitations of Window Matching

• Accuracy
• A pixel is always matched to integer location on
  the grid. The image motion is usually not
  integer.
• Neighborhood/Scene constraints
• High level knowledge about the scene/camera may
  help in limiting the search, and reducing errors.
  (for example, the scene is planar)