A New Correspondence Algorithm

1
A New Correspondence Algorithm
Jitendra Malik Computer Science
Division University of California, Berkeley
Joint work with Serge Belongie, Jan Puzicha, Alex
Berg
2
Key contributions Years 1-4

• The FAÇADE system for semi-automated modeling of
  architectural scenes
• High dynamic range image acquisition
• Image based lighting
• Inverse global illumination for recovering
  reflectance and lighting properties
• Segmented objects from range images

3
Contributors

• Paul Debevec, now at ICT
• George Borshukov, recipient of Technical
  Achievement Award 2001 with colleagues at Manex
  visual effects
• Yizhou Yu, Asst. Prof., UIUC

4
What remains?

• High quality automated correspondence is
  essential
• 3D Structure recovery algorithms need to scale up
• Geometric and reflectance properties need to be
  modeled for a much larger range of scenes than
  previously considered

5
Towards better correspondence

• Humans use contextual information much more
  effectively than current algorithms.
• Features are not robust to changes in viewpoint.

6
How big a window?
7
The solution to the dilemma.

• Large windows capture more context but suffer
  from increased distortion.
• Goal Design a similarity measure which can
  tolerate affine distortion.
• Similarity should decrease linearly with the
  amount of distortion.
• Cross correlation does not have this property

8
An example

• Solution is to blur the signals, but how exactly?

9
Blurring the right way
10
Affine Robustness Condition
11
Affine Robust Feature
The bounded distortion blur of a signal f is the
Affine Robust Feature B(f). Constructively B is
a linear mapping with
And we take
0
1
2
0
1
2
12
In 2d

• Six oriented filters, half-wave rectified to
  provide12 channels
• Bounded distortion blur applied to each channel
• Similarity is the sum of similarities in each
  channel computed separately

13
Bounded Distortion Blur in 2D
14
Comparing three techniques
15
Another example
Given points in one image, find corresponding
points.
16
Another application Matching shapes
...
model
target

• Find correspondences between points on shape
• Estimate transformation
• Measure similarity

17
Shape Context
Count the number of points inside each bin, e.g.
Count 4
...
Count 10

• Compact representation of distribution of points
  relative to each point

18
Hand-written Digit Recognition

• MNIST 60 000
• linear 12.0
• 40 PCA quad 3.3
• 1000 RBF linear 3.6
• K-NN 5
• K-NN (deskewed) 2.4
• K-NN (tangent dist.) 1.1
• SVM 1.1
• LeNet 5 0.95

• MNIST 600 000 (distortions)
• LeNet 5 0.8
• SVM 0.8
• Boosted LeNet 4 0.7
• MNIST 20 000
• K-NN, Shape context matching 0.63

19
Conclusion

• A new image descriptor which is robust to affine
  image deformations
• Preliminary results suggest that this could
  result in a considerable improvement in quality
  of correspondence for long baseline multiple view
  analysis.

20
Plans for next 6 months

• Combine the use of the affine robust window
  features with the use of epipolar constraints and
  probabilistic matching.
• Test technique on stereo and motion imagery.
• Explore this in the context of an end to end
  system for scene reconstruction.