Correspondence and Stereopsis

1
Correspondence and Stereopsis
2
Introduction

• Disparity
• Informally difference between two pictures
• Allows us to gain a strong sense of depth
• Stereopsis
• Ability to perceive depth from disparity
• Goal of this chapter
• Design algorithms that mimic stereopsis

3
Applications of Stereopsis

• Visual robot navigation
• Cartography
• Aerial reconnaissance
• Close-range photogrammetry
• Image segmentation for object recognition

4
Stereo Vision

• Two processes
• Binocular fusion of features observed by the eyes
• Reconstruction of their three-dimensional preimage

5
Stereo Vision Easy Case

• 1 single point being observed
• The preimage can be found at the intersection of
  the rays from the focal points to the image points

6
Stereo Vision Hard Case

• Many points being observed
• Need some method to establish correct
  correspondences

7
Components of Stereo Vision Systems

• Camera calibration previous lectures
• Image rectification simplifies the search for
  correspondences
• Correspondence which item in the left image
  corresponds to which item in the right image
• Reconstruction recovers 3-D information from the
  2-D correspondences

8
Epipolar Geometry

• Epipolar constraint corresponding points must
  lie on conjugated epipolar lines
• Search for correspondences becomes a 1-D problem

9
Image Rectification

• Corresponding epipolar lines become collinear

10
Image Rectification (cont.)

• Not equivalent to rotation
• The lines through the centers become parallel to
  each other, and the epipoles move to infinity

11
Image Rectification (cont.)

• Given extrinsic parameters T and R (relative
  position and orientation of the two cameras)
• Rotate the left camera about the projection
  center so that the the epipolar lines become
  parallel to the horizontal axis
• Apply the same rotation to the right camera
• Rotate the right camera by R
• Adjust the scale in both camera reference frames

12
Image Rectification (cont.)

• Formal definition of disparity du'u

13
Correspondence

• Given an element in the left image, find the
  corresponding element in the right image
• Classes of methods
• Correlation-based
• Feature-based

14
Correlation-Based Correspondence

• Input rectified stereo pair and a point (u,v) in
  the first image
• Method
• Associate a window of size p(2m1)(2n1)
  centered in (u,v) and form the vector w(u,v) in
  Rp
• For each potential match (ud,v) in the second
  image, compute w' and the normalized correlation
  between w and w'

15
Correlation-Based Correspondence (cont.)

• Main problem
• Implicitly assume that the observed surface is
  locally parallel to the two image planes
• Alleviated by computing an initial disparity and
  using it to warp the correlation windows to
  compensate for unequal amounts of foreshortening
• Other problems
• Not robust against noise
• Similar pixels may not correspond to physical
  features

16
Feature-Based Correspondence

• Main idea physically-significant features should
  be preferred to matches between raw pixel
  intensities
• Instead of correlation-like measures, use a
  measure of the distance between feature
  descriptors
• Typical features points, lines, and corners
• Example Marr-Poggio-Grimson algorithm

17
Marr-Poggio-Grimson Algorithm

• Convolve images with Laplacian of Gaussian
  filters with standard deviations s1lts2lts3lts4
• Find zero crossings of the Laplacian along
  horizontal scanlines of the filtered images
• For each s, match zero crossings with same parity
  and similar orientations in a ws,ws disparity
  range, with

18
Marr-Poggio-Grimson Algorithm (cont.)

• Use disparities found at larger scales to control
  eye vergence and cause unmatched regions at
  smaller scales to come into correspondence

19
Marr-Poggio-Grimson algorithm (cont.)
20
Marr-Poggio-Grimson algorithm (cont.)
21
Ordering Constraint

• The order of matching image features along a pair
  of epipolar lines is (usually) the inverse of the
  order of the corresponding surface attributes
  along the curve where the epipolar plane
  intersects the object's boundary

22
Ordering Constraint (cont.)

• May not be satisfied in real scenes due to
  occlusion
• Still useful to devise efficient algorithms
  relying on dynamic programming to establish
  stereo correspondences

23
Reconstruction

• Given pair of image points p and p', and focal
  points O and O', find preimage P
• In theory find P by intersecting the rays ROp
  and R'Op'
• In practice R and R' won't actually intersect
  due to calibration and feature localization errors

24
Reconstruction Approaches

• Geometric
• Construct the line segment perpendicular to R and
  R' that intersects both rays and take its
  mid-point

25
Reconstruction Approaches (cont.)

• Algebraic (linear)
• Write down the projection equations
• The resulting linear system is overconstrained
• Solve it by linear least-squares

26
Reconstruction Approaches (cont.)

• Algebraic (non-linear)
• Find the point Q that minimizes d2(p,q)d2(p',q')
  by non-linear least-squares
• Reconstructions obtained by the previous methods
  can be used as initial guesses for the
  optimization