Last%20Time

1
Last Time

• Pinhole camera model, projection
• A taste of projective geometry
• Two view geometry
• Homography
• Epipolar geometry, the essential matrix
• Camera calibration, the fundamental matrix

2
Epipolar Lines
epipolar plane
epipolar lines
epipolar lines
O
O
Baseline
3
Stereo Vision

• Objective 3D reconstruction
• Input 2 (or more) images taken with calibrated
  cameras
• Output 3D structure of scene
• Steps
• Rectification
• Matching
• Depth estimation

4
Rectification

• We will assume images have been rectified so that
  epipolar lines correspond to scan lines
• Image planes of cameras are parallel.
• Focal points are at same height.
• Focal lengths same.
• Then, epipolar lines fall along the horizontal
  scan lines of the images
• Any stereo pair can be rectified by rotating and
  scaling the two image planes (homography) so
  that they become parallel to baseline

5
Rectification

• Image Reprojection
• reproject image planes onto common plane parallel
  to baseline
• Notice, only focal point of camera really matters

(Seitz)
6
Cyclopean Coordinates

• Origin at midpoint between camera centers
• Axes parallel to those of the two (rectified)
  cameras

7
Disparity

• The difference is called
  disparity
• d is inversely related to Z greater sensitivity
  to nearby points
• d is directly related to b sensitivity to small
  baseline

8
Main Step Correspondence Search

• What to match?
• Objects?
• More identifiable, but difficult to compute
• Pixels?
• Easier to handle, but maybe ambiguous
• Edges?
• Collections of pixels (regions)?

9
Matching objects vs. Pixels
Left
Right
scanline
10
Random Dot Stereogram

• Using random dot pairs Julesz showed that
  recognition is not needed for stereo

11
Random Dot in Motion
12
Finding Matches

• Under what conditions pixels can be matched?
• Ignoring specularities, we can assume that
  matching pixels have the same brightness
  (constant brightness assumption)
• Still, changes in gain and sensitivity may change
  the values of pixels
• Common solution
• Use larger windows
• Normalized correlation
• Pros and cons
• Small window accurate match is more likely
• Large window fewer candidates
• We need a method to eliminate false matches

13
Window Size
W 3
W 20
14
Constraining the Search

• Restrict search to epipolar lines (1D search)
• Use larger elements (larger windows, edges,
  regions)
• Problem large elements may be distorted
• Enforce smoothness
• Problem discontinuities at object boundaries
• Enforce ordering
• Problem not always true

15
1D Search
16
1D Search

• More efficient
• Fewer false matches

SSD error
disparity
17
Ordering
18
Ordering
19
Correspondence as Optimization

• Most stereo algorithms attempt to minimize a
  functional that usually consists of two terms
• where
• - penalizes for quality of a match
  (unary)
• - penalizes non smooth (or even
  non fronto-parallel) reconstructions (binary)
• Many different optimization approaches were
  proposed

20
Comparison of Stereo Algorithms
Ground truth
Scene

• D. Scharstein and R. Szeliski. "A Taxonomy and
  Evaluation of Dense Two-Frame Stereo
  Correspondence Algorithms," International Journal
  of Computer Vision, 47 (2002), pp. 7-42.

21
Scharstein and Szeliski
22
Results with window correlation
Window-based matching (best window size)
Ground truth
23
Graph Cuts
Ground truth
Graph cuts
24
Stereo Algorithms

• Well briefly review several algorithms
• Dynamic programming
• Minimal cut/Max flow
• Space carving
• Graph cut optimization

25
?
26
1D Methods Dynamic Programming

• Discretize the 3-D space
• Find the correct curve at every slice
• (A slice epipolar plane)

27
Dynamic programming
Find correspondences of each epipolar line
separately















28
Dynamic programming































29
Dynamic programming

• How do we find the best curve?
• Assign weight of all edges

insertion
match
deletion
30
Dynamic programming

• How do we find the best curve?
• Assign weight of all edges
• Find shortest path
• Dijkstra

insertion
match
deletion
31
Results
32
Dynamic programming

• Advantages
• Simple, efficient
• Globally optimal
• Disadvantages
• Each slice computed independently (smoothness is
  not enforced between slices)
• Problems due to discretization (tilted planes)

33
Min Cut/Max Flow
34
Min Cut/Max Flow
35
Min Cut/Max Flow
36
Min Cut/Max Flow
37
Min Cut/Max Flow
Objective find the optimal cut using all the
slices simultaneously.
38
Min Cut/Max Flow

• Construct a graph
• Every voxel (3-D point in space) is a node
• Every node is connected to its 6 neighbors

39
Min Cut/Max Flow

• Weights on the edges
• Data cost change in pixel value

data
data
40
Min Cut/Max Flow

• Weights on the edges
• Data cost change in pixel value
• Smoothness cost change in depth

smooth
smooth
smooth
smooth
41
Min Cut/Max Flow

• Weights on the edges
• Data cost change in pixel value
• Smoothness cost change in depth

42
Min Cut/Max Flow
Source

• Add source and sink
• Find min cut

8
8
Sink
43
Min Cut/Max Flow
Data penalty Smoothness penalty
44
Results
Input Min cut
Dynamic programming
45
Min Cut/Max Flow

• Advantages
• All slices are optimized simultaneously
• Efficient
• Disadvantages
• Extension to multi-camera is difficult
• Discretization

46
Space Carving

• Multi-view stereo
• Every point in space corresponds to a match in
  the images
• Compute data term for each match

47
Space Carving

• Multi-view stereo
• Every point in space corresponds to a match in
  the images
• Compute data term for each match
  (photo-consistency)

0.2 0.3 0.9 0.8 0.9 0.9 0.8 0.4 0.5

48
Space Carving

• Dynamic data term (taking occlusion into account)
• Order of sweep is important

49
Space Carving









50
Space Carving

• Done for all slices simultaneously

51
Space Carving

• Done for all slices simultaneously

52
Space Carving

• Done for all slices simultaneously

53
Space Carving

• Computes a bound on the object, the visual hull
• More camera views better result

54
Space Carving Results
55
Space Carving Results
56
Space Carving

• Advantages
• True multi-views stereo
• Handles occlusion
• Disadvantages
• Limited to visual hull
• Lacks smoothness term
• Noise may introduce holes, allowing for noise may
  thicken shape
• Discretization

57
Graph Cut Optimization

• Stereo is a minimization problem
• Possible solution local search (gradient
  descent)
• Problem inefficient, local minima
• Instead, search larger areas at every iteration

58
Graph Cut Optimization

• Construct a graph to represent the problem
• Nodes
• Pixels (in first image)
• k discrete depth values
• Edges
• From every pixel node to a
• depth node (data term)
• Neighboring nodes (smoothness)
• Assign weights corresponding to pixel intensities
  to get a global cost function

depths
pixels
59
Graph Cut Optimization

• Objective Multiway cut
• Edges
• Every pixel remains connected to one depth node
• Edges between neighboring nodes only if they are
  connected to same depth node
• Nodes are assigned the depth that they are
  connected to
• Multiway cut is NP-complete, solve iteratively

1
2
k
3


depths
pixels
60
Graph Cut Optimization

• a-ß swap
• Nodes labeled a or ß, (i.e., connected to
  or )
• can change their labeling to a or ß
• Edges between neighbors are updated according to
  the new labeling
• Other edges are not changed
• Finding best swap min cut!

1
2
k
3
a
ß


61
Graph Cut Optimization
Example 1-2 swap


1
1
2
2
k
k
3
3




62
Graph Cut Optimization

• Example 1-2 swap

1
2
k
3
1
2
k
3


Connect the nodes labeled 1 or 2 to both labels
63
Graph Cut Optimization

• Example 1-2 swap

1
1
k
k
3
3
Mark 1 as source and 2 as sink
Find minimal cut
2
2
64
Graph Cut Optimization
Example 1-2 swap

1
k
3

1
2
k
3
Erase edges that were on the cut
Result a new labeling of the 1,2 nodes
2
65
Graph Cut Optimization

• Start with an arbitrary labeling
• For every pair a, ß ? 1,,k
• Find the best a-ß swap (minimizing the function)
• Update the graph (add and erase edges)
• Quit when no pair improves the cost function
• Induce pixel labels

66
Graph Cuts Results
67
Graph Cut Optimization

2
1
k
3

• Advantages
• State of the art results
• Efficient
• Bound on approximation quality
• Same technique can be applied to other problems
  (e.g., image restoration)
• Disadvantages
• Discretization
• Occlusion
• Still room for improvement

68
Summary

• Stereo vision shape reconstruction from two or
  more images
• Steps
• Rectification
• Correspondence search
• Depth estimation
• Algorithms
• Dynamic programming
• Min cut/max flow
• Space carving
• Graph cuts