Discontinuity Preserving Stereo with Small Baseline Multi-Flash Illumination

1
Discontinuity Preserving Stereo with Small
Baseline Multi-Flash Illumination

• Rogerio Feris1, Ramesh Raskar2, Longbin Chen1,
  Karhan Tan3 and Matthew Turk1

1University of California, Santa
Barbara 2Mitsubishi Electric Research Labs 3Epson
Palo Alto Lab
2
Introduction

• Correspondence Problem
• Stereo Near Depth Discontinuities
• Occlusion Problem
• Perspective Distortions
• Violation of Smoothness Constraints
• Passive Versus Active Methods

3
Introduction

• Our Approach
• Small Baseline Multi-Flash Illumination
• Simple, Inexpensive
• Compact, Self-Contained
• Discontinuity Preserving

4
Depth Edges with Multi-Flash

• Raskar, Tan, Feris, Yu, Turk ACM SIGGRAPH 2004

5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
Shadow-Free
Bottom Flash Top Flash Left Flash
Right Flash
Ratio images and directions of epipolar traversal
Depth Edges
10
Qualitative Depth Map
11
Qualitative Depth

• Sign of Depth Edge
• Indicates which side is the foreground and which
  side is the background
• Shadow Width
• - Encodes object relative distances

12
Sign of Depth Edge
Original Ratio Left
Ratio Right Signed Edges
-
-


() Foreground (-) Background
13
Shadow Width
Bottom Flash Image
Ratio Image
Plot Along Scanline
14
Shadow Width
Bottom Flash Image
Ratio Image
Shadow Width Estimation Meanshift
Segmentation algorithm applied on the ratio image
15
Imaging Geometry
Camera
Flash
B
f
d
Shadow Width
z1
Object
z2
Shadow
16
Qualitative Depth

• Working on this Equation

Log Depth Difference
Shadow Width
17
Qualitative Depth

• 1) Compute Sharp Depth Gradient G (Gh,Gv)

Sign of depth edge
Log Depth Difference
2) Compute Q by integrating G (Poisson Equation)
3) Qualitative depth map Q exp(Q)
18
Qualitative Depth

• Useful Prior Information for Stereo !

19
Occlusion Map
20
Partial Occlusion Problem
Occlusion
(Seen by A but not by B)
Object
Camera
A
B
21
Occlusion Bounded by Shadows
Occlusion
(Seen by A but not by B)
Object
Flash
Camera
A
B
22
Occlusion Bounded by Shadows
Lower Bound Shadow
Object
Flash
Camera
A
B
23
Occlusion Bounded by Shadows
Upper Bound Shadow
Object
Flash
Camera
A
B
24
Occlusion Bounded by Shadows
Occlusion
? Average of Upper/Lower Shadow widths
Object
Flash
Camera
A
B
25
Occlusion Bounded by Shadows
Left View
Right View
Occlusion Map
26
Discontinuity Preserving Stereo Matching
27
Local Stereo
Problem Shape and size of correlation window -
Small Window ? Ambiguities / Noise - Large Window
? Problems at Depth Discontinuities
Depth Edge Preserving Local Stereo
28
Local Stereo
Problem Shape and size of correlation window -
Small Window ? Ambiguities / Noise - Large Window
? Problems at Depth Discontinuities
Depth Edge Preserving Local Stereo
Correlation Window
Smooth Disparity Delimited by depth edges
Occlusions
29
Local Stereo
Challenging Scene - Ambiguous patterns,
textureless regions, geometrically complex
object, thin structures
Depth Edges Occlusion
Ground Truth
Left View
30
Local Stereo
Conventional 31x31
Conventional 9x9
Our Approach 31x31
31
Global Stereo
Global Optimization Markov Random Field
(MAP-MRF)
X xs Disparity of each pixel (Hidden) Y
ys Matching cost at each disparity (Observed)
X4
X6
y1
y2
X3
X1
X2
X7
X5
X8
32
Global Stereo
Global Optimization Markov Random Field
(MAP-MRF)
X xs Disparity of each pixel (Hidden) Y
ys Matching cost at each disparity (Observed)
Smoothness Term
Data Term
Inference by Belief Propagation Jian Sun et al,
2003
33
Global Stereo

• Qualitative Depth Map as Evidence
• - Used to set the smoothness term
• Information propagation is stopped at depth
  edges
• Encourage disparities for neighboring pixels
  according to depth difference in qualitative map
• Occlusion Penalty

34
Global Stereo
Conventional Belief Propagation Our
Approach
RMS 0.9589
RMS 0.4590
35
Conclusions

• Contributions
• - Stereo with small baseline illumination
• Useful Feature Maps (Qualitative Depth
  Occlusion Map)
• Enhanced Local and Global Stereo Algorithms
• Pros / Cons
• Robust, Simple, Inexpensive and Compact
• Limited to handle outdoor scenes and motion
• Website (datasets, source code)
• - http//www.cs.ucsb.edu/rferis/multi-flash-stere
  o

36
Thank you ! Multi-Flash Stereo
Webpage http//www.cs.ucsb.edu/rferis/multi-flash
-stereo Four Eyes Lab, UCSB http//ilab.cs.ucsb.e
du
37
(No Transcript)
38
Occlusion Bounded by Shadows
Images taken with light sources surrounding the
other camera
Occlusion Detection by averaging length of shadows