Active Lighting for Appearance Decomposition

1
Active Lighting for Appearance Decomposition

• Todd Zickler
• DEAS, Harvard University

2
Appearance
3
Research Overview
COLOR IMAGE FILTERING
3D RECONSTRUCTION
APPEARANCE CAPTURE
PHOTOMETRIC INVARIANTS
4
Getting 3D ShapeImage-based Reconstruction
5
Reflectance BRDF
Bi-directional Reflectance Distribution Function
6
Conventional 3D ReconstructionRestrictive
Assumptions
7
Example Conventional Stereo
Il
Ir
ASSUMPTION Il Ir
8
Example Conventional Stereo
Il
Ir
ASSUMPTION Il Ir
9
Conventional 3D ReconstructionRestrictive
Assumptions
Variational Stereo Faugeras and Keriven, 1998
Shape from shading Tsai and Shaw, 1994
Multiple-window stereo Fusiello et al., 1997
Space Carving Kutulakos and Seitz, 1998
10
Reflectance BRDF
11
Reflectance BRDF
12
Helmholtz Reciprocity
Helmholtz 1925 Minnaert 1941 Nicodemus et al.
1977
13
Stereo vs. Helmholtz Stereo
STEREO
HELMHOLTZ STEREO
14
Stereo vs. Helmholtz Stereo
STEREO
HELMHOLTZ STEREO
15
Stereo vs. Helmholtz Stereo
STEREO
HELMHOLTZ STEREO
16
Reciprocal Images
17
Reciprocity Constraint
p
ol
or
18
Reciprocity Constraint
19
Reciprocal Acquisition
CAMERA
LIGHT SOURCE
20
Recovered Normals
Zickler et al. 2002
21
Recovered Surface
Zickler et al., ECCV 2002
22
In Practice

1. Arbitrary Reflectance
2. Off-the-shelf components
3. Direct surface normals
4. Images aligned with recovered shape
5. Self-calibrating (coming)

23
Ongoing Work Auto-calibration
Zickler et al., CVPR 2003, CVPR 2006,
24
Research Overview
COLOR IMAGE FILTERING
3D RECONSTRUCTION
APPEARANCE CAPTURE
PHOTOMETRIC INVARIANTS
25
Reflectance Decomposition
Phong 1975 Shafer, 1985
26
Reflectance Decomposition
Shafer, 1985
27
Reflectance Decomposition Simplifies the Vision
Problem


28
Reflectance Decomposition A Difficult Inverse
Problem
DIFFUSE
SPECULAR




Bajscy et al., 1996 Criminisi et al., 2005 Lee
and Bajscy, 1992 Lin et al., 2002 Lin and Shum,
2001 Miyazaki et al., 2003 Nayar et al., 1997
Ragheb and Hancock, 2001 Sato and Ikeutchi,
1994 Tan and Ikeutchi, 2005 Wolfe and Boult,
1991,
29
Known Illuminant Still Ill-posed
B
S
IRGB
D?
G
R
30
Known Illuminant Still Ill-posed
B
S
IRGB
D?
G
R
31
ObservationExplicit Decomposition not Required
B
S
IRGB
r1
G
r2
J

1. INVARIANT TOSPECULAR REFLECTIONS
2. BEHAVES LAMBERTIAN

R
32
ObservationExplicit Decomposition not Required
IRGB
J
Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005
33
Generalization Mixed Illumination
SINGLE ILLUMINANT
MIXED ILLUMINATION
B
B
S1
S
S2
IRGB
IRGB
r1
G
G
r2
J
r1
J
R
R
Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006
34
Generalization Mixed Illumination
35
Example Binocular Stereo
Conventional Grayscale(RGB)/3
Specular Invariant, J (blue illuminant)
Specular Invariant, J (blue yellow
illuminants)
One image from input stereo pair
Recovered depth
Algorithm Boykov, Veksler and Zabih, CVPR 1998
36
Example Optical Flow
Conventional Grayscale(R-GB)/3
Specular Invariant, J (blue illuminant)
Specular Invariant, J (blue yellow
illuminants)
Ground truth flow
Algorithm Black and Anandan, 1993
37
Example Photometric Stereo
J behaves Lambertian ? Linear function of
surface normal
Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005
38
Example Photometric Stereo
J behaves Lambertian ? Linear function of
surface normal
Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005
39
Example Photometric Stereo
J behaves Lambertian ? Linear function of
surface normal
Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005
40
Example Photometric Stereo
Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005
41
Example Photometric Stereo
Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005
42
Generalized Hue
B
S
IRGB
r1
G
r2
J
R
43
Example Material-based Segmentation
Conventional Grayscale
Specular Invariant J
Input image
Generalized Hue y
Conventional Hue
Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006
44
Active Lighting for Image-guided Surgery?

• Active lighting can provide
• Precise shape (surface normals) for a broad class
  of (non-Lambertian) surfaces
• Specular and/or shading invariance (e.g.,
  optical flow, tracking, segmentation)
• Minimal hardware requirements

• Endoscopic imagery
• Illuminant(s) is/are controlled and known
• Non-Lambertian surfaces
• Lack of texture

45
Acknowledgements
Satya Mallick, UCSD Peter Belhumeur, Columbia
University David Kriegman, UCSD Sebastian
Enrique, Columbia University Ravi Ramamoorthi,
Columbia University
zickler_at_eecs.harvard.edu http//www.eecs.harvard.e
du/zickler