Course 3: Computational Photography

1
Course 3 Computational Photography
Ramesh Raskar, Mitsubishi Electric ResearchJack
Tumblin, Northwestern U.
Course WebPagehttp//www.merl.com/people/raskar/p
hoto
Course Evaluationhttp//www.siggraph.org/courses_
evaluation
Send us your computational photos !
2
Course 3 Computational PhotographyE
Reconstruction
Ramesh Raskar Mitsubishi Electric Research
Labs Jack Tumblin Northwestern University
Course WebPage http//www.merl.com/people/raska
r/photo
3
Schedule
830 Introduction (Raskar) 840
Photographic Signal Film-like Photography
(Tumblin) 910 Image Processing Tools
(Raskar) 940 Improving Film-like
Photography (Tumblin) 1015 Break 1030 Image
Reconstruction Techniques (Raskar) 1115 Smart
Lights and Beyond Photography (Tumblin) 1145
Smart Optics and Sensors (Raskar) 1205
Discussion
Course Page http//www.merl.com/people/raskar/ph
oto
4
Course 3 Computational Photography
Course WebPage http//www.merl.com/people/raskar
/photo
Course Evaluation http//www.siggraph.org
5
Welcome

• Understanding Film-like Photography
• Parameters, Nonlinearities, Ray-based concepts
• Image Processing and Reconstruction Tools
• Multi-image Fusion, Gradient domain, Graph Cuts
• Improving Camera Performance
• Better dynamic range, focus, frame rate,
  resolution
• Future Directions
• HDR cameras, Gradient sensing, Smart
  optics/lighting

6
Goals

• Review of 30 recent papers
• Understand computational aspects of cameras
• Discuss issues in traditional cameras
• Explore alternative imaging methods
• Learn Vision and Optics techniques
• Discuss image processing and reconstruction tools
• What we will not cover
• Film Cameras, Novel view rendering (IBR), Color
  issues, Traditional image processing/editing

7
Image Fusion and Reconstruction

• Epsilon Photography
• Vary time, view
• Vary focus, exposure polarization, illumination
• Better than any one photo
• Achieve effects via multi-image fusion
• Understand computer vision methods
• Exploit lighting

8
Time-Lapse

• Duchamp
• Nude Descending a Staircase

9
Time-Lapse

• Richard Hundley 2001

10
Shape Time Photography
Freeman et al 2003
11
Varying Focus Extended depth-of-field
Agrawala et al, Digital Photomontage, Siggraph
2004
12
Source images
Computed labeling
Composite
13
Computer Vision Techniques

• Photometric Stereo
• Varying light source positions
• Estimate surface normal from shading
• Diffuse objects minimum 3 lights
• Depth from Defocus
• Varying focus
• Defogging
• Varying time and polarization

14
Varying Focus Depth from Defocus
(Nayar, Watanabe and Noguchi, 95 )
image detectors
lens
scene
P
f
Q
i
o
near focus
aperture
Previous Work Pentland 87, Subbarao 88, Nayar
89.
15
Varying Focus Depth from Defocus
(Nayar, Watanabe and Noguchi, 95 )
image detectors
lens
scene
P
f
Q
i
o
far focus
aperture
Previous Work Pentland 87, Subbarao 88, Nayar
89.
16
Real Time Defocus Depth Camera (Movies)
(Nayar , Watanabe , Noguchi 95 )
Performance 512 x 480 Depth map at 30 frames
per sec.
17
Clear Day from Foggy Days
(Shree Nayar, Srinivasa Narasimhan 00)
Two Different Foggy Conditions
Time 3 PM
Time 530 PM
18
Varying PolarizationYoav Y. Schechner, Nir
Karpel 2005
Best polarization state
Worst polarization state
Best polarization state
Recovered image
Left The raw images taken through a polarizer.
Right White-balanced results The recovered
image is much clearer, especially at distant
objects, than the raw image
19
Varying Polarization

• Schechner, Narasimhan, Nayar
• Instant dehazing of images using polarization

20
Varying Wavelength Multispectral Fusion
Vegetation Mapping of the Forest


SAR
Optical Landsat
21
Varying IR Wavelength Image Fusion
SWIR
NIR
LWIR
Uniform fusion across image
Adaptive fusion by sub region
22
Non-photorealistic Camera Depth Edge Detection
and Stylized Rendering using Multi-Flash Imaging

• Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi
  Yu, Matthew Turk
• Mitsubishi Electric Research Labs (MERL),
  Cambridge, MA
• U of California at Santa Barbara
• U of North Carolina at Chapel Hill

23
(No Transcript)
24
Car Manuals
25
What are the problems with real photo in
conveying information ?
Why do we hire artists to draw what can be
photographed ?
26
Shadows Clutter Many Colors
Highlight Shape Edges Mark moving parts Basic
colors
27
A New Problem
Shadows Clutter Many Colors
Highlight Edges Mark moving parts Basic colors
28
Depth Edge Camera
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
Depth Discontinuities
Internal and externalShape boundaries, Occluding
contour, Silhouettes
34
Depth Edges
35
Our Method
Canny
36
Result
Photo
Canny Intensity Edge Detection
Our Method
37
(No Transcript)
38
Our Method
Canny Intensity Edge Detection
39
(No Transcript)
40
(No Transcript)
41
Computational Illumination
42
Synthetic LightingPaul Haeberli, Jan 1992
43
A Night Time Scene Objects are Difficult to
Understand due to Lack of Context
Dark Bldgs
Reflections on bldgs
Unknown shapes
44
Enhanced Context All features from night scene
are preserved, but background in clear
Well-lit Bldgs
Reflections in bldgs windows
Tree, Street shapes
45
Night Image
Background is captured from day-time scene using
the same fixed camera
Result Enhanced Image
Day Image
46
Mask is automatically computed from scene
contrast
47
But, Simple Pixel Blending Creates Ugly
Artifacts
48
Pixel Blending
Our MethodIntegration of blended Gradients
49
Denoising Challenging Images

• Available light
• nice lighting
• noise/blurriness
• color

50

• Flash
• details
• color
• flat/artificial

Flash
51
Elmar Eisemann and Frédo Durand , Flash
Photography Enhancement via Intrinsic
RelightingGeorg Petschnigg, Maneesh Agrawala,
Hugues Hoppe, Richard Szeliski, Michael Cohen,
Kentaro Toyama. Digital Photography with Flash
and No-Flash Image Pairs

• Use no-flash image relight flash image

52
Introduction

• Our approach
• Use no-flash image relight flash image

original lighting details/sharpness color
53
Cross-Bilateral Filter based Approach
54
Cross Bilateral Filter

• Similar to joint bilateral filter by Petschnigg
  et al.
• When no-flash image is too noisy
• Borrow similarity from flash image
• edge stopping from flash image
• See detail in paper

Bilateral
Cross Bilateral
55
Detail Layer

Intensity
Large-scale
Recombination Large scale Detail Intensity
56
(No Transcript)
57
Flash and Ambient Images Agrawal, Raskar,
Nayar, Li Siggraph05
Result
Reflection Layer
Flash
Ambient
58
Intensity Gradient Vector Projection
59
Intensity Gradient Vectors in Flash and Ambient
Images
Same gradient vector direction
Flash Gradient Vector
Ambient Gradient Vector
Ambient
Flash
No reflections
60
Reflection Ambient Gradient Vector
Different gradient vector direction
Flash Gradient Vector
Ambient
Flash
With reflections
61
Reflection Ambient Gradient Vector
Intensity Gradient Vector Projection
Residual Gradient Vector
Flash Gradient Vector
Result Gradient Vector
Ambient
Flash
Result
Residual
62
Residual Reflection Layer
Projection Result
Flash
Ambient
Co-located Artifacts
63
Flash
Ambient
Checkerboard outside glass window
Reflections on glass window
64
Forward Differences
Checkerboard
Gradient Difference
Checkerboard
removed
Flash
2D Integration
2D Integration
Result
Reflection Layer
Result
Ambient
65
Need flash component!
Flash
Ambient
66
Build Exposure HDR image

• Multiple images with different exposure
• Debevec Malik, Siggraph 97
• Nayar Mitsunaga, CVPR 00

Increasing Exposure
67
Build Flash HDR image
Flash Intensity
68
Flash-Exposure Sampling
Build Flash-Exposure HDR image
Flash Intensity
Exposure
69
Exposure HDR image
Flash HDR image
Flash-Exposure HDR image
70
Image Fusion and Reconstruction

• Epsilon Photography
• Vary focus, exposure polarization, illumination
• Vary time, view
• Better than any one photo
• Achieve effects via multi-image fusion
• Understand computer vision methods
• Exploit lighting

71
Schedule
830 Introduction (Raskar) 840
Photographic Signal Film-like Photography
(Tumblin) 910 Image Processing Tools
(Raskar) 940 Improving Film-like
Photography (Tumblin) 1015 Break 1030 Image
Reconstruction Techniques (Raskar) 1115 Smart
Lights and Beyond Photography (Tumblin) 1145
Smart Optics and Sensors (Raskar) 1205
Discussion
Course Page http//www.merl.com/people/raskar/ph
oto