Amit Agrawal, Ramesh Raskar,

1
Removing Photography Artifacts using Gradient
Projection and Flash-Exposure Sampling

• Amit Agrawal, Ramesh Raskar,
• Shree K. Nayar and Yuanzhen Li
• Mitsubishi Electric Research Labs (MERL),
  Cambridge, MA
• Columbia University, NY

2
Photography Artifacts Flash Hotspot
Ambient
Flash
Flash Hotspot
3
Reflections due to Flash
Underexposed
Reflections
Ambient
Flash
4
Distance Dependance
Flash
Distant people underexposed
5
Self-Reflections and Flash Hotspot
Ambient
Flash
Face
Hands
Tripod
6
Ambient
Result
Reflection Layer
Flash
Face
Hands
Tripod
7
Combining Flash/No-Flash Images

• Previous Work
• Eisemann Durand 04, Petschnigg et. al. 04
• De-noise low quality ambient image
• Transfer detail/color
• Current work
• Remove artifacts such as reflections

8
Intensity Gradient Vector Projection
9
Intensity Gradient Vectors in Flash and Ambient
Images
Same gradient vector direction
Flash Gradient Vector
Ambient Gradient Vector
Ambient
Flash
No reflections
10
Reflection Ambient Gradient Vector
Different gradient vector direction
Flash Gradient Vector
Ambient
Flash
With reflections
11
Intensity Gradient Vector Projection
Reflection Ambient Gradient Vector
Residual Gradient Vector
Flash Gradient Vector
Result Gradient Vector
Result
Residual
Ambient
Flash
12
Residual Reflection Layer
Projection Result
Flash
Ambient
Co-located Artifacts
13
Flash
Ambient
Checkerboard outside glass window
Reflections on glass window
14
Forward Differences
Checkerboard
Gradient Difference
Checkerboard
removed
Flash
2D Integration
2D Integration
Result
Reflection Layer
Result
Ambient
15
Invariance of Gradient Vectors Orientation
(Gradient Orientation Coherency)
Ambient
Flash
?

• Reflectance Edge
• ??? Geometric Edge
• Illumination Edge

16
Related Work

• Reflection Removal
• Nayar et. al. 96, Schechner00 (polarization),
  Farid Adelson99 (ICA), Levin04
• Gradient Based Techniques
• HDR compression (Fattal02), Poisson Image
  editing (Perez03), Poisson matting (Sun04),
  Image fusion (Raskar04), Intrinsic Images
  (Weiss01), Multi-flash Camera (Raskar04) etc.
• 2D Integration
• Frankot Chellappa88, Poisson solver
  (Multi-grid, Sine Transform)
• Agrawal et. al. 05

17
Removing Reflections due to Flash
Ambient
Flash
Ambient Flash
Result
Reflections
removed
18
Removing Flash Hotspot
Ambient
Flash
Result
Saturation Weights Ws
Linearly combine flash and ambient image
gradients using Ws and Gradient Coherency
19
Depth Compensation
Flash
Ambient
Result
Distant Persons
Scale flash gradients using the ratio of flash
and ambient images
20
Flash reflections of textured cloth outside
window
Ambient
Mannequin behind glass window
High contrast but reflections
21
Removing Reflections due to Flash
Ambient
Flash
Result
X Homogeneous Regions
22
Limitations

• Difficult Scenarios
• Dynamic scenes
• Co-located artifacts
• Strong ambient illumination edges
• Issues
• Lack of reliable gradients
• Homogeneous or dark regions
• Color coherency

23
Combining Flash/No-flash Images for High Dynamic
Range (HDR) Imaging
24
Need Both Ambient and Flash!!
HDR Scene
Flash
Ambient
Well-lit in Ambient
Underexposed
Well-lit in Flash
25
Exposure HDRVarying Exposure Time

• Mann and Picard, 95
• Debevec and Malik, 97
• Nayar and Mitsunaga, 00

1/100
1/20
1/5
1
4
1/250
Exposure Time
26
Underexposed to well-exposed
Flash Brightness
Well-exposed to saturation
Flash HDRVarying Flash Brightness
Scene distance dependence
27
Flash-Exposure Sampling
Flash-Exposure HDRVarying both
Flash Brightness
Exposure Time
28
Capturing HDR Image
Varying Exposure time
Varying Flash brightness
Varying both
29
Do We Need All Images ?
Flash Brightness
Exposure Time
Regular 2D Sampling 24 Pictures
Adaptive Sampling 5 pictures

• Next Best Picture ?

• Based on all previous pictures
• Maximize well-lit pixels over the image
• Exclude pixels already captured as well-exposed

30
Exposure Time
Underexposed
Still Underexposed
Well-exposed
?
HDR Image using N images
HDR Image using N1 images
?
?
Flash Brightness
N1th picture ?
31
Summary
Reflection Layer
F

• Intensity Gradient Vector Projection for Removing
  Reflections
• Flash-Exposure HDR Image
• Adaptive Flash-Exposure Sampling

A
Result
Flash Brightness
Exposure Time
32
Acknowledgements

• MERL members
• Joe Marks, MERL
• Rama Chellappa, UMD
• Ted Adelson, MIT
• Karhan Tan, Epson
• Siggraph Reviewers

33

• Flash-Exposure Sampling

Gradient Vector Projection for Removing
Reflections
Adaptive Sampling for HDR
?
Flash Brightness
?
Exposure Time
34
Extra Slides
35
Rotating Intensity Gradient Vectors
36
Projection vs. Rotation
F
A
Projection
Rotation
37
Projection vs. Rotation
Ambient
Flash
Projection
Rotation
38
Flash patch
Gradient Vector Projection
Joint Bilateral filtering
39
Gradient Direction Reversal at Depth Edges
Ambient
Flash
Geometric Edge
a
40
Result
Ambient
Flash
High magnitude gradients ? gt p/2 (cos ?lt0)
41
X
Y
Flash
X
Gradient Vector Projection
Y
Result
X
Y
Ambient
42
Not accounting for direction reversal
Accounting for direction reversal
cos ? at high gradients
43
HDR using N images
HDR using correct N1th image
HDR using incorrect N1th image
44
Extra Slides
45
Ambient
Gradient Vector Projection for removing
reflections
Flash
46
Reflection Layer
Flash
Ambient with Reflections
Residual Reflection
Result
Reflection Free Result
47
Future Directions

• Incorporation of global information for
  reflection removal
• Better schemes for adaptive sampling
• Improved gradient reconstruction techniques
• Recovering intrinsic scene properties

48
Intensity Gradients based Artifact Removal

• Flash Image F, Ambient Image ?

• Gradient Coherency Model
• Illumination invariants, Chen, CVPR 00
• Invariant directions, Finlayson ECCV04

Gradient Coherency Map