Defocus Magnification

1
Defocus Magnification
Soonmin Bae and Frédo Durand MIT CSAIL
2
SLR vs. Point-and-Shoot
SLR camera
Point-and-Shoot camera
3
Shallow Depth of Field
Sharp foreground with blurred background
4
A Point-and-Shoot Camera
Background is not blurred enough
5
Defocus

• Point in focus rays converge to sensor
• Farther points are blurrier

sensor
lens
focal plane
6
Defocus and Aperture size

• Bigger aperture produces more defocus
• F-number N gives the aperture diameter A as a
  fraction of the focal length f (A Nf )
• Example f 100 mm, f/2 A 50mm, f/4 A 25mm

f/2
f/4
sensor
lens
focal plane
7
Defocus and Aperture size

• Aperture the size of the lens opening
• Wide aperture shallow depth of field
• Pinhole aperture infinite depth of field

Narrow aperture (f/32) background remained sharp
Wide aperture (f/2) blurred background
8
Defocus and Sensor size

• Sensor size
• Small sensor ? small lens ? less defocus
• Defocus size is mostly proportional to the sensor
  size(see paper)

Small sensor (7.18 x 5.32), f/2.8 background
remained sharp
Large sensor (22.2 x 14.8), f/2.8 blurred
background
9
Defocus and Sensor size

• Sensor size
• Small sensor ? small lens ? less defocus
• Defocus size is mostly proportional to the sensor
  size(see paper)

Small sensor (7.18 x 5.32), f/2.8 background
remained sharp
Large sensor (22.2 x 14.8), f/2.8 blurred
background
10
Goals

• Magnify defocus given a single image
• Blur blurry regions and keep sharp regions sharp
• Simulate shallow depth of field

input
magnified defocusing result
11
Overview
12
We do not require

• precise depth estimation
• disambiguation b/w out-of-focus edges and
  originally smooth edges

• But we simply compute the amount of blur and
  increase it

13
Related Work Depth from De/focus

• Seeks the exact depth map
• Horn 68 Pentland 87 Darrell 88 Ens 93 Nayar
  94 Watanabe 98 Favaro 02 Jin 02 Favaro 05
  Hasinoff 06
• is a hard problem
• needs multiple images with different focus
  settings
• In contrast, we want to estimate the blur
  kernel, not the depth
• In addition, we analyze the blur kernel only at
  edges

near-focused
far-focused
depth from defocus
Durand 02
Watanabe 98
14
Related Work - remove blur from images

• Blind deconvolution Reeves 92 Trussell 92
  Özkan 94, Fergus 06
• Extended depth of field using multiple images
• Adelson 83 Eltoukhy 03 Agarwala 04 Kubota 05
• In contrast, we want to increase defocus
• In addition, we use a single image to estimate
  spatially variant blurs

Kubota 98
15
Related Work - computational camera

• Defocus manipulation
• Ng 05, Green 07, Levin 07, Moreno-Noguer 07,
  Veeraraghavan 07
• In contrast, we want to use image-processing
  techniques

Levin 07
16
Related Work Synthetic Lens Blur

• Given image depth map,simulate defocus
• Potmesil 81
• e.g. Adobe Photoshop and
  
  Depth of Field Generator Pro

We use Photoshop Lens Blur to generate results
with our defocus map instead of a depth map
17
Our work

• Measure blurriness
• Estimate the spatially-varying amount of blur at
  edges
• Propagate blurriness (defocus map)
• Assume that blurriness is smooth except at image
  edges
• Blur the blurry regions
• Use Photoshop lens blur

input
defocus map
blur (defocus) measure
result
18
Our work

• Measure blurriness
• Estimate the spatially-varying amount of blur at
  edges
• Propagate blurriness (defocus map)
• Assume that blurriness is smooth except at image
  edges
• Blur the blurry regions
• Use Photoshop lens blur

input
defocus map
blur (defocus) measure
result
19
Blur Estimation at Edges

• an edge
• a step function in intensity
• the blur of this edge
• a Gaussian blurring kernel

20
Blur Estimation at Edges

• an edge
• a step function in intensity
• the blur of this edge
• a Gaussian blurring kernel

edge
gaussian blur
blurred edge

• Multiscale edge detector
• Blur estimation at edges

21
Blur Estimation at Edges Elder 98

• Multiscale edge detector
• output a sparse set of pixels

• Blur amount sb at edges
• related to the distance between extrema of the
  second derivative
• Elder and Zucker use the zero-cossing of the
  third derivative

2nd derivative
edge
gradient
22
Distance between Zero-crossing of the Third
Derivative Elder 98

• works in simple cases (i.e. a single line, no
  texture)

Y axis
input
Y axis
23
Blur Estimation at Edges

• Fit response models of various sizes

less blurry
edge
more blurry
response model
24
Robust Blur Estimation

• Successfully measure the blur size in spite of
  the influence of scene events nearby

blurry
sharp
25
Our blur measure

• A sparse set
• values only at edges
• Grey means no value

blurry
input
blur measure
sharp
26
Refinement of Blur Estimation

• Erroneous blur estimates
• due to soft shadows and glossy highlights

blurry
input
blur measure
sharp
27
Refinement of Blur Estimation

• Erroneous blur estimates
• due to soft shadows and glossy highlights

blurry
blur measure
input
sharp
28
Remove Outliers

• Using cross bilateral filtering Eisemann 04,
  Petschnigg 04
• a weighted mean of neighboring blur measures
• Smoothes the blur measure near in spatial
  distance and close in range difference of a
  reference image

blurry
before refinement
after refinement
sharp
29
Our work

• Measure blurriness
• Estimate the spatially-varying amount of blur at
  edges
• Propagate blurriness (defocus map)
• Assume that blurriness is smooth except at image
  edges
• Blur the blurry regions
• Use Photoshop lens blur

input
defocus map
blur (defocus) measure
result
30
Blur Propagation

• Given a sparse set of the blur measure (BM)
• Propagate the blur measure to the entire image
• Assumption blurriness (B) is smooth except at
  image edges
• Inspired by Levin et al. 2004

31
Blur Propagation

• Given a sparse set of the blur measure (BM)
• Propagate the blur measure to the entire image
• Assumption blurriness (B) is smooth except at
  image edges

• We minimize

data term
smoothness term
proporsional to e - C(p) C(q) 2
32
Blur Propagation

• Edge-preserving propagation
• propagation stops at input edges

33
Our work

• Measure blurriness
• Estimate the spatially-varying amount of blur at
  edges
• Propagate blurriness (defocus map)
• Assume that blurriness is smooth except at image
  edges
• Blur the blurry regions
• Use Photoshop lens blur

input
defocus map
blur (defocus) measure
result
34
Recap
35
Input
Result
Defocus Map
36
Input
Result
37
Input
Result
Defocus Map
38
Input
Result
39
Input
Result
Defocus Map
40
Input
Result
41
Comparison with the ground truth
42
Summary

• Analyze existing defocus
• multiscale edge detector fitting
• non-homogeneous propagation
• Magnify defocus

43
Preliminary Refocusing Result

• Synthesize refocusing effects
• Perform deconvolution using our defocus map

44
Contributions

• Our defocus map captures blurriness
• Our defocus map can be used to increase defocus

45
Acknowledgement

• MIT Computer Graphics Group
• anonymous reviewers
• NSF/ Royal Dutch / Shell Group
• a Microsoft Research New Faculty Fellowship, a
  Sloan Fellowship, and a Jamieson chair
• Samsung Scholarship

46
Depth of Field and Aperture size

• Aperture the size of the lens opening
• Wide aperture shallow depth of field
• Pinhole aperture infinite depth of field

sensor
lens
focus distance
47
Blur Propagation

• Given a sparse set of the blur measure (BM)
• Propagate the blur measure to the entire image
• Assumption blurriness (B) is smooth except at
  image edges
• We minimize

smoothness term
data term
48
Input
Result
Defocus Map
49
Input
Result
Defocus Map
50
Preliminary Refocusing Result

• Synthesize refocusing effects
• Perform deconvolution using our defocus map

51
Preliminary Refocusing Result

• Synthesize refocusing effects
• Perform deconvolution using our defocus map

52
Blur Estimation at Edges

• an edge a step function in intensity
• the blur of this edge a Gaussian blurring
  kernel

53
Blur Estimation at Edges Elder 98

• Multiscale edge detector
• Output a sparse set of pixels
• blur amount sb at edges
• related to the distance between zero-crossings of
  the third derivative, and
  

2nd derivative
edge
54
Depth of Field

• Focused rays converge enough
• Depth of field the range of distance around the
  focal plane which produces acceptably small
  circles

sensor
lens
focal plane
55
Defocus and Aperture size

• Bigger aperture produces more defocus
• F-number N gives the aperture diameter A as a
  fraction of the focal length f (A Nf )
• Example f 100 mm, f/2 A 50mm, f/4 A 25mm

f/2
f/4
sensor
lens
focal plane