Removing Camera Shake from a Single Photograph

1
Removing Camera Shake from a Single Photograph
Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T.
Roweis and William T. Freeman
Massachusetts Institute of Technology
andUniversity of Toronto
2
Overview
Our algorithm
Original
3
Close-up
Original
Naïve Sharpening
Our algorithm
4
Lets take a photo
Blurry result
5
Slow-motion replay
6
Slow-motion replay
Motion of camera
7
Image formation process
?

Blur kernel
Blurry image
Sharp image
Input to algorithm
Desired output
Convolutionoperator
Model is approximation
8
Why is this hard?
Simple analogy 11 is the product of two
numbers. What are they?
No unique solution 11 1 x 11 11 2 x
5.5 11 3 x 3.667 etc..
Need more information !!!!
9
Multiple possible solutions
Sharp image
Blur kernel

?
Blurry image
10
Natural image statistics
Characteristic distribution with heavy tails
Histogram of image gradients
11
Blury images have different statistics
Histogram of image gradients
12
Parametric distribution
Histogram of image gradients
Use parametric model of sharp image statistics
13
Uses of natural image statistics

• Denoising Roth and Black 2005
• Superresolution Tappen et al. 2005
• Intrinsic images Weiss 2001
• Inpainting Levin et al. 2003
• Reflections Levin and Weiss 2004
• Video matting Apostoloff Fitzgibbon 2005
• Corruption process assumed known

14
Existing work on image deblurring

• Software algorithms
• Extensive literature in signal processing
  community

• Mainly Fourier and/or Wavelet based
• Strong assumptions about blur ? not true for
  camera shake

• Image constraints are frequency-domain power-laws

15
Hardware approaches
Existing work on image deblurring
Coded shutter
Dual cameras
Image stabilizers
Raskar et al. SIGGRAPH 2006
Ben-Ezra and Nayar 2004
Our approach can be combined with these hardware
methods
16
Three sources of information

• 1. Reconstruction constraint

2. Image prior
Distribution of gradients
17
How do we use this information?

• Obvious thing to do
• Combine 3 terms into an objective function
• Run conjugate gradient descent
• This is Maximum a-Posteriori (MAP)

18
Results from MAP estimation
Input blurry image
Maximum a-Posteriori (MAP)
Our method Variational Bayes
19
Variational Bayesian method

• Based on work of Miskin Mackay 2000
• Keeps track of uncertainty in estimates of image
  and blur by using a distribution instead of a
  single estimate
• Helps avoid local maxima and over-fitting

20
Variational Bayesian method
Objective function for a single variable
Maximum a-Posteriori (MAP)
Variational Bayes
Score
Pixel intensity
21
Overview of algorithm
Input image

• Pre-processing
• Kernel estimation
• Multi-scale approach
• Image reconstruction
• - Standard non-blind deconvolution routine

22
Preprocessing
Input image
Convert tograyscale
Remove gammacorrection
User selects patch from image

• Bayesian inference too slow to run on whole
  image

Infer kernel from this patch
23
Initialization
Input image
Convert tograyscale
Remove gammacorrection
User selects patch from image
Initialize 3x3 blur kernel
Initial image estimate
Initial blur kernel
Blurry patch
24
Inferring the kernel multiscale method
Input image
Convert tograyscale
Remove gammacorrection
User selects patch from image
Loop over scales
VariationalBayes
Upsampleestimates
Initialize 3x3 blur kernel
Use multi-scale approach to avoid local minima
25
Image Reconstruction
Input image
Convert tograyscale
Remove gammacorrection
User selects patch from image
Loop over scales
VariationalBayes
Upsampleestimates
Initialize 3x3 blur kernel
Full resolutionblur estimate
Non-blind deconvolution (Richardson-Lucy)
Deblurred image
26
Results on real images

• Submitted by people from their own photo
  collections
• Type of camera unknown
• Output does contain artifacts
• Increased noise
• Ringing
• Compares well to existing methods

27
Original photograph
28
Our output
Blur kernel
29
Original photograph
Matlabs deconvblind
30
Close-up of garland
Original
Matlabs deconvblind
Our output
31
Original photograph
32
Matlabs deconvblind
33
Photoshop sharpen more
34
Our output
Blur kernel
35
(No Transcript)
36
Original photograph
37
Our output
Blur kernel
38
Original photograph
39
Our output
Blur kernel
40
Matlabs deconvblind
41
Original photograph
42
Our output
Blur kernel
43
Close-up of child
Our output
Original photograph
44
Original photograph
45
Our output
Blur kernel
46
Close-up of bird
Original
Unsharp mask
Our output
47
Original photograph
48
Blur kernel
Our output
49
Image artifacts estimated kernels
Blur kernels
Image patterns
Note blur kernels were inferred from large image
patches, NOT the image patterns shown
50
Summary

• Method for removing camera shake from real
  photographs
• First method that can handle complicated blur
  kernels
• Uses natural image statistics
• Non-blind deconvolution currently simplistic
• Things we have yet to model
• Correlations in colors, scales, kernel continuity
• JPEG noise, saturation, object motion

51
Acknowledgements

• James Miskin David Mackay for putting their
  code on the web
• Antonio Torralba, Fredo Durand and Berthold
  Horn for their insights and suggestions.
• People who submitted their home photos
• Omar Khan, Reinhard Klette, Michael Lewicki,
  Pietro Perona, Lyndsey Pickup, Mukta Prasad, Ian
  Reid and Elizabeth van Ruitenbeek

52
nts
53
Blurry synthetic image
54
Our output
55
Ground truth
56
Matlabs deconvblind
57
True kernel
Estimated kernel
58
Does real camera shake give a spatially-invariant
blur kernel?
Subjects handholding DSLR with 1 sec exposure
Close-up of dots
59
Does real camera shake give a spatially-invariant
blur kernel?
Subjects handholding DSLR with 1 sec exposure
60
MAP vs Variational
MAP
Variational
MAP using variational initialization
61
Histograms of café scene
62
Close-up view of the dots at each corner of
photos taken
Person 1
Person 2
Topleft
Topright
Bot.left
Bot.right
Person 3
Person 4
63
Uses of natural image statistics

• Denoising Roth and Black 2005

64
Uses of natural image statistics

• Superresolution Tappen et al. 2005

65
Comparison of non-blind deconvolution methods

1. Variational inference hold b fixed and
   marginalize out over it
2. Conjugate gradient descent on MAP solution (with
   field of experts prior Roth Black)
3. Richardson-Lucy

66
Comparison of non-blind deconvolution methods

1. Variational inference hold b fixed and
   marginalize out over it
2. Conjugate gradient descent on MAP solution (with
   field of experts prior Roth Black)
3. Richardson-Lucy

67
Comparison of non-blind deconvolution methods

1. Variational inference hold b fixed and
   marginalize out over it
2. Conjugate gradient descent on MAP solution (with
   field of experts prior Roth Black)
3. Richardson-Lucy

68
A very simple image
Start
image
x
blur
b
69
Miskin and Mackay, 2000

• Binary images
• Priors on intensities
• Small, synthetic blurs
• Not applicable to natural images