Super-Resolution

1
Super-Resolution

• Digital PhotographyCSE558, Spring 2003Richard
  Szeliski

2
Super-resolution

• convolutions, blur, and de-blurring
• Bayesian methods
• Wiener filtering and Markov Random Fields
• sampling, aliasing, and interpolation
• multiple (shifted) images
• prior-based methods
• MRFs
• learned models
• domain-specific models (faces)- Gary

3
Linear systems

• Basic properties
• homogeneity Ta X a TX
• additivity TX1X2 TX1TX2
• superposition TaX1bX2 aTX1bTX2
• Linear system ? superposition
• Examples
• matrix operations (additions, multiplication)
• convolutions

4
Signals and linear operators

• Continuous I(x)
• Discrete Ik or Ik
• Vector form I
• Discrete linear operator y A x
• Continuous linear operator
• convolution integral
• g(x) sh(?,x) f(?) d?, h(?,x) impulse
  response
• g(x) s h(?-x) f(?) d? f h(x) shift
  invariant

5
2-D signals and convolutions

• Continuous I(x,y)
• Discrete Ik,l or Ik,l
• 2-D convolutions (discrete)
• gk,l ?m,n fm,n hk-m,l-n
• ?m,n fm,n h1k-mh2l-n separable
• Gaussian kernel is separable and radial
• h(x,y) (2??2)-1exp-(x2y2)/?2

6
Convolution and blurring
7
Separable binomial low-pass filter
8
Fourier transforms

• Project onto a series of complex sinusoids
• Fm,n ?k fk,l e-i 2?(kmln)
• Properties
• shifting g(x-x0) ? exp(-i 2?fxx0)G(fx)
• differentiation dg(x)/dx ? i 2?fxG(fx)
• convolution f g(x) ? F G (fx)

9
Blurring examples

• Increasing amounts of blur Fourier transform

10
Sharpening

• Unsharp mask (darkroom photography)
• remove some low-frequency content y y s (y
  g y)spatial (blur, sharp) freq
  (blur,sharp)

11
Sharpening - result

• Unsharp mask original, blur (s1), sharp(s0,
  1, 2)

12
Deconvolution

• Filter by inverse of blur
• easiest to do in the Fourier domain
• problem high-frequency noise amplification

13
Bayesian modeling

• Use prior model for image and noise
• y g x n, x is original, y is blurred
• p(xy) p(yx)p(x) exp(-y gx2/2sn-2)
  exp(-x2/2sx-2)
• -log p(xy) ? y gx2sn-2 x2sx-2where
  the norm is summed squares over all pixels

14
Parsevals Theorem

• Energy equivalence in spatial ? frequency domain
• x2 F(x)2
• -log p(xy) ? Y(f) G(f)X(f)2sn-2
  X(f)2sx-2
• least squares solution (?/?X 0)X(f) G(f)Y(f)
  / G2(f) sn2/sx2

15
Wiener filtering

• Optimal linear filter given noise and signal
  statistics
• X(f) G(f)Y(f) / G2(f) sn2/sx2
• low frequencies X(f) G-1(f)Y(f)boost by
  inverse gain (blur)
• high frequencies X(f) G(f) sn-2sx2
  Y(f)attenuate by blur (gain)

16
Wiener filtering white noise prior

• Assume all frequencies equally likely
• p(x) N(0,sx2)
• X(f) G(f)Y(f) / G2(f) sn2/sx2
• solution is too noisy in high frequencies

17
Wiener filtering pink noise prior

• Assume frequency falloff (natural statistics)
• p(X(f)) N(0,f-ßsx2)
• X(f) G(f)Y(f) / G2(f) fßsn2/sx2
• greater attenuation at high frequencies
  G(f) H(f)

18
Markov Random Field modeling

• Use spatial neighborhood prior for image
• -log p(x) ?ij?C?(xi-xj)where ?(v) is a robust
  norm
• ?(v) v2 quadratic norm ? pink noise
• ?(v) v total variation (popular with maths)
• ?(v) vß natural statistics
• ?(v) v2,v Huber normSchultz, R.R.
  Stevenson, IEEE TIP, 1996

i
j
19
MRF estimation

• Set up discrete energy (quadratic or non-)
• -log p(xy) ? sn-2 y Gx2 ?ij?C?(xi-xj)where
  G is sparse convolution matrix
• quadratic solve sparse linear system
• non-quadratic use sparse non-linear least
  squares (Levenberg-Marquardt, gradient descent,
  conjugate gradient, )

20
Sampling a signal

• sampling
• creating a discrete signal from a continuous
  signal
• downsampling (decimation)
• subsampling a discrete signal
• upsampling
• introducing zeros between samples
• aliasing
• two sampled signals that differ in their original
  form (many ? one mapping)

21
Sampling
interpolation
22
Nyquist sampling theorem

• Signal to be (down-) sampled must have a
  bandwidth no larger than twice the sample
  frequency
• ?s 2? / ns gt 2 ?0

23
Box filter (top hat)
24
Ideal low-pass filter
25
Simplified camera optics

1. Blur pill-boxBessel2 (diffr.) Gaussian
2. Integrate box filter
3. Sample produce single digital sample
4. Noise additive white noise

26
Aliasing

• Aliasing (jaggies and crawl) is present
  ifblur amount lt sampling (s 1)
• shift each image in previous pipeline by 1

27
Aliasing - less

• Less aliasing (jaggies and crawl) is present
  ifblur amount sampling (s 2)
• shift each image in previous pipeline by 1

28
Multi-image super-resolution

• Exploit aliasing to recover frequencies above
  Nyquist cutoff
• ?ksn-2 yk Gkx2 ?ij?C?(xi-xj)where Gk are
  sparse convolution matrices
• quadratic solve sparse linear system
• non-quadratic use sparse non-linear least
  squares (Levenberg-Marquardt, gradient descent,
  conjugate gradient, )
• projection onto convex sets (POCS)

29
Multi-image super-resolution

• Need
• accurate (sub-pixel) motion estimates(Wednesdays
  lecture)
• accurate models of blur (pre-filtering)
• accurate photometry
• no (or known) non-linear pre-processing(Bayer
  mosaics)
• sufficient images and low-noise relative to
  amount of aliasing

30
Prior-based Super-Resolution

• Classical non-Gaussian priors
• robust or natural statistics
• maximum entropy (least blurry)
• constant colors (black white images)

31
Example-based Super-Resolution

• William T. Freeman, Thouis R. Jones, and Egon C.
  Pasztor,IEEE Computer Graphics and Applications,
  March/April, 2002
• learn the association between low-resolution
  patches and high-resolution patches
• use Markov Network Model (another name for Markov
  Random Field) to encourage adjacent patch
  coherence

32
Example-based Super-Resolution

• William T. Freeman, Thouis R. Jones, and Egon C.
  Pasztor,IEEE Computer Graphics and Applications,
  March/April, 2002

33
References classic

• Irani, M. and Peleg. Improving Resolution by
  Image Registration. Graphical Models and Image
  Processing, 53(3), May 1991, 231-239.
• Schultz, R.R. Stevenson, R.L. Extraction of
  high-resolution frames from video sequences. IEEE
  Trans. Image Proc., 5(6), Jun 1996, 996-1011.
• Elad, M. Feuer, A.. Restoration of a single
  superresolution image from several blurred,
  noisy, and undersampled measured images. IEEE
  Trans. Image Proc., 6(12) , Dec 1997, 1646-1658.
• Elad, M. Feuer, A.. Super-resolution
  reconstruction of image sequences. IEEE PAMI
  21(9), Sep 1999, 817-834.
• Capel, D. Zisserman, A.. Super-resolution
  enhancement of text image sequences. CVPR 2000,
  I-600-605 vol. 1.
• Chaudhuri, S. (editor). Super-Resolution
  Imaging. Kluwer Academic Publishers. 2001.

34
References strong priors

• Freeman, W.T. Pasztor, E.C.. Learning low-level
  vision, CVPR 1999, 182-1189 vol.2
• William T. Freeman, Thouis R. Jones, and Egon C.
  Pasztor, Example-based super-resolution, IEEE
  Computer Graphics and Applications, March/April,
  2002
• Baker, S. Kanade, T. Hallucinating faces.
  Automatic Face Gesture Recognition, 2000, 83-88.
• Ce Liu Heung-Yeung Shum Chang-Shui Zhang. A
  two-step approach to hallucinating faces global
  parametric model and local nonparametric model.
  CVPR 2001. I-192-8.