NonParametric Image SuperResolution Using Multiple Frames

1
Non-Parametric Image Super-Resolution Using
Multiple Frames

• Mithun Das Gupta, Shyamsundar Rajaram, Thomas S.
  Huang
• University of Illinois at Urbana-Champaign, USA
• Nemanja Petrovic
• Siemens Corporate Research, New Jersey, USA

2
In this talk

• Introduction Multi-Frame Super-Resolution
• Problem Statement and Image Model
• Review Belief Propagation
• Non-parametric Belief Propagation
• Multi-frame Super-resolution
• Conclusion and Future Directions

3
Super-resolution

• Multi-Frame Super-resolution is the process of
  inferring a high-resolution frame from a set of
  low-resolution frames.
• Application
• video super-resolution (high resolution frame
  extraction)
• We consider the case where each low-resolution
  frame is fractionally shifted (sub-pixel) with
  respect to the others.

4
Multi-Frame Super-Resolution
The first step is registration, to align all the
low-res frames up to sub-pixel accuracy. Given a
set of low resolution frames with sub-pixel
shifts, they can be arranged on a high resolution
grid.
Unknown pixel locations
5
Multi-Frame Super-Resolution
4 low-resolution frames (arrows represent ½ pixel
shift)
Reconstructed high-resolution image
Actual high-resolution image
6
Multi-Frame Super-Resolution
In most practical applications not all the data
points on the high-res grid are available, and
hence registration is followed by an
interpolation step.
3 low-res frames (arrows represent ½ pixel shift)
Reconstructed high-res image
Actual high-res image
7
In this talk

• Introduction Multi-Frame Super-Resolution
• Problem Statement and Image Model
• Review Belief Propagation
• Non-parametric Belief Propagation
• Multi-frame Super-resolution
• Conclusion and Future Directions

8
Problem Statement and Image Model
Problem Perform Interpolation on the
high-resolution grid to obtain the unknown
pixels Our approach Supervised
Learning Training set of images X1, X2, , Xn
set of high resolution images Learning
algorithm Obtain a model which can be used to
infer the image X from observed low resolution
(filtered, down-sampled) images Y1, Y2, , Ym
which are registered on a high-resolution
grid. Model Markov Random Field over image
patches. The patches are 4x4 pixels in dimensions
and are non-overlapping.
9
Problem Statement and Image Model
Markov Random Field
X1
X2
Hidden node xi
Xi
Y2
Y1
Yi
Observed node yi
X3
X4
Y3
Y4
10
Problem Statement and Image Model

• learning phase and
• inference phase.

X1
X2
X3
interaction potential association
potential
Y3
Y2
Y1
11
Problem Statement and Image Model
Unknown Pixels
12
Problem Statement and Image Model
The association potential between the image patch
xi and the observed patch yi has the following
factorized form. Let the sets Ki and Ui represent
the observed and unobserved pixels of the ith
patch respectively.
xiu xij, for all j ? Ui xik xij, for all
j ? Ki
13
Learning Potentials
6
8
14
Association Potential
16
1
xi
2
1
5
9
13
3
2
6
10
14
4
7
11
15
3
5
4
8
12
16
7
9
Known Pixels
10
11
Unknown Pixels
12
13
Feature Vector
15
14
Learning Potentials
13
14
15
Interaction Potential
Feature Vector
16
9
xj
xi
10
1
5
9
13
1
5
9
13
11
10
2
6
10
14
2
6
14
12
11
7
11
3
15
7
3
15
1
12
16
4
8
12
16
4
8
2
3

• Partial Messages (CVPR 05)
• Parametric densities Smoothing issues
• Non-parametric Representation Place Gaussian
  kernels at each data point. Variances are learnt
  using cross validation

4
5
6
7
8
15
In this talk

• Introduction Multi-Frame Super-Resolution
• Problem Statement and Image Model
• Review Belief Propagation
• Non-parametric Belief Propagation
• Multi-frame Super-resolution
• Conclusion and Future Directions

16
Belief Propagation
x1
m45
x2
x5
x4
?(x5,x4)
y4
For simplicity on local observation for node x4
has been shown.
17
Belief Propagation
m2
m3
m1
xi
m4
F(xi,yi)
yi
18
In this talk

• Introduction Multi-Frame Super-Resolution
• Problem Statement and Image Model
• Review Belief Propagation
• Non-parametric Belief Propagation
• Multi-frame Super-resolution
• Conclusion and Future Directions

19
Non Parametric Belief Propagation
In Belief Propagation the messages are mixtures
of Gaussians
Suppose each message has 10 components and we
have 4 messages. The resultant product consists
of 104 components!!! For most applications the
computational complexity to evaluate the product
grows beyond bounds.
20
Non Parametric Belief Propagation
One way is to approximate the messages as mixture
of smaller number of Gaussians (pruning)
21
Non Parametric Belief Propagation
We represent the messages as non-parametric
kernel densities. In other words, we put a
Gaussian kernel over each point.
Means of the Gaussians are given by the
individual points. The variances are chosen by
Cross-Validation technique Silverman85.
22
Non Parametric Belief Propagation
Once the kernel densities are obtained we still
need to evaluate the integral, which is still not
feasible.

• Next we use sampling techniques to approximate
  the product.
• Alternating parallel Gibbs sampling for
  computing the product of mixture of Gaussians
• Stochastic Integration

23
Alternating Gibbs sampler
Suppose we have 3 messages (1D) with 5 Gaussian
components each.
Generate a random point z
Compute the likelihood of the point being
generated by each component of each message.
Sample from the multinomial to generate a label
for each message.
Multiply the Gaussians corresponding to the
chosen labels and sample from the resultant
distribution.
Repeat the steps for more samples
24
In this talk

• Introduction Multi-Frame Super-Resolution
• Problem Statement and Image Model
• Review Belief Propagation
• Non-parametric Belief Propagation
• Multi-frame Super-resolution
• Conclusion and Future Directions

25
Algorithm

• Registration Probabilistic upsample and search
  based registration
• Learning the potentials based on the unknown
  locations.
• Restoration using NBP

26
Registration
27
Training Data

• Training
• Faces 9 faces of 5 persons were used for
  training (ORL data base). The training images
  were of resolution 40x40.
• Digits 20 fonts for each digit. The training
  images were of resolution 40x40.
• Experiments
• 2 low resolution frames shifted in x or y
  direction
• 2 low resolution frames shifted diagonally
• 3 low resolution frames
• The resolution enhancement in each direction was
  2.

28
Super-resolution Results
Left one of two input images diagonally shifted
Left one of two input images shifted
horizontally
Left one of three input images
Middle result after 10 iterations, Right
original image
29
Super-resolution Results
Left image is one of the two input images to the
system. Second and third images are the high
resolution images after 5 and 10 iterations of
the inference algorithm. Right image is the
actual high resolution image.
30
Super-resolution Results
3 low-resolution frames shifted upto ½ pixel from
each other
Proposed method
Bilinear
31
In this talk

• Introduction Multi-Frame Super-Resolution
• Problem Statement and Image Model
• Review Belief Propagation
• Non-parametric Belief Propagation
• Multi-frame Super-resolution
• Conclusion and Future Directions

32
Conclusion

• Observed images are registered and placed on a
  high-resolution grid
• Supervised learning framework for performing
  interpolation of unknown pixels in the
  high-resolution grid
• Non-parametric Markov Random Fields used for
  modeling images
• NBP for inferring unknown pixels

33
Future Work

• Incorporating location information in the
  inference agorithm
• Cross-gender testing evaluations
• Natural scenes

34
Thank You