Super-Resolution

1
Super-Resolution

• Authors Wagner, Waagen and Cassabaum
• Presented By Mukul Apte

2
Introduction

• Definition Create High Resolution visual output
  from Low Resolution visual input.
• Mathematical assistance to features viz. motion
  detection, face recognition, person detection.
• Action Packed Sports Images
• Astronomy
• Medical Imaging
• Surveillance
• Range of resources low (single camera, polaroid
  lenses) to high (super-resolution chips,
  androids-ASIMO)

3
Basic idea
Given A set of degraded (warped, blurred,
decimated, noised) images
Required Fusion of the measurements into a
higher resolution image
4
Types

• Static Super-Resolution (SSR) - The creation of
  a single improved image, from the finite measured
  sequence of images
• Dynamic Super-Resolution (SSR) - Low Quality
  Movie In - High Quality Movie Out

5
Simple Example

• Concepts I Raster, CCD, Image in mathematical
  domain, Matlab
• Concepts II Bresenhams line algorithm,
  non-uniform interpolation, frequency domain

6
Simple Example
7
Simple Example
Due to our limited camera resolution, we sample
using an insufficient 2D grid
8
Simple Example
However, we are allowed to take a second picture
and so, shifting the camera slightly to the
right we obtain
9
Simple Example
Similarly, by shifting down we get a third image
2D
2D
10
Simple Example
And finally, by shifting down and to the right we
get the fourth image
2D
2D
11
Simple Example - Conclusion
It is trivial to see that interlacing the four
images, we get that the desired resolution is
obtained, and thus perfect reconstruction is
guaranteed.
12
Uncontrolled Displacements
In the previous example we counted on exact
movement of the camera by D in each direction.
What if the camera displacement is uncontrolled?
13
Uncontrolled Displacements
It turns out that there is a sampling theorem due
to Yen (1956) and Papoulis (1977) covering this
case, guaranteeing perfect reconstruction for
periodic uniform sampling if the sampling density
is high enough (1 sample per each D-by-D square).
14
Uncontrolled Rotation/Scale/Disp.
In the previous examples we restricted the camera
to move horizontally/vertically parallel to the
photograph object. What if the camera rotates?
Gets closer to the object (zoom)?
15
Uncontrolled Rotation/Scale/Disp.
16
Further Complications

• Sampling is not a point operation there is a
  blur
• Motion may include perspective warp, local
  motion, etc.
• Samples may be noisy any reconstruction process
  must take that into account.

17
Static Super-Resolution
Static Super-Resolution
t
Low Resolution Measurements
Static Super-Resolution Algorithm
High Resolution Reconstructed Image
18
Dynamic Super-Resolution
Low Resolution Measurements
t
High Resolution Reconstructed Images
t
19
Approach

• Image Registration
• Motion Estimation
• Projection onto High-Resolution Grid
• Non-Uniform Interpolation
• Frequency domain alias correction

Projection
Registration
Low-res Images
Registration (sub-pixel grid)
High Res Grid
20
1.1 Registration (angle)

• Rotation Calculation
• Correlate 1st LR image with all LR images at all
  angles
• OR
• Calculate energy at all angles for all LR images.
  Correlate energy vector to find the rotation angle

Anglei max index(correlation(I1(?), Ii (?)))
LR image 1
LR image 2
i 2,3,..,N (number of LR images)
Energy at angle Ii(?)
Energy at angle I2(?)
21
1.1 Registration (shift)

• Shift Calculated using Frequency Domain Method

?s ? ?x ?yT u ? fx fy

• Used only 6 lower u (high freq could be aliased)
• Used least square to calculate ?s

22
2.1 Frequency Domain

• Input Down-sampled aliased images
• Goal I Correct the low-freq aliased data
• Goal II Predict the lost high freq values

p
-p
p
-p
Original High-Res
Down-sampled
Aliased (fix it)
Lost (find it)
p/2
-p/2
p
p
-p
Up-sampled
Desired High-Res
23
2.2 Projection onto High-res grid

• Papoulis-Gerchberg Algorithm
• Projection onto convex sets
• Known pixel values
• Known Cut-off freq in the HR image
• Algorithm

24
SSR The Model
25
Warp Linear Operation
Z
X
Per every point in X find a matching point in Z
26
Model Assumptions
We assume that the images Yk and the operators
Hk, Dk, Fk, Wk are known to us, and we use them
for the recovery of X. Yk The measured images
(noisy, blurry, down-sampled ..) Hk The blur
can be extracted from the camera
characteristics Dk The decimation is dictated
by the required resolution ratio Fk The warp
can be estimated using motion estimation Wk The
noise covariance can be extracted from the camera
characteristics
27
Special Condition - Noiseless
Clearly, this linear system of equations should
have more equations than unknowns in order to
make it possible to have a unique solution.
28
SSR Handling Problems
Single image de-noising
Single image restoration
Single image scaling
Motion compensation average
Using
29
SSR Standard Equation
30
Thumb Rule on Desired Resolution

• Assume that we have N images of M-by-M pixels,
  and we would like to produce an image X of size
  L-by-L. Then

31
Papoulis Gerchberg Algorithm
Initial Setup
FFT (Initial image)
Taj Mahal Low-res image I
32
Papoulis Gerchberg Algorithm
Known Pixel Values
Image at iteration 0
Image after 1st iteration
FFT
I(high freq) 0
33
Papoulis Gerchberg Algorithm
Known Pixel Values
Image at iteration 1
Image after 10 iterations
FFT
I(high freq) 0
34
Papoulis Gerchberg Algorithm
After 50 iterations
SR Reconstructed image
Bilinear Interpolation
Bicubic Interpolation
35
Results (Images - I)

• Input 4 snaps using a high-res digital camera
• Cropped the same part of each image
• SR algorithm compared with bicubic interpolation

Original Low-res images (Courtesy Patrick
Vandewalle)
36
Results (Images - I)
Bicubic Interpolation
37
Results (Images - I)
Super-resolution
38
Results (Images - II)
Low-Res Image I
Low-Res Image II

• SR Algorithm didnt work as expected !!!
• Reason
• Motion was not restricted to shifts rotation
• Images had affine mapping
• Rule I - Need Correct Registration

39
Results (Frames - I)

Original
Bicubic
SR

• Why didnt SR work???
• Low-res images were created by forcing shifts at
  critical velocities
• Rule II - If low-res frames are at critical
  velocities, cant create good high-res frame

40
Results (Frames - II)
Original
Bicubic
SR

• Why did SR work so well???
• Low-res images were created by forcing shifts at
  non-critical velocities
• Rule III ? If low-res images have all the info
  about high-res then HR image can be perfectly
  constructed

41
Future Work

• Superresolution with multiple motions between
  frames - create high res video
• Predict the high-res frequency components using
  wavelet methods

Predict
Predict
Predict
42
Acknowledgements

• Prof John Apostolopoulos
• Prof Susie Wee
• Patrick Vandewalle

43
THANK YOU!!!

• QUESTIONS?
• COMMENTS?