Fast Video Enhancement using Superresolution

1
Fast Video Enhancement using Super-resolution

• 18798 Seminar
• Anup Doshi Jae K. Lee
• 4/23/04

2
Super-resolution

• Process of creating single high-resolution image
  from a sequence of low-resolution frames.
• Can be very useful for enhancing low-quality
  video
• Many algorithms are slow and/or computationally
  intensive
• Were trying out a faster algorithm

3
Algorithm

• Our super-resolution algorithm is based mostly
  upon algorithm described in 1 for pure
  translational motion and space-invariant blur.
• General Idea Given a sequence of low-resolution
  images, we can model them as a set of linear
  operators on a single high-resolution image.
• Thus it is possible to reconstruct the
  high-resolution image by applying the inverse
  operators
• 1 Elad, M. Hel-Or, Y. A fast super-resolution
  reconstruction algorithm for pure translational
  motion and common space-invariant blur Image
  Processing, IEEE Transactions on, Vol.10, Iss.8,
  Aug 2001Pages1187-1193

4
Algorithm from paper

• Estimate translational motion of low-res images
  with respect to base frame
• Model motion, blur, and decimation as linear
  transforms
• Lexicographically order blocks and apply
  necessary inverse transforms
• De-blur if necessary (assuming blur is
  space-invariant)
• Apply iteratively to enhance video sequence

5
Algorithm Specifics

• Define Hi-res image (vector) X and Low-res images
  (vectors) Yk, k1..N.
• Define Linear Matrix Operations
• Decimation Dk, Geometric Warp Fk, Blur Hk
• Then we assume the model
• Yk DkHkFkX
• Further Assumptions Hk, Dk constant for all k
  warp is pure translation
• Yk(pqx1) D(pqxmn)H(mnxmn)F(mnxmn)X(mnx1)

6
Algorithm Specifics

• Conclusions (from 1)
• We can apply inverse operations to construct X
  given Fks, D, H, and Yks
• Let
• Then X H-1(R-1P).
• De-blurring can be applied separately

7
Example

• Constructed 4 low-res images from single hi-res
  image downsampled by 2 with different offsets.

Original
Low-res image sequence
8
Example

• Reconstructed super-resolution image (same as
  original) via fast algorithm.
• Comparison of super-resolution to
  nearest-neighbor interpolation shows dramatic
  improvements.
• This specific (well-defined) case was chosen to
  demonstrate capabilities of algorithm. It is
  suitable to be generalized to more complicated
  cases.

9
Example

• More general example Assume only 3 out of 4
  low-res images from above are available
• Missing information Use median filtering or
  bilinear interpolation possibly minimize error
  via Steepest-Descent algorithm
• Output still very high quality

Super-resolution result with missing information
Using simple median filtering to fill in the gaps
10
Our Work Expanding on Prior Results

• Problem Assumption of pure translational motion
  is very limiting
• Solution Use block-based algorithm
• Split each frame into smaller blocks operate on
  each block
• Also eases computational load
• Allows for slightly different motions within
  frames.

11
Our Work Expanding on Prior Results

• Problem Estimating Motion
• Need to find out how the image sequence is moving
• Need to gauge whether our algorithm will work
  well
• Solution Optical Flow
• Operating on a few small blocks within image can
  quickly tell if the image has several different
  translations (or rotations), what they are, and
  if our algorithm will be suitable.

12
Optical Flow Example
Still some kinks!
13
Putting it together Our Algorithm

• Frame-by-Frame
• Split into blocks
• Optical Flow
• Determine what movement exists
• Determine if algorithm is viable
• Apply transforms
• Re-assemble blocks
• De-blur if necessary
• Get your super-resolution!

14
Questions?