A Robust Super Resolution Method for Images of 3D Scenes

1
A Robust Super Resolution Method for Images of 3D
Scenes

• Pablo L. Sala
• Department of Computer Science
• University of Toronto

2

• Super Resolution
• The process of obtaining, from a set of images
  that overlap, a new image that in the regions of
  overlapping has a higher resolution than that of
  each individual image.
• This work
• Introduces a method for robust super
  resolution
• Applies it to obtain higher resolution images
  of a 3D scene from a set of calibrated
  low-resolution images under the assumption that
  the scene can be approximated by planes lying in
  3D space.

3
The Registration Problem
The algorithm presented here starts from a
discreet set of planes in 3D space and for each
plane it back-projects all the input images onto
that plane and performs super-resolution . Only
the image pixels of 3D points of the scene that
lie on the plane will be back-projected to the
same points in the plane.
4
Problem Formulation

• Assumptions
• The scene can be thought of as mostly lying on
  planes in 3D space.
• All viewpoints are located in a region
  separable by a plane from the scene, so to hold
  to a notion of scene depth.
• The Idea
• A set of N calibrated images of the scene.
• A list of planes ordered by its depth.
• Images set is partitioned in disjoint subsets of
  relatively same size.
• Super-resolution attempted on each plane using
  the back-projection of the images of each subset
  on that plane.
• The obtained higher resolution images are
  compared. The regions of coincidence will be
  assumed to correspond to parts of the scene lying
  on the plane.
• The portions of each image that correspond to
  these regions will not be taken into account when
  applying super-resolution on the following planes.

5
Standard Super-Resolution Super-resolution can be
formulated as an optimization problem   X is
the unknown higher resolution image xj are the
low resolution images Fj Dj Hj Wj are the
imaging formation matrices, with Dj a
decimation, Hj a blurring, and Wj a
geometric warp matrix. Deriving E with respect
to X and equaling it to 0 we get to 
6
Robust Super-Resolution Objective
function Error  Cauchys robust estimator
7
Robust Super-Resolution Deriving E with respect
to X     where Wj is a diagonal matrix such
that Iteratively re-weighted least
squares Initial guess X0 the higher
resolution image computed using the standard
version of super-resolution.
8

• The Algorithm
• Input L a list of planes in 3D space Ij images
  Pj camera matrices
• Set O ? (set of the occluding 3D points)
• Randomly partition the images set into two or
  more disjoint subsets S1,..., SR.
• For each plane ? in L do
• For each subset Si do
• Use ?, Pj and O to compute the image formation
  matrices Fj
• X0 S-Resolution of images in Si
• Yi Robust S-R of images in Si, using X0 as
  the initial guess
• R region R of ? where all Yi coincide
• Assume R is the portion of the scene that lies
  on ?
• O O ? R

9

• Observations
• For the algorithm to be effective
• The 3D planes uniformly distributed, close one
  to another, and covering all scene depth, so to
  detect all scene portions to be aware of
  occlusions.
• The scene should be highly textured for the
  similarity check to work well.

10
Experimental Results Synthetic scene
11
Experimental Results

• 12 images were used as input
• 2 disjoint subsets

12
Experimental Results
Standard vs Robust Super-Resolution
13
Experimental Results
Output images
14
Observations and Future Work

• Results get worse as we advance in the 3D
  planes due to errors in defining precisely what
  portion of the scene lies in each plane
• Using more images and partitioning the images
  set in more than just two subsets will definitely
  give more accuracy to the similarity check.
• Using color images instead of just BW. The
  super-resolution steps of the algorithm would be
  applied to each color channel separately and
  similarity would be assumed when the difference
  in reconstruction is close to zero for all three
  channels.