Image-Based Rendering

1
Image-Based Rendering

• Geometry and light interaction may be difficult
  and expensive to model
• Think of how hard radiosity is
• Imagine the complexity of modeling the exact
  geometry of carpet (as just one example)
• Image based rendering seeks to replace geometry
  and surface properties with images

2
Texture Mapping

• Replace small scale geometry and surface changes
  with an image attached to the surface
• Define a mapping from surface points into a
  texture image
• When the properties of that point are required,
  look up the pixel in the texture image
• A research field in itself
• Various methods of sampling the texture
• Various ways of combining the samples with direct
  lighting calculations
• Ways of arranging and accessing textures in memory

3
Environment Mapping

• Replace the world around an object with images
• Generally used for fast reflection computations
• Typically, define a cube around the object and
  associate images with the inside of each face
• When seeking reflection color, shot ray from
  surface point onto cube and take corresponding
  image point
• Images represent a view of the world from the
  center of the object
• Assumes that the world wouldnt look much
  different if viewed from points on the objects
  surface

4
Plenoptic Function

• Describes the intensity of light
• passing through a given point, x
• in a given direction, (?,?)
• with given wavelength, ?
• at a given time, t
• Many image-based rendering approaches can be cast
  as sampling from and reconstructing the plenoptic
  function
• Note, function is generally constant along
  segments of a line (assuming vacuum)

5
Image-Based Rendering

• Aim Generate an image from a desired view using
  existing images from other views
• May or may not know the viewing parameters for
  the existing images
• Existing images may be photographs or computer
  generated renderings

6
A Plethora of Approaches

• Methods differ in many ways
• The range of new viewpoints allowed
• The density of input images
• The representation for samples (known images)
• The amount of user help required
• The amount of additional information required
  (such as intrinsic camera parameters)
• The method for gathering the input images

7
Movie-Map Approaches

• Film views from fixed locations, closely spaced,
  and store on video-disc (would be DVD now)
• Allow the user to jump from location to location,
  and pan
• Appropriate images are retrieved from disk and
  displayed
• No reprojection just uses nearest existing
  sample
• Still used in video games today, but with
  computer generated movies

8
Quicktime VR (Chen, 1995)

• Movie-maps in software
• Construct panoramic images by stitching together
  a series of photographs
• Semi automatic process, based on correlation
• Scale/shift images so that they look most alike
• Works best with gt50 overlap
• Finite set of panoramas user jumps from one to
  the other

9
View Interpolation (Chen and Williams, 1993)

• Input A set of synthetic images with known depth
  and camera parameters (location, focal length,
  etc)
• Computes optical flow maps related each pair of
  images
• Optical flow map is the set of vectors describing
  where each point in the first image moves to in
  the second image
• Morphs between images by moving points along flow
  vectors
• Intermediate views are real views only in
  special cases

10
View Morphing (Seitz and Dyer, 1997)

• Uses interpolation to generate new views such
  that the intermediate views represent real camera
  motion
• Observation Interpolation gives correct
  intermediate views if the initial and final
  images are parallel views

11
View Morphing Process

• Basic algorithm
• Pre-warp input images to get them into parallel
  views
• Interpolate for intermediate view
• Post-warp to get final result
• User specifies a camera path that rotates and
  translates initial camera onto final camera
• Requires knowledge of projection matrices for the
  input images
• Found with vision algorithms. User may supply
  correspondences
• Intermediate motion can be specified by giving
  trajectories of four points

12
Plenoptic Modeling (McMillan and Bishop, 1995)

• Input Set of panoramic images gathered by
  panning a video camera about its vertical optical
  axis on a tripod
• The algorithm
• Determines the properties of each camera and
  registers the images associated with each pan
• Determines the relative locations of each camera
• Determines the depth of each point seen by
  multiple cameras
• Generates new views by reconstructing the
  plenoptic function from the available samples

13
Fitting Each Camera

• Problem
• Given several images all taken with the same
  camera rotated about its optical center
• Determine the camera parameters
• Determine the angle between each image
• Approach
• Multi-stage optimization
• First stage estimates angle between images and
  focal length
• Second stage refines and gets remaining
  parameters
• When done, can map a pixel in one image to its
  correct position in any other images

14
Locating the Cameras

• Given cylindrical projections (panoramic images)
  for two cameras
• User identifies 100-500 tie-points (points seen
  in both images)
• Each tie-point defines two rays one from the
  center of each camera through the tie-point
• These rays should intersect at the world location
  for the point
• Minimization step finds the camera locations and
  some other parameters that minimize the
  perpendicular distance between all the rays

15
Determining Disparity

• The minimization algorithm gives us the world
  location of the tie-points, but what about the
  rest of the points in the image?
• Use standard computer vision techniques to find
  the remaining disparities
• Disparity is the angular difference between the
  locations of a point in two images
• Directly related to the depth of the point
• Makes heavy use of the epipolar constraint

16
The Epipolar Constraint

• The location of a point in one image constrains
  it to lie on a ray in space passing through the
  camera and image point
• This ray projects to a curve in the second image
• Line for planar projection
• Sine curve for cylindrical projection
• Since the point must lie on the ray in world
  space, it must lie on the curve in the second
  image
• Reduces the search for correspondences to a
  linear one along the line

17
Reconstructing a New View

• The disparity from a known cylinder pair can be
  transferred to a new cylinder, and then
  reprojected onto a plane (the image)
• Can all be done in one step
• Have to resolve occlusion problems
• Two points from the reference image could map to
  the same point in the output image
• Solution Define a fill ordering that guarantees
  correct occlusion (an important contribution of
  this paper)
• Also have to fill holes

18
Summary

• Image-based rendering obviously relies heavily on
  computer vision techniques
• Particularly Depth from stereo
• The problem is very hard with real images
• These techniques are not perfect!
• Sampling remains a problem
• Images tend to appear blurry
• Relatively little work on reconstruction
  algorithms