Virtual Environments: An ImageBased Approach

1
Virtual EnvironmentsAn Image-Based Approach

• How to get from Point A to Point C and figure out
  what Point B probably looked like

2
Two Papers

• View Interpolation for Image Synthesis
• Shenchang Eric Chen, Lance Williams
• QuickTime VR An Image-Based to Approach Virtual
  Environment Navigation
• Shenchang Eric Chen

3
Some Common Themes

• Pre-calculation
• Both papers achieve faster rendering time by
  pre-calculating portions of the scene offline
  from which images are derived at runtime
• Decouple rendering time from scene complexity
• All scenes are static

4
View Interpolation

• GOAL Use interpolated images to portray
  three-dimensional scenes
• Morphing is used to compute arbitrary
  intermediate frames from an array of prestored
  images.

5
Two Main Advantages

• The 3D representation of the scene may be
  replaced with images.
• The image synthesis time is independent of the
  scene complexity.

6
Previous Work

• Preprocess a subset of the scene that uses only
  those objects that are visible.
• This doesnt completely solve the problem because
  there could be point in the scene from which all
  objects are visible.

7
Previous Work

• Greene and Kass developed a method to approximate
  the visibility at a location from adjacent
  environment maps.
• Environment maps are pre-computed and stored with
  viewpoints arranged in a structured way, such as
  a 3D lattice.
• Advantage rendering time is proportional to the
  environment map resolutions and is independent of
  the scene complexity.
• Disadvantage requires Z-buffer hardware to
  render a relatively large number of polygons
  interactively

8
Visibility Morphing

• Image morphing is the simultaneous interpolation
  of shape and texture
• The technique involves two steps
• Establish the correspondence between two images
• Use to correspondence to blend the two images
  together into a new interpolated image

9
Establishing Pixel Correspondence

• Each pixels screen coordinates (x,y and z) and
  the cameras location are used to create a 4x4
  matrix transformation
• Transformations are precomputed and stored as
  offset vectors in a morph map

10
Interpolating Correspondences

• To generate an in-between view of a pair of
  images the offset vectors are interpolated
  linearly and the pixels in the source image are
  moved by the interpolated vector to their
  destinations
• What this does is approximate the transformation
  of the pixels coordinates by a perspective view
  matrix

11
Interpolating Correspondences
12
Compositing Images

• Two main problems
• Overlaps
• Caused by local image contraction
• Occurs when several samples in a local
  neighborhood of the source image move to the same
  pixel in the interpolated image
• Holes
• Caused by local image expansion
• May also arise from sample locations invisible in
  each of the source images but visible in the
  interpolated image

13
Holes
14
Implementation - Preprocessing

• Preprocessing
• This stage establishes the correspondence between
  each pair of source and destination images
• Get input data
• A source node image, range data and camera
  parameters
• A destination node only the camera parameters
  are needed
• A threshold factor for quadtree decomposition
• Create a morph map from the source to the
  destination
• Decompose the morph map into quadtree blocks and
  add the blocks to the block list
• Repeat the above steps for each arc connecting
  the set of nodes
• Sort the block list from back to front by the
  blocks Z-coordinates

15
Implementation - Interpolation

• Interactive Interpolation
• Get input data
• Fill the interpolated image with a distinguished
  background color
• Compute new location for every block in the list
  in back-to-front order
• Copy the pixel block from the source image to its
  new location in the interpolated image
• For every pixel in the interpolated image that
  still retains the background color, compute its
  color by filtering the colors of the adjacent
  non-background pixels

16
Applications

• Virtual Reality
• Motion Blur
• Generate additional temporal samples by
  interpolating between existing frames.
• Shadows
• Approximates an area light source by
  interpolating between multiple point light
  sources
• Image-Based Primitives
• Incremental Rendering

17
Some Examples
18
Some Examples
19
Some Examples
20
QuickTime VR

• GOAL A new approach for the creation and
  navigation of virtual environments
• Three objectives
• The system should playback at an interactive
  speed on most personal computers available at the
  time without hardware acceleration
• The system should accommodate both real and
  synthetic scenes
• The system should be able to display high quality
  images independent of scene complexity

21
Previous Work

• 3D Modeling and Rendering
• Branching Movies

22
3D Modeling and Rendering

• The entire virtual environment is rendered in
  real-time
• There are three problems with this approach
• Creating the scene is a tedious and
  time-consuming process
• Real-time rendering places limits on scene
  complexity and rendering quality.
• Special rendering hardware is needed which limits
  availability for most people.

23
Branching Movies

• Used extensively in the video game industry
• Multiple movie segments depicting spatial
  navigation paths are connected together at
  selected branch points
• The user is allowed to move on to a different
  path only at these branching points

24
Branching Movies

• Some problems
• Limited navigability and interaction.
• Requires a large amount of storage space for all
  the possible movies.
• Some benefits
• Does not require 3D modeling and rendering
• Uses photographs or movies
• Decouples rendering from interactive playback

25
Image-Based Rendering

• Simulates a virtual cameras motions in
  photographic or computer synthesized spaces
• Camera Motions
• Camera Rotation
• Object Rotation
• Camera Movement
• Camera Zooming

26
Camera Rotation
27
Camera Movement
28
The QuickTime Environment

• Composed of two types of players
• The Panoramic Movie Player
• The Object Movie Player

29
The Panoramic Movie Player

• A new panoramic track was added to the existing
  QuickTime player architecture
• Allows the user to perform continuous panning and
  continuous zooming.
• Utilizes cylindrical environment maps
• The image warp, which projects sections of the
  cylindrical image onto a planar view, is computed
  in real-time

30
Cylindrical Warping
31
The Object Movie Player

• Typically contains a two-dimensional array of
  frames.
• More frames can be inserted to allow for
  time-varying behavior (i.e. a flickering candle).

32
The Authoring Environment
33
Stitching

• The stitcher uses a correlation based image
  registration algorithm to match and blend
  adjacent images
• Adjacent pictures must have some overlap.
• In practice, a 50 overlap seems to work best
  because adjacent pictures may have very different
  brightness levels

34
Stitching

• Sometimes automatic stitching fails and manual
  intervention is necessary
• Factors which contribute to this are
• Missing Pictures
• Extreme Intensity Change
• Insufficient Image Features
• Improper Camera Mounting
• Significant Object Motion
• Film Scanning Errors

35
Stitching
36
Applications
37
Applications

• Star Trek/The Next GenerationInteractive
  Technical Manual.
• Lets the user navigate in the Starship Enterprise
  using panoramic movies.
• Several thousand still photographs were shot to
  create more than two hundred panoramic images,
  which cover most areas in the starship.
• Took only 2 months to complete
• Scientific/Engineering Simulation
• Interactive TV

38
Examples

• Object Movie
• Panoramic Movie 1
• Panoramic Movie 2
• Panoramic Movie 3

39
Any questions?