Introduction%20to%20Image-Based%20Rendering

1
Introduction to Image-Based Rendering

• Lining Yang
• yangl1_at_ornl.gov
• A part of this set of slides reference slides
  used at Standford by Prof. Pat Hanrahan and
  Philipp Slusallek.

2
References

• S. E. Chen, QuickTime VR An Image-Based
  Approach to Virtual Environment Navigation,
  Proc. SIGGRAPH 95, pp. 29-38, 1995
• S. Gortler, R. Grzeszczuk, R. Szeliski, and M.
  Cohen, The Lumigraph, Proc SIGGRAPH 96, pp.
  43-54, 1996
• M. Levoy and P. Hanrahan, Light Field
  Rendering, Proc. SIGGRAPH 96, 1996.
• L. McMillan and G. Bishop, Plenoptic Modeling
  An Image-Based Rendering System, Proc. SIGGRAPH
  95, pp. 39-46, 1995
• J. Shade, S. Gortler, Li-Wei He, and R. Szeliski,
  Layered Depth Images, Proc. SIGGRAPH 98, pp
  231-242, 1998
• Heung-Yeung Shum, Li-Wei He, Rendering With
  Concentric Mosaics, Proc. SIGGRAPH 99, pp.
  299-306, 1999

3
Problem Description

• Complex Rendering of Synthetic Scene takes too
  long to finish
• Interactivity is impossible
• Interactive visualization of extremely large
  scientific data is also not possible
• Image-Based Rendering (IBR) is used to accelerate
  the renderings.

4
Examples of Complex Rendering
Povray quaterly competition site March June,
2001
5
Examples of Large Dataset
LLNL ASCI Quantum molecular simulation site
6
Image-Based Rendering (IBR)

• The models for conventional polygon-based
  graphics have become too complex.
• IBR represents complex 3D environments using a
  set of images from different (pre-defined)
  viewpoints
• It produces images for new views using these
  finite initial images and additional information,
  such as depth.
• The computation complexity is bounded by the
  image resolution, instead of the scene complexity.

7
Image-Based Rendering (IBR)
Mark Levoys 1997 Siggraph talk
8
Overview of IBR Systems

• Plenoptic Function
• QuicktimeVR
• Light fields/lumigraph
• Concentric Mosaics
• Plenoptic Modeling and Layered Depth Image

9
Plenoptic Function

• Plenoptic function (7D) depicts light rays
  passing through
• center of camera at any location (x,y,z)
• at any viewing angle ( ?,? )
• for every wavelength ( ? )
• for any time ( t )

10
Limiting Dimensions of Plenoptic Functions

• Plenoptic modeling (5D) ignore time
  wavelength
• Lumigraph/Lightfield (4D) constrain the scene
  (or the camera view) to a bounding box
• 2D Panorama fix viewpoint, allow only the
  viewing direction and camera zoom to be changed

11
Limiting Dimensions of Plenoptic Functions

• Concentric mosaics (3D) index all input image
  rays in 3 parameters radius, rotation angle and
  vertical elevation

12
Quicktime VR

• Using environmental maps
• Cylindrical
• Cubic
• spherical
• At a fixed point, sample all the ray directions.
• Users can look in both horizontal and vertical
  directions

13
Mars Pathfinder Panorama
14
Creating a Cylindrical Panorama
From www.quicktimevr.apple.com
15
Commercial Products

• QuickTime VR, LivePicture, IBM (Panoramix)
• VideoBrush
• IPIX (PhotoBubbles), Be Here, etc.

16
Panoramic Cameras

• Rotating Cameras
• Kodak Cirkut
• Globuscope
• Stationary Cameras
• Be Here

17
Quicktime VR

• Advantages
• Using environmental map
• Easy and efficient
• Disadvantages
• Cannot move away from the current viewpoint
• No Motion Parallax

18
Light Field and Lumigraph

• Take advantage of empty space to
• Reduce Plenoptic Function to 4D
• Object or viewpoint inside a convex hull
• Radiance does not change along a line unless
  blocked

19
Lightfield Parameterization

• Parameterize the radiance lines by the
  intersections with two planes.
• A light Slab

t
L(u,v,s,t)
v
s
u
20
Two Plane Parametrization
Object
Focal plane (st)
Camera plane (uv)
21
Reconstruction

• (u, v) and (s, t) can be calculated by
  determining the intersection of image ray with
  the two planes
• This can also be done via texture mapping
• (x, y) to (u, v) or (s, t) is a projective
  mapping

22
(No Transcript)
23
Capturing Lightfields

• Need a 2D set of (2D) images
• Choices
• Camera motion human vs. computer
• Constraints on camera motion planar vs.
  spherical
• Easier to construct
• Coverage and sampling uniformity

24
Light field gantry

• Applications
• digitizing light fields
• measuring BRDFs
• range scanning
• Designed by
• Marc Levoy et al.

25
Light Field

• Key Ideas
• 4D function
• - Valid outside convex hull
• 2D slice image
• - Insert to create
• - Extract to display

26
Lightfields

• Advantages
• Simpler computation vs. traditional CG
• Cost independent of scene complexity
• Cost independent of material properties and other
  optical effects
• Disadvantages
• Static geometry
• Fixed lighting
• High storage cost

27
Concentric Mosaics

• Concentric mosaics easy to capture, small in
  storage size

28
Concentric Mosaics

• A set of manifold mosaics constructed from slit
  images taken by cameras rotating on concentric
  circles

29
Sample Images
30
Rendering a Novel View
31
Construction of Concentric Mosaics

• Synthetic scenes
• uniform angular direction sampling
• square root sampling in radial direction

32
Construction of Concentric Mosaics (2)

• Real scenes

Bulky, costly
Cheaper, easier
33
Construction of Concentric Mosaics (3)

• Problems with single camera
• Limited horizontal fov
• Non-uniform spatial horizontal resolution
• Video sequence can be compressed with VQ and
  entropy encoding (25X)
• Compressed stream gives 20fps on PII300

34
Results
35
Results (2)
36
Image Warping

• McMillans 5D plenoptic modeling system
• Render or capture reference views
• Creating Novel Views
• Using reference views color and depth
  information with the warping equation
• For opaque scenes, the location or depth of the
  point reflecting the color is usually determined.
• Calculated using vision techniques for real
  imagery.

37
Image Warping (filling holes)

• Dis-occlusion problem Previously occluded
  objects in the reference view can be visible in
  the new view
• Fill in holes from other viewpoints or images
  (Mark William et al).

38
Layered Depth Images

• Different primitives according to depth values
• Image
• Image with depth
• LDI
• polygons

39
Layered Depth Images

• Idea
• Handle disocclusion
• Store invisible geometry in depth images

40
Layered Depth Image

• Data structure
• Per pixel list of depth samples
• Per depth sample
• RGBA
• Z
• Encoded Normal direction, distance

41
Layered Depth Images

• Computation
• Implicit ordering information
• LDI is broken into four regions according to
  epipolar point
• Incremental warping computation
• Start xincr (back to front order)
• Splat size computation
• Table lookup

42
Layered Depth Images