Image Based Modeling and Rendering (PI: Malik)

1
Image Based Modeling and Rendering (PI Malik)

• Capture photorealistic models of real world
  scenes that can support interaction
• Vary viewpoint
• Vary lighting
• Vary scene configuration
• This objective was achieved by combining and
  inventing new techniques in computer vision and
  graphics.

2
Image-based Modeling and Rendering

• 1st Generation---- vary viewpoint but not
  lighting
• Acquire photographs
• Recover geometry (explicit or implicit)
• Texture map

3
Image-based Modeling and Rendering

• Photographs are not Reflectance Maps !
• 2nd Generation---- vary viewpoint and lighting
  for non-diffuse scenes
• Recover geometry
• Recover reflectance properties
• Render using light transport simulation

Illumination
Radiance
Reflectance
4
Image-based Modeling

• 2nd Generation---- vary viewpoint and lighting
• Recover geometry reflectance properties
• Render using light transport simulation or local
  shading

Original Lighting Viewpoint
Novel Lighting Viewpoint
5
Inverse Global Illumination (Yu et al, SIGGRAPH
99)
Reflectance Properties
Radiance Maps
Light Sources
Geometry
6
Real vs. Synthetic
7
Real vs. Synthetic
8
Key contributions in Inverse Global Illumination

• A digital camera can undertake all the data
  acquisition tasks involved.
• Both specular and high resolution diffuse
  reflectance properties can be recovered from
  photographs.
• Reflectance recovery can re-render non-diffuse
  real scenes under novel illumination as well as
  from novel viewpoints.

9
Image-based Modeling and Rendering

• 3rd Generation--Vary spatial configurations in
  addition to viewpoint and lighting

Novel Viewpoint
Novel Viewpoint Configuration
10
Framework

• Input
• Multiple range scans of a scene
• Multiple photographs of the same scene

• Output
• Geometric meshes of each object in the scene
• Registered texture maps for objects

11
Segmentation ResultsYu, Ferencz and Malik TVCG
01
12
Models of Individual Objects
13
Texture-Mapping and Object Manipulation
14
Transitions

• Façade has inspired several image-based modeling
  packages including Canoma from Metacreations.
• Technique for registering range and radiance
  images being used by Cyra, a leading time of
  flight laser scanner company.

15
Continuing Challenges

• Finding correspondences between widely separated
  camera views automatically
• Models of reflectance and texture for natural
  materials and objects