Convergence of vision and graphics

1
Convergence of vision and graphics

• Jitendra Malik
• University of California at Berkeley

2
Overview
3D capture
Modeling, simulation
Rendering
Display
Applications Simulation Virtual Reality Remote
collaboration
3
Graphics and Vision

• Computer graphics is the forward problem given
  scene geometry, reflectances and lighting,
  synthesize an image.
• Computer vision must address the inverse problem
  given an image/multiple images, reconstruct the
  scene geometry, reflectacnes and illumination.

4
Image-based Modeling
Recover Models of Real World Scenes and Make
Possible Various Visual Interactions

• Vary viewpoint
• Vary lighting
• Vary scene configuration

5
Image-based Modeling

• 1st Generation---- vary viewpoint but not
  lighting
• Acquire photographs
• Recover geometry (explicit or implicit)
• Texture map

6
Recovering geometry

• Historical roots in photogrammetry and analysis
  of 3D cues in human vision
• Single images adequate given knowledge of object
  class
• Multiple images make the problem easier, but not
  trivial as corresponding points must be
  identified.

7
Arc de Triomphe
8
The Taj Mahal
Taj Mahal modeled from one photograph by G.
Borshukov
9
Campus Model of UC Berkeley

• Campanile 40 Buildings (Debevec et al)

10
(No Transcript)
11
(No Transcript)
12
Image-based Modeling

• 2nd Generation---- vary viewpoint and lighting
• Recover geometry reflectance properties
• Render using light transport simulation or local
  shading

Original Lighting Viewpoint
Novel Lighting Viewpoint
13
Inverse Global Illumination (Yu et al)
Reflectance Properties
Radiance Maps
Light Sources
Geometry
14
Real vs. Synthetic
15
Real vs. Synthetic
16
Image-based Modeling

• 3rd Generation--Vary spatial configurations in
  addition to viewpoint and lighting

Novel Viewpoint
Novel Viewpoint Configuration
17
Our Framework

• Input
• Multiple range scans of a scene
• Multiple photographs of the same scene

• Output
• Geometric meshes of each object in the scene
• Registered texture maps for objects

18
Segmentation From images to objects
19
Segmentation ResultsYu, Ferencz and Malik 00
20
Models of Individual Objects
21
Texture-Mapping and Object Manipulation
22
Image Based modeling for motion capture
Body Suits, Markers
Video Motion Capture
23
Eadweard Muybridge Bregler and Malik 98
24
Continuing Challenges

• Finding correspondences automatically
• Optimal estimation of structure from n views
  under perspective projection
• Models of reflectance and texture for natural
  materials and objects