Combining Approximate Geometry

1

SIBGRAPI 2002

• Combining Approximate Geometry
• with VDTM A Hybrid Approach to
• 3D Video Teleconferencing
• Celso Kurashima2, Ruigang Yang1, Anselmo Lastra1
• 1Department of Computer Science
• University of North Carolina at Chapel Hill
• 2Laboratório de Sistemas Integráveis - LSI
• Escola Politécnica da Universidade de São Paulo
• Fortaleza, October 8th, 2002

2
Introduction
3
Introduction

• Video Conference
• 2D vs. 3D

1 camera Many cameras
Fixed viewpont Free viewpoint
No eye contact Eye contact
Standard Video Computer Vision Computer Graphics
4
Outline

• 3D Video Teleconference System
• Geometry Extraction
• Rendering Images with VDTM
• Results
• Conclusion

5
3D Video Teleconference System (1/3)

• Top View

(A) (B)
6
3D-Video Teleconference System (2/3)
7
3D-Video Teleconference System (3/3)

• How does it work?
• 1st Create a geometry proxy of the person
• 2nd Texture map images onto the proxy with VDTM
  an IBR method developed by Debevec et
  al.(96, 98)

8
Geometry Proxy Extraction (1/5)

• The proxy a simple geometric representation of
  the person
• Mesh of triangles
• Map textures onto the triangles faces
• Extraction a pair of cameras
• May be two of texture cameras

9
Geometry Proxy Extraction (2/5)

• Algorithm- Plane Parallax (Kumar, 94) method
• Robust Plane Fitting
• Stereo Feature tracking

10
Geometry Proxy Extraction (3/5)

• Robust Plane Fitting
• Segmentation
• Points on
• the Silhouette
• Matching
• Fit Plane
• Distance
• Std. Dev.
• Remove distant points
• Repeat

11
Geometry Proxy Extraction (4/5)
Stereo Feature TrackingKLT tracker
(Kanade-Lucas-Tomasi, 91, 94)
12
Geometry Proxy Extraction (5/5)
Triangulation Proxy formation


13
Rendering Images with VDTM View Dependent
Texture Mapping (1/2)

• VDTM requires a good spatial geometry of the
  objects (Debevec98)
• Our system Buehler et al (2001) and Heigl et al
  (1999)
• Spatial geometry proxy
• Textures live images (from cameras)
• Textures are mapped onto the proxy according to
  the viewpoint

14
Rendering Images with VDTM View Dependent
Texture Mapping (2/2)

• Virtual Camera at Viewpoint D
• Texture from cameras Ci mapped onto the
  triangles faces
• Blending weights in vertex V
• Angles qi, used to compute the weights values
• wi exp(-qi2/2.s2)

15
Results (1/3)

• Geometry Proxy Image rendering with VDTM

• Geometry Proxy

• Image rendered with 2 texture cameras

• Image rendered with 4 texture cameras

16
Results (2/3)

• Cameras Firewire IEEE 1394 SONY
• Frame size 320x240 pixels
• Geometry/Renderer PC
• Processor Intel Pentium4, 2.2 GHz
• Graphics card nVidia GeForce3
• Video Frame Rate 3-4 fps

17
Results (3/3)

• 3D Video Teleconferencing
• Movie

18
Conclusions

• A hybrid system for 3D Video Teleconferencing
• Fast geometry proxy extraction, using a robust
  plane fitting method and stereo feature tracking,
  combined with view-dependent texture mapping
• Real-time demonstration with personal computers
  and commodity graphics card

19
Future Work

• Vision-based head-tracking for viewpoint control
• Increase of the frame rate with faster
  segmentation

20
Acknowledgements

• Herman Towles
• Office of the Future (OOTF) group at UNC-CH
• U.S. Dept. of Energy and Sandia National Labs.
• U.S. National Science Foundation
• NEC/CPDIA

21
Thank you!

• http//www.cs.unc.edu/Research/ootf/
• http//www.lsi.usp.br/kurash/