Video And Mesh Processing for 3D Cinematography

1
Video And Mesh Processing for 3D Cinematography

• Remi Ronfard
• INRIA Rhone-Alpes
• VAMP Associate Team
• March 2006

2
Plan of presentation

• What is 3D cinematography?
• Goal statement for VAMP
• People in VAMP
• Work Items in VAMP
• Joint activities in 2005
• Joint activities in 2006
• Future work and applications

3
What is 3D Cinematography?

• 3D cinematography, also sometimes called
  free-viewpoint video, is the process of building
  3D models of dynamic scenes from multiple video
  camera inputs.
• This is an area of active research with enormous
  potential applications in education,
  entertainment, interactive television,
  video-conferencing, etc.
• It is one of three scenarios actively
  investigated by MPEG for future extensions of
  their audio and video coding standards to 3D
  (3DAV).

4
Goal statement

• Our teams at INRIA and Brown University have
  separately developped methods and tools for 3D
  photography
• Moving to 3D cinematography raises difficult
  technological and scientific challenges, where we
  need to collaborate actively.
• In this new application, the processing pipeline
  must be fully automated, so that 3D
  reconstruction can be performed at video frame
  rates.

5
Goal statement

• The goal of the associate team is to contribute
  to 3D cinematography by joint work in two crucial
  areas
• real-time video processing for large networks of
  camera systems, using distributed/parallel and
  embedded architectures
• real-time mesh processing for reconstruction,
  deformation, texturing and view interpolation of
  3D objects from multiple video, using
  multi-resolution models amenable to efficient,
  scalable compression.

6
Real-time video processing for multiple cameras

• The goal of the associate team is to develop and
  share software and hardware for the 3D
  cinematography systems built separately by INRIA
  and Brown University.
• Parallelization/embedding of dedicated algorithms
  for solving problems of camera calibration,
  background subtraction, feature extraction,
  feature matching and so forth.
• Extend algorithms to dynamic cameras at multiple
  resolutions and viewpoints.

7
Real-time mesh processing of 3D objects

• Our goal is to obtain high quality approximations
  of the deforming surfaces of objects in a dynamic
  scene at any point in time, and to render them by
  projecting video textures on them.
• Subdivision surfaces and their multi-resolution
  extensions are the right tools for achieving both
  high-quality renderings, which are necessary to
  convey the illusion of reality, and
  high-compression ratios, which are necessary to
  overcome the massive amounts of data involved in
  multiple-view video.
• This necessitates substantial extensions of the
  current state-of-the art in both computer vision
  and graphics, since the projective constraints
  arising from multiple cameras are essentially
  non-linear.

8
Team Composition

• The associate team is composed of 5 senior
  researchers, 10 PhD students and 2 engineers.
• Gabriel Taubin and 4 students at Brown (EE)
• Peter Sibley, Daniel Crispell, Yong Zhao, Douglas
  Lanman
• Chad Jenkins and 3 students at Brown (CS)
• Matt Loper, German Gonzales, Dan Grollman
• Rémi Ronfard and 2 students at INRIA (MOVI)
• David Knossow, Daniel Weinland
• Edmond Boyer and 1 student (MOVI)
• Clement Menier
• Frederic Devernay, Loic Lefort and Herve Mathieu

9
Previous work

• Gabriel Taubin and Rémi Ronfard has a close
  collaboration at IBM TJ Watson Research Center in
  1992 and 2000 on efficient data structures and
  algorithms for manipulating multi-resolution
  curves and surfaces
• Gabriel Taubin, Remi Ronfard Implicit simplical
  models for adaptive curve reconstruction. IEEE
  transactions on Pattern Analysis and Machine
  Intelligence, 3(18), pp. 321-325, 1996
• Ioana Martin, Remi Ronfard, Fausto Bernardini
  Detail-Preserving Variational Surface Design with
  Multiresolution Constraints. International
  Conference on Shape Modeling and Applications,
  June 2004.

10
Team Complementarity

• The MOVI team at INRIA brings expertise in camera
  calibration and synchronization multiple-view
  stereo and shape from silhouette real-time and
  distributed systems action recognition.
• The two teams at Brown bring complementary
  expertise in embedded systems programming and
  image-based rendering, mesh processing, machine
  learning and behaviour-based robotics.

11
Initial work items

• Distributed video capture, including camera
  synchronization and calibration
• Multi-resolution object surfaces from silhouettes
  using subdivision surfaces, with applications on
  modeling deformable objects and interpolating new
  viewpoints
• Human motion tracking and action understanding

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
Blinky Distributed Camera Control
16
Camera Recording and Playback
17
Camera Calibration
18
3D models from multiple views

• Visual Hulls and Shapes (Edmond Boyer)

19
From multiple cameras to 3D
20
(No Transcript)
21
Beyond visual hulls and towards multi-resolution
(work in progress)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
Markerless articulated motion
27
Human Action Recognition
28
Human action recognition example
29
Recovering body skeleton from video (work in
progress)
30
Achievement in 2005

• June/July 2005 - The initial associate team
  meetings were very successful and appealed to PhD
  students enormously on both sides.
• Those meetings revealed that both labs are
  heading very rapidly towards very large camera
  arrays (100's at INRIA, 1000's at Brown).
• Although the initial investment in terms of
  hardware and software developments are still
  high, our collaboration will save efforts and
  expenses.
• INRIA recording and playback software has been
  deployed at Brown for initial experiments,
  calibration and background subtraction to follow

31
Exchanges in 2005

32
Scheduled exchanges in 2006

• Chad Jenkins will visit INRIA in April
• continue ongoing collaboration with Clement
  Menier on medial-axis reconstruction from
  multiple silhouettes
• start a new collaboration with Daniel Weinland on
  motion modeling and action recognition.
• In June, Gabriel Taubin and Remi Ronfard host an
  international workshop on 3D cinematography at
  CVPR in New York City.
• More student exchanges throughout summer (thanks
  to NSF)
• multi-resolution surface from silhouettes
• multi-view stereo, deformable surfaces

33
Scheduled exchanges for 2006

   
34
Future work Cinematized Reality

• Track and follow action
• Tracking paths of multiple people
• Tracking articulated motion
• Tracking activities
• Generate new views
• View-dependent textures mapped to 3D model
• Camera placement problem - How to choose the best
  views ?
• Generate new movies in 2D or 3D

35
Applications

• Digitizing Live Performances
• Cultural Heritage and Education
• Interactive Television, Games
• Holographic theatre
• Tele-presence
• Remote actors, dancers, musicians
• Theatre of operations in medecine and military
• Producing content for 3D cinema (2010) and 3D
  television (2020)

36
(No Transcript)