ModelDriven VideoBased Rendering for Vehicles

1
Model-Driven Video-Based Rendering for Vehicles

• Ismail Oner Sebe
• Suya You, Ulrich Neumann
• Computer Graphics and Interactive Technologies
  Laboratory

2
Outline

• Model-Based 3D Modeling from image(s)
• Camera Calibration
• Texture Mapping
• User interaction
• Part-Based Model Modification
• Video-Based Rendering
• Model-Aided Tracking
• Model-Aided Pose Estimation
• Environment Modeling
• Results and Discussion

3
Model-Based 3D Modeling

• Model-Based Modeling
• Architectural Structures Lie99, Hoe05
• Humans Car03
• Motivation for our modeling technique
• Rapidly create high quality 3D model of objects
  (one object class at a time) from a single
  un-calibrated image
• Rapid
• High quality
• Single image
• Un-calibrated/unknown camera

4
Part-Based Generic Model

• Prior knowledge an instance of class
• Part-based generic model
• Part San01
• Unique
• Bounded
• Closed
• Connected
• Compact
• For vehicles
• Front/Middle/Back
• Bottom/Middle/Top
• Left/Center/Right

32 Parts
5
Camera Calibration

• Vanishing Points Approach (architectural
  modeling)
• Assumptions
• Zero skew
• Fixed aspect ratio
• No radial distortion
• Fixed camera center
• 2 vanishing points (8 image clicks)
• Arbitrary scaling
• 3 Scaling points from tires (3 image clicks)
• Total 11 image clicks for 3x4 projection matrix
• 2D image with 3D model overlay

6
Texture Mapping

• 3D Model ? 2D Image
• Z-Depth buffering (visibility analysis)
• Texture
• Color estimation
• Fill the gaps in the texture map
• Assign to the parts without texture

7
User Interaction
v
p
2D Input

• Click and Drag (in 2D)
• CLICK User clicks to select a part (p)
• DRAG Creates a vector, (v) (2D constraint)
• A 3D vector, V, is estimated
• Satisfy 2D constraint
• Smallest 3D movement

8
Part-Based Modification I

• Movement in 3D ? V
• Connectedness and speed
• Model modification
• by part
• by boundary
• Parts can move in 4 ways
• Interpolate Scattered Data Interpolation
• Triangle Based SDI
• Thin Plate SDI
• Shepard Interpolator

9
Part-Based Modification II
Gray DoNotMove Red MoveSame Green
Interpolate
10
3D Modeling Results I
3D model with estimated color
3D model with Texture
Input 2D Image
11
3D Modeling Results II
3D model with estimated color
3D model with Texture
Input 2D Image
12
Part-Based 3D Modeling
13
Video-Based Rendering

• Previous work
• The Matrix (1999)
• Eye Vision system in Superbowl XXXV (2001)
• UC Berkeley Deb98
• AVE Neu04
• Our problem
• Given the 3D model of the scene and the video,
  update the 3D model such that 2D/3D coupling is
  satisfied
• Stationary background
• Fixed camera
• Single foreground vehicle
• Planar move (2 translation and one rotation)

14
Model-Aided Pose Estimation

• Stationary camera planar movement 3 unknowns
  (Ry, Tx and Ty)
• Nonlinear minimization vs. Expectation
  Maximization (EM)
• Starting point is close to correct solution
• Small number of parameters to estimate
• Filtering of both parameters and tracking points
• Re-projection of tracking points via 3D/2D
  coupling

15
Model-Aided Tracking

• Tracking of 3D patches in 2D
• Minimization of normalized correlation
• Computationally equivalent to 2D motion search
• More robust
• Less drift

2D Patches
3D Patches
16
Environment Modeling

• Simplistic world model
• World is composed of walls and ground
• Ground is inferred from tires
• Walls are perpendicular to the ground
• Smart subdivision of triangles

17
Free Viewpoint Video

• World is textured with estimated background image
• Car is textured with the method from IBR
• Texture is filtered in time to prevent flicker in
  video
• Shadow is added later (optional)
• Sun position is entered manually
• User can control time and camera location at any
  point of the video/rendering

18
Video-Based Rendering
19
Discussion

• 3D Modeling System
• Pros
• Fast (1-2) minutes
• High-quality 3D models
• Single uncalibrated image
• Intuitive user interface
• Cons
• Create generic model for every class
• Such a model may not exist
• Need at least one photo

• Video Based Rendering
• Pros
• Almost automatic creation
• Good quality in rendering
• Fast
• Cons
• No outlier detection
• Stationary camera
• Restricted object movement
• Single foreground object

20
Conclusion and Future work

• We present a model-driven video based rendering
  system for vehicles
• Our system is fast and doesnt require any
  information about camera
• It is able to create photorealistic renderings in
  mere minutes
• Current/Future Work
• Automatic car modeling Detection of camera pose,
  detection of parts, etc.
• Other object types Insects (joint work with SFSU
  Ilmi Yoon)
• Multiple vehicle rendering scenarios Occlusion,
  texture atlas

21
References

• Deb98 P. Debevec, G. Borshukov, and Y. You,
  Efficent video-Dependent Image-Based Rendering
  with Projective Texture-Mapping. In 9th
  Eurographics Rendering Workshop, Vienna, Austria,
  June 1998.
• Neu04 U. Neumann, S. You, J. Hu, B. Jiang, I.O.
  Sebe, Visualizing Reality in an Augmented
  Virtual Environment. Presence Teleoperators and
  Virtual Environments Journal, pp. 222-233, April
  2004.
• Car03 J. Carranza, C. Theobalt. M. Magnor, H.P.
  Seidel, Free-Viewpoint Video of Human Actors.
  SIGGRAPH03, San Diego, CA, 2003.
• Lie99 D. Liebowitz, A. Criminisi, A. Zisserman,
  Creating architectural models from images. In
  Eurographics 99, pages 39-50, 1999.
• Hoe05 D. Hoeim, A.A. Efros, M. Hebert,
  Automatic Photo Pop-up.SIGGRAPH05, Los Angeles,
  CA, 2005.
• San01 P.V. Sander, J. Snyder, S. Gortler, H.
  Hoppe. Texture Mapping Progressive Meshes.
  SIGGRAPH01,Los Angeles, CA, 2001.