ImageBased Rendering using Hardware Accelerated Dynamic Textures

1
Image-Based Rendering using Hardware Accelerated
Dynamic Textures

• Keith Yerex
• Dana Cobzas Martin Jagersand

2
Motivation

• Rendering

• Traditional geometry based techniques
• detailed 3D model texture
• hard to achieve photorealism

• Image-based models
• non-geometric model from images
• practically hard to apply

3
Challenges

• Hard to generate detailed 3D models
• Texturing from images require very precise
  alignment with the model
• Rendering arbitrary views using IBM requires
  dense sample of the plenoptic function
• IBR techniques dont deal with dynamic scenes

4
Overview
Training
Model
New view
I1
It
Structure P
New pose (R a b)



(R1 a1 b1) (Rt at bt)
Motion params


Texture basis
Warped texture
y1 yt
Texture coeff
5
Structure from motion

• Input Structure from set of corresponding
  points tracked in a set of images
• Assumptions
• Static scene
• Camera model
• injective ? perspective ? weak perspective ?
  orthographic
• Estimated model
• projective ? affine ? metric ? euclidean

6
Structure from motion
Tracked features
poses
structure
Structure from motion algorithm
7
SFM algorithms

• Few images, perspective camera, precise
  calibration
• epipolar constraint
• trilinear tensor
• Long motion, affine or perspective structure
• factorization methods

8
Metric structure

• Weak perspective camera
• Extension of Tomasi Kanade factorisation
• algorithm
• Extract affine structure
• Relation between the sffine structure and camera
  coordinate frame
• Transform the structure into metric (unit pixel
  size)

9
Weak perspective projection

• N points
• Normalized with respect to centroid
• Rank theorem
• Factorization

10
Metric constraints

• Extract motion parameters
• Eliminate scale
• Compute direction of camera axis k i x j
• parameterize rotation with Euler angles
• Model P Reprojection
• Pose x (r,s,a,b)

11
Dynamic Textures

• Purpose
• Model image intensity variations due to
• Small geometric errors due to tracking
• Non planarity of real surface
• Non-rigidity of real object
• Pose varying lighting effects
• Non-geometric, mixing of spatial basis

12
Spatial Basis Intro

• Moving sine wave can be modeled
• Small image motion

Spatially fixed basis
2 basis vectors
6 basis vectors
13
Image Variability

• Formally consider residual variation in an image
  stabilization problem
• Optic flow type constraint

14
Structural Image Variability

• Affine warp function
• Corresponding image variability
• Discretized for images

15
Composite Image variability

• Similarily can show that composite image
  variability
• Can be modeled as sum of basis

Struct Depth Non-plan Light Res Err
16
Example Lighting variation
17
Statistical Image Variability

• In practice image variability hard to compute
  from one image
• Instead we use PCA to estimate image variability
  from a large sequence of images
• This yields a transformed basis
• Can estimate linear model J
• In practice Delaunay triang bi-linear model

18
Image variability comparison

Derivatives from one picture
Statistically estimated variability
19
Implementation

• Matlab for geometric modeling and prototyping
• mexVision for tracking (30Hz frame rate)
• Hardware accelerated OpenGL for rendering (2.8Hz
  in SW, 18Hz on GeForce 2)
• pthreads and pvm for parallel processing

MATLAB
OpenGL
meXVision
20
Hardware rendering

• Unsigned basis
• Scaling to 8 bit
• Where

21
OpenGL
22
Example Renderings
23
Kinematic arm
24
Geometric errors
static
dynamic
25
Geometric errors
dynamic
static
26
Geometric errors
Dynamic
Static
Dynamic
Static texturing
27
Pixel error
28
Conclusions

• Coarse geometry tractable to estimate
• Errors from small geometric misalignments
  compensated for using dynamic texture
• System runs on consumer PC with web cam and game
  graphics card
• Applications
• Insert characters/objects into games
• Video phone