Practical Surface Light Fields

1
Practical Surface Light Fields

• Greg Coombe
• Advisor Prof. Anselmo Lastra

2
Traditional Graphics

• Surface represented as mesh
• Appearance is defined as a function over the
  surface
• Simple color at each point
• Complex shaders

Anistropic BRDFs Kautz99
3
Image-Based Modeling

• Surface appearance is created from images
• Acquire appearance rather than simulate it

Levoy96, Gortler96
4
Image-Based Modeling

• Demand for photorealistic imagery

Matrix Reloaded, ESC Entertainment
Spiderman 2, Sony Imageworks
5
Image-Based Modeling

• Scanners and high-speed cameras enable us to
  capture large amounts of data

DeltaSphere 25,000 samples/sec
Bumblebee 1024x768 with depth _at_ 18fps
Flea 1024x768 _at_ 30fps
3rdTech
PointGrey
6
Surface Light Fields

• A Surface Light Field (SLF) represents the
  appearance of a model with known geometry and
  static lighting

(?,F)
surface position
(u,v)
viewing direction
Miller98,Wood00
7
How to create a SLF

• Step 1. Acquire the geometry of object
• Step 2. Acquire images of object
• 100-300 is typical

8
How to create a SLF

• Step 3. Map images onto geometry and compress
• Research focus
• Step 4. Render

Pitcher 3k triangles
9
SLF - Batch Process
Surface Light Field Construction
GPU Renderer
disk
X

X
X
Chen02, Hillesland03
10
My research focus

• Identify 3 problems of Surface Light Fields
  construction
• Batch construction
• Missing data
• Matching desired lighting
• Address these limitations
• Human-in-the-loop construction
• Scattered data approximation
• Capture under virtual illumination

11
Thesis

• Three major problems with Surface Light Field
  construction, (1) lack of feedback, (2)
  difficulty handling missing data, and (3)
  matching desired illumination, can be addressed
  by (1) enabling incremental construction, (2)
  employing scattered data approximation
  techniques, and (3) capturing under virtual
  lighting environments.

12
3 Limitations of SLF Capture

• 1. Lack of Feedback
• All images must be available before SLF
  construction starts
• Quality cannot be determined until after process
  is completed

• My Approach
• Incremental SLF construction using Online SVD
• Data-driven heuristic to provide feedback

13
3 Limitations of SLF Capture

• 2. Missing Data
• Occlusions cause holes in data
• Representation requires full set of data

• My Approaches
• Imputation using Online SVD
• Scattered Data Approximation using Incremental
  Weighted Least Squares

14
3 Limitations of SLF Capture

• 3. Matching Desired Lighting
• SLF has fixed lighting
• Desired lighting must be physically duplicated at
  capture
• Ex. Museum of objects

• My approach
• Design an inexpensive capture device to capture
  SLFs under virtual illumination

15
Outline

• Introduction
• Online Construction
• Online SVD
• Data-driven Heuristic
• Missing Data
• Imputation
• Incremental Weighted Least Squares
• Virtual Illumination
• Conclusion and Future Work

16
OpenLF
Surface Light Field Construction
GPU Renderer
disk
X

X
X
Chen02, Hillesland03
17
SLF - Online Process
GPU Renderer
Incremental SLF Construction
1024x768 _at_ 15fps
18
Video Camera Input

• Track position and orientation of camera using
  fiducials in environment

• Real-time tracking enables free-form capture

OpenCV, ARToolkit
19
SLF Representation

• Discretize function over surface and hemisphere

L
Nishino01, Chen02
20
SLF Representation

• Decompose using Singular Value Decomposition (SVD)

Nishino01, Chen02
21
SLFs using batch SVD

• Problems
• Requires that the entire data set be available
• Difficult to locate undersampled regions
• Requires recomputation when new images are added

h
g
L

SVD
22
Online SVD

• Online SVD Brand02 is a incremental SVD
• Update output matrices one sample at a time
• Advantages
• Never store the entire data matrix at once
• Stream images

h
g
L1
L2
Ln

...
Online SVD
23
Comparison
24
Error
MSE
rank
Reference SVD
Online SVD
25
Feedback

• Aid the user in capturing surface light fields
• Direct attention towards undersampled areas
• Use a data-driven error heuristic

• e0 surface error
• Projection error of SVD
• e1 hemisphere error
• Sampling density

red indicates more data needed
26
Results
SLF after incorporating new image
Captured image
SLF before incorporating new image
Online Construction of Surface Light Fields.
Coombe G., Hantak C., Lastra A., and Grzeszczuk
R. Eurographics Symposium on Rendering 2005
27
Results
28
Timing

• Time to incorporate a new image

Intel 1.8Ghz Nvidia GeForce5900
29
Outline

• Introduction
• Online Construction
• Online SVD
• Data-driven Heuristic
• Missing Data
• Imputation
• Incremental Weighted Least Squares
• Virtual Illumination
• Conclusion and Future Work

30
Missing Data
Acquired image
Rendered model

• What are the black spots?

31
Missing Data

• Matrix factorization approach requires fully
  resampled data matrices
• Missing data is a common problem
• Occlusion, meshing errors
• If data is missing, must throw out entire column

x
x
L
32
Approach 1 Imputation

• Could fill holes using zeros or mean
• Better to impute values
• Use the current Online SVD approximation to
  generate values
• In practice, need about 5-10 initial images, and
  at least 50 coverage

Missing data in red
Imputed
33
Resampling

• Samples are at arbitrary locations in domain due
  to geometry and camera

?
?
34
Approach 2

• Change the SLF representation
• Represent lighting as non-linear combination of
  low-degree polynomials (Weighted Least Squares)
• Develop Incremental Weighted Least Squares
• Adaptive and Hierarchical

35
Representation
L1(T, F)
L2(T, F)
L3(T, F)
L4(T, F)
36
Least Squares Fitting

• Find the best polynomial approximation to the
  input samples
• best means minimizes sum of squared differences
• the coefficients are determined by solving a
  linear system

input samples
reconstruction
37
Least Squares
reconstructed function
input samples
domain

• Problem LS is a global approximation

38
Weighted Least Squares
reconstructed function
polynomial approximations
input samples
domains
centers
39
Weighted Least Squares
T
x
x
x
x
F
40
Incremental WLS

• Feedback is important in SLF construction
• As each image is captured, it must be
  incorporated into the representation
• How do we build this incrementally?
• Centers are fixed
• Domains are variable

41
Adaptive Construction

• Start out with large domains
• Adaptive shrink as more points arrive

x
x
x
x
x
x
x
x
42
Hierarchical Construction

• Start out with a single domain
• Subdivide as more points arrive (quadtree)

x
x
x
x
x
x
x
x
x
x
x
43
Hierarchical and Adaptive

• Hierarchical
• The highest level (1x1) is similar to Polynomial
  Texture Mapping Malzbender01
• Fast at first, slows down as refines
• Hierarchy is expanded for GPU rendering
• Adaptive
• Slow at first, accelerates as domains shrink
• Can handle arbitrary number of domains

44
Results
Hierarchical Construction, First 10 images
45
Results
4K patches
29K patches
Comparison with image not in training set, 14K
patches
An Incremental WLS Approach To Surface Light
Fields. Coombe G., and Lastra A. GRAPP 2006
46
Results
Pitcher model, 65 images
47
Performance

• 0.5 - 2 seconds per image for hierarchical
  construction
• 0.5s for 4K bust model
• 2s for 30K pitcher model
• 95 is Least Squares Fitting
• Adaptive is 2-3x more expensive
• Rendering is 30fps

48
Comparison

• Online SVD
• Distributes variance across surface and
  hemisphere
• Problems with missing data
• Incremental WLS
• Order-of-magnitude larger
• Construction of WLS is 20-40 slower
• Higher lighting frequency
• Rendering speed is realtime (30-60fps)

49
Outline

• Introduction
• Online Construction
• Online SVD
• Data-driven Heuristic
• Missing Data
• Imputation
• Incremental Weighted Least Squares
• Virtual Illumination
• Conclusion and Future Work

50
Lighting Environments

• Games and movies use high-resolution lighting
  environments
• How can we incorporate an object into these
  environments?

Half-life Map By Crinity
Uffizi Gallery, Florence
Grace Cathedral, San Francisco
St. Peter's Basilica, Rome
debevec.org, Crinity
51
Matching Illumination

• How do we capture the SLF of an object under a
  desired lighting condition?
• Physically re-create the illumination using
  lights
• Often used in movies
• Difficult to precisely match, low-resolution
• My approach A projector-camera system to
  simulate lighting environment for SLF capture

52
Projecting Light
Schechner03, Matusik04
53
Physical Setup
54
Multiple Cameras
One Camera
Four Cameras
55
Display

• How do we know what to display on projector?
• Need to determine rays from object to screen

Projector output
Desired lighting environment
56
Calibration
Technique 1. Fast, planar parallelogram screens
57
Calibration

• Technique 2. Slower, arbitrary screens

Planar screen
Corner screen
Calibration of a Surface Lightfield Capture
System. Frahm, J.M., Coombe G., and Lastra A.
ProCams 2006
58
High-Dynamic Range

• To match a lighting environment, need to use
  High-Dynamic Range Images
• Split into multiple exposures
• Recombine acquired images
• Requires linear color response for projector and
  camera Ilie05

Debevec97, Cohen01
59
Results
Capturing Surface Lightfields of Real Objects
Under Virtual Illumination. Coombe G., Frahm,
J.M., and Lastra A. Under preparation
60
Results
61
Conclusion

• Identify 3 problems with Surface Light Fields
  construction
• Lack of feedback
• Missing data
• Matching desired lighting
• Address these problems
• Human-in-the-loop construction
• Scattered data approximation
• Capture under virtual illumination

62
Conclusion - Part 1

• Incremental Construction of Surface Light Field
• Provide feedback to user about capture process
• Data-driven quality heuristic
• Online SVD is well-suited for streaming model
• Requires only one pass over data

63
Conclusion - Part 2

• Imputation within Online SVD framework
• Represent the surface lightfield using Weighted
  Least Squares approximation
• Modify WLS for the incremental framework
• Adaptive and Hierarchical

64
Conclusion - Part 3

• Capturing a SLF under desired illumination
  conditions
• Design an inexpensive capture device for
  illumination environments
• Develop two physical calibration techniques for
  projector screens
• Use HDR techniques with Multiplexed Illumination
  to capture HDR SLF

65
Limitations

• Compression rate of Incremental WLS
• Many parameters determined a priori
• Surface and hemisphere resolution, geometric
  quality, placement of WLS centers, SVD rank
• Quality is highly-dependent upon calibration
• Intrinsic and extrinsic camera calibration,
  registration of geometry with tracking,
  registration of light probe with geometry
• Physical capture difficulties
• Extreme angles, light scattering from walls,
  camera blocks light, fidicual board blocks camera

66
Future Work

• SLF is tightly coupled to geometry
• Causes problems when geometry is poor
• Loosely-coupled SLF representation
• Capture geometry and SLF at same time
• Deformable geometry such as faces
• Missing Data
• Use Texture Synthesis algorithms Liu01

67
Future Work

• Both Online SVD and Incremental WLS were
  implemented on CPU
• Initial interest in Online SVD was due to
  streaming nature
• GPU implementation would significantly accelerate
  process
• SVD is a fundamental algorithm in many scientific
  areas
• How does Online SVD affect approach?

68
Future Work

• Lack of Artist Control
• Can only capture objects that physically exist
• Acquire a database of BRDFs Matusik03
• Editing SLFs Wood00, Chen05
• Extend Incremental Construction methodology to
  spatially-varying BDRF
• Order-of-magnitude more data
• Construction process is much longer

69
Publications

• Capturing Surface Lightfields of Real Objects
  Under Virtual Illumination. Coombe G., Frahm,
  J.M., and Lastra A. Under preparation
• Calibration of a Surface Lightfield Capture
  System. Frahm, J.M., Coombe G., and Lastra A.
  ProCams 2006.
• An Incremental WLS Approach To Surface Light
  Fields. Coombe G., and Lastra A. GRAPP 2006.
• Online Construction of Surface Light Fields.
  Coombe G., Hantak C., Lastra A., and Grzeszczuk
  R. Eurographics Symposium on Rendering 2005.
• Reordering for Cache Conscious Photon Mapping.
  Joshua Steinhurst, Greg Coombe, Anselmo Lastra.
  Graphics Interface 2005.
• SKIT A Sketching Instruction Tool. Coombe G.,
  Salomon B. Edutainment 06.
• Global Illumination using Progressive Refinement
  Radiosity. Coombe G., Harris M. GPU Gems II
• Radiosity on Graphics Hardware. Coombe G, Harris
  M, and Lastra A. Graphics Interface 2004.
• Physically-Based Visual Simulation on Graphics
  Hardware. Harris M, Coombe G, Scheuerman T, and
  Lastra A. Workshop on Graphics Hardware 2002.

70
Acknowledgements

• My advisor, Anselmo Lastra
• Committee members Gary Bishop, Radek Grzeszczuk,
  Leonard McMillan, Marc Pollefeys
• Co-authors and collaborators Mark Harris, Andrew
  Nashel, Thorsten Sheuermann, Josh Steinhurst,
  Chad Hantak, Brian Salomon, Justin Hensley,
  Jan-Michael Frahm

71
Acknowledgements

• ARGH! (Advanced Research in Graphics Hardware)
  Group
• EVE Group
• Google

72
Acknowledgements

• NVIDIA Graduate Student Fellowship
• Intel OpenLF Group Radek Grzeszczuk, Wei-Chao
  Chen, Jean-Yves Bouguet, Sergey Molinov
• http//sourceforge.net/projects/openlf/
• Staff David Harrison, John Thomas, Herman
  Towles, Bil Hays, Alan Forrest, Mike Carter, Mike
  Stone, Janet Jones, Sandra Neely

73
Friends and Family
74
Questions?
75
This slide was intentionally left blank
76
Surface IBR Taxonomy
Bidirectional Texture Function (BTF)
fixed lighting
fixed viewpoint
uniform material
Surface Light Field (SLF)
Surface Reflectance Field (SRF)
Bidirectional Reflectance Distribution Function
(BRDF)
Eurographics STAR 2004
77
Surface Reflectance Fields

• A surface reflectance field represents the
  appearance of a model under arbitrary lighting
  from a fixed viewpoint

(?,F)
surface position
(u,v)
lighting direction
78
SRF Capture
Hawkins05
79
Online SVD Algorithm
80
Online SVD

• Problem Multiple small rotations can accumulate
  error
• Brand proposes splitting output matrices to avoid
  accumulating error
• Advantages

81
Online SVD Performance

• O(n2r) lt O(n3)
• Due to small working set, most data in cache

82
Online SVD Convergence
Averaged over twenty random vertices
83
Online SVD Error
MSE
Rank
84
Online SVD Sensitivity
85
SLF Rendering
Nishino01, Chen02
86
Data-driven quality metric

• Surface quality
• The difference between the approximation and the
  new image
• Computed as projection error of SVD
• Smoothed with exponential falloff
• Viewpoint Quality
• The density of samples in the neighborhood
• Computed as area of the triangles in the Delaunay
  triangulation of viewpoints

87
Higher Dimensions

• We would like to extend Online SVD to
  higher-order factorizations
• Moveable light source
• Introduces another dimension of data
• Requires tensor product expansion

View maps
Surface maps
Light maps
88
Data-driven quality metric

• How well does heuristic predict where more data
  is needed?

89
WLS Center Placement

• The centers are determined a priori
• Where should they go?
• With knowledge of lighting and/or reflectance,
  can place centers intelligently

90
Meshing Errors

• What happened?

good
bad
91
Camera Overlap

• Problem Cameras overlap
• Solution
• Compute 4 half-planes
• for each camera
• On GPU Test if pixel ray is inside all 4
  planes of a previous camera position

image showing occlusion regions
92
GPU Implementation
93
Previous Work

• Regular parameterizations light field data
  Levoy96, Gortler96
• Sparse, scattered data Debevec96, Debevec98,
  Buehler01
• BRDF capture systems Dana99, Lafortune97,
  Marschner99, Debevec01, Gardner03
• Function fitting
• Lafortune BRDF McAllister02, Torrance-Sparrow
  BRDF Sato97, Clusters of BRDFs Lensh01,
  Homomorphic Factorization McCool?, Bi-quadratic
  polynomials Malzbender01
• Online Methods
• Fixed viewpoint, progressive refinement
  Matusik04
• Adaptive Meshing of light field Schirmacher99
• Streaming non-linear optimization Hillesland04
• Data Mining Brand02, Roweis97

94
Scattered Data Approximation

• Scattered Data Approximation in lightfields
• Unstructured Lightfields Buehler01
• Tesselation of pure lightfield
• Polynomial Texture Maps Malzbender01
• Fit polynomials to set of images
• Radial Basis Functions Zickler05
• Interpolate sparse reflectance data

95
System Implementation
96
Weighted Least Squares

• Problem LS is a global approximation
• Solution Divide domain into multiple LS
  approximations, and combine to get global
  approximation
• Use a set of low-degree polynomials
• Non-linear blending (Partition of Unity)
• Good discussion in Scattered Data Approximation,
  Holgar Wendland

97
Results
-2
0
-2
Exposure
98
Bust model, 75 training images