Direct-to-Indirect Transfer for Cinematic Relighting

1
Direct-to-Indirect Transfer for Cinematic
Relighting

• Milos Hasan (Cornell University)
• Fabio Pellacini (Dartmouth College)
• Kavita Bala (Cornell University)

2
Introduction

• Cinematic Relighting Gershbein 2000, Pellacini
  2005
• Interactive local light movement, fixed camera
• Procedural shaders, various materials
• High-complexity scenes
• Only direct illumination
• The goal of our system
• Add multiple bouncesof indirect illumination

3
Direct-to-Indirect Transfer

• Direct illumination from local lights
• Use known techniques
• Deep frame-buffer, shadow maps
• Indirect illumination key idea
• Precompute direct-to-indirect transfer matrix
• Compute indirect from direct on the fly

4
Interactive Demo
5
Related Work

• Cinematic relighting engines
• Gershbein 00 Pellacini 05, Tabellion 04
• Precomputed radiance transfer
• Sloan 02, 03, 05 Kautz 02 Ng 03, 04 Liu 04
  Wang 04, 05 Annen 04, etc.
• Linear combinations of full solutions
• Kristensen 05 Dorsey 91 Debevec 00 etc.

6
Related Work

• Sparse sampling
• Walter 99, 02 Bala 99, 03 Ward 99 Tole 02
  Nijasure 03 Gautron 05 Dayal 05 Simmons 01
  etc.
• Hierarchical / instant radiosity methods
• Hanrahan 91 Keller 97 Drettakis 97
  Christensen 97 Dachsbacher 05, 06 etc.
• Wavelet radiance transport
• Kontkanen 06 (concurrent work)

7
Outline

• Key concepts
• Transfer matrix
• Efficient precomputation
• Efficient compression
• Efficient evaluation
• Results
• Conclusions and Future work

8
Key Concepts

• View samples
• Gather samples
• Transfer A large matrix
• Contributions of gather samples to view samples

9
View Samples
Camera
A scene
View samples
10
Gather Samples
Gather samples
A scene
11
Transfer Matrix Elements
Tij ??
Gather sample j
View sample i
12
Direct Illumination

• Standard techniques GPU, shaders, shadow maps

Direct on gather samples
Direct on view samples
13
Conceptual Overview
Direct on view
Transfer matrix
Final
Direct on gather
Indirect on view
14
Outline

• Key concepts
• Transfer matrix
• Efficient precomputation
• Efficient compression
• Efficient evaluation
• Results
• Conclusions and future work

15
Why Is Precomputation Hard

• Transfer matrix is huge!
• View samples640 x 480 300k rows
• Gather samples 64k columns
• 19 billion elements
• Columns
• Images with 1 light and full global illumination

64k gather
300k view
Full global illumination images
16
Multi-bounce Final Gather

• Split full transfer matrix
• Multiple-bounce matrix (lower accuracy)
• Final gather matrix (higher accuracy)

Full transferMany gather-to-view
Multi-bounceMany gather-to-gather
Final gatherOne gather-to-view
17
Final Gather Matrix Elements
Gather sample j
Fij ??
Single-bounce contribution
View sample i
18
Multi-bounce Matrix Elements
Gather sample i
Mij ??
Gather sample j
19
Multi-bounce MatrixSparse Approximation

• Photon mapping Jensen 96 analogy
• But, dont know light positions
• Shoot photons from gather samples instead!
• Photons carry gather sample ID
• Like standard photon mapping with 64k lights!

20
Multi-bounce MatrixComputing by Photon Mapping
A gather sample
i-th row of M ??
K nearest photons
Gather sample i
21
Conceptual Overview, Updated
Multi-bounce matrix
Add direct
Direct on gather
Indirect on gather
Dir Ind on gather
Final gather matrix
Add direct
Indirect on view
Final
22
Outline

• Key concepts
• Transfer matrix
• Efficient precomputation
• Efficient compression
• Efficient evaluation
• Results
• Conclusions and future work

23
Defining a Wavelet Basis

• Want to use wavelets
• How? There is no obvious domain to apply them
• Unstructured cloud of gather samples
• Still possible to define wavelets on the cloud!
• Flatten gather cloud to a 2D array
• Maintain coherence
• Use 2D Haar wavelets on the 2D array

24
Defining a Wavelet Basis
64k
256 x 256
256 x 256
Gather samples
Flattened in a 2D array
Wavelet transform
25
Hierarchical Partitioning
26
Tree Arrangement
27
Wavelet Compression

• Transform rows into Haar wavelets
• Remove small coefficients

Final gather matrix
Multi-bounce matrix
28
Columns of the Wavelet-Transformed Matrix
Final gather matrix in wavelets
Images lit by wavelet lights
29
Full System Overview
Multi-bounce matrix in waveletspace
Wavelet Xform
Wavelet Xform
Final gather matrix in waveletspace
Add direct
Dir Ind on gather in wavelet space
Indirect on view
Final Image
30
Outline

• Key concepts
• Transfer matrix
• Efficient precomputation
• Efficient compression
• Efficient evaluation
• Results
• Conclusions and future work

31
Sparse Matrix-Vector Multiplication
Sparse matrix
Linear combination of columns
Sparse vector
32
Sparse Matrix-Vector Multiplication on the GPU

• Problem
• Columns are sparse themselves
• How to represent on the GPU?
• Solution
• Non-zero elements tend to cluster
• Cut out rectangular blocks
• Pack blocks into texture atlases
• Converted problem to blending!

33
Scene Still Life
Precomputation 1.6 hours
11.4 18.7 fps
Polygon count 107k
34
Scene Temple
Precomputation 2.5 hours
8.5 25.8 fps
Polygon count 2 million
35
Scene Hair Ball
Precomputation 2.9 hours
9.7 24.7 fps
Polygon count 320k
36
Scene Sponza Atrium
Precomputation 1.5 hours
Frame-rate 13.7 24.9
Polygon count 66k
37
Comparison
Our system 8-25 fps (2.5 hr precomputation)
Monte Carlo path tracer 32 hours
38
Conclusions and Future Work

• Conclusion
• Extend cinematic relighting with indirect
  illumination
• Interactive performance (GPU), complex scenes
• Arbitrary light shaders, efficient precomputation
• Future work
• Environment mapping
• Subsurface scattering
• Moving camera

39
Questions?

• Acknowledgements
• NSF grant CCF-0539996
• Bruce Walter (code, support)
• Veronica Sunstedt, Patrick Ledda (Temple)
• Marko Dabrovic (Sponza atrium)
• Cornell animation TAs (Still life)
• Pixar, NVidia
• E-mail mhasan_at_cs.cornell.edu