A SignalProcessing Framework for Forward and Inverse Rendering

1
A Signal-Processing Framework for Forward and
Inverse Rendering
Ravi Ramamoorthi ravir_at_graphics.stanford.edu
Pat Hanrahan hanrahan_at_graphics.stanford.edu
2
Outline

• Motivation
• Forward Rendering
• Inverse Rendering
• Object Recognition
• Reflection as Convolution
• Efficient Rendering Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary

3
Interactive Rendering
Directional Source
Complex Illumination
Ramamoorthi and Hanrahan, SIGGRAPH 2001b
4
Reflection Maps
Blinn and Newell, 1976
5
Environment Maps
Miller and Hoffman, 1984 Later, Greene 86, Cabral
et al. 87
6
Reflectance Space Shading
Cabral, Olano, Nemec 1999
7
Reflectance Maps

• Reflectance Maps (Index by N)
• Horn, 1977
• Irradiance (N) and Phong (R) Reflection Maps
• Hoffman and Miller, 1984

Chrome Sphere
Matte (Lambertian) Sphere Irradiance Environment
Map
Mirror Sphere
8
Complex Illumination

• Must (pre)compute hemispherical integral of
  lighting
• Efficient Prefiltering (gt 1000x faster)
• Traditionally, requires irradiance map textures
• Real-Time Procedural Rendering (no textures)
• New representation for lighting design, IBR

Irradiance Environment Map
Illumination
Directional Source
Complex Lighting
9
Photorealistic Rendering
10
Inverse Rendering

• How to measure realistic material models,
  lighting?
• From real photographs by inverse rendering
• Can then change viewpoint, lighting, reflectance
• Rendered images very realistic they use real
  data

Illumination Mirror Sphere Grace
Cathedral courtesy Paul Debevec
BRDF (reflectance) Images using point light
source
11
Flowchart
12
Results
Photograph
Computer rendering
New view, new lighting
Ramamoorthi and Hanrahan, SIGGRAPH 2001a
13
Inverse Rendering Goals

• Complex (possibly unknown) illumination
• Estimate both lighting and reflectance
  (factorization)

Photographs of 4 spheres in 3 different lighting
conditions courtesy Dror and Adelson
14
Factorization Ambiguities
15
Inverse Problems

• Sometimes ill-posed
• No solution or several solutions given data
• Often numerically ill-conditioned
• Answer not robust, sensitive to noise
• Need general framework to address these issues
• Mathematical theory for complex illumination

16
Lighting effects in recognition

• Space of Images (Lighting) is Infinite
  Dimensional
• Prior empirical work 5D subspace captures
  variability
• We explain empirical data, subspace methods

Peter Belhumeur Yale Face Database A
17
Outline

• Motivation
• Signal Processing Framework Reflection as
  Convolution
• Reflection Equation (2D)
• Fourier Analysis (2D)
• Spherical Harmonic Analysis (3D)
• Examples
• Efficient Rendering Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary

18
Reflection as Convolution (2D)
L
B
19
Fourier Analysis (2D)
20
Spherical Harmonics (3D)
21
Spherical Harmonic Analysis
2D
22
Insights

• Signal processing framework for reflection
• Light is the signal
• BRDF is the filter
• Reflection on a curved surface is convolution
• Inverse rendering is deconvolution
• Our contribution Formal Frequency-space analysis

23
Example Mirror BRDF

• BRDF is delta function
• Harmonic Transform is constant (infinite width)
• Reflected light field corresponds directly to
  lighting
• Mirror Sphere (Gazing Ball)

24
Phong, Microfacet Models

• Rough surfaces blur highlight
• Analytic Formula
• Approximately Gaussian

Mirror
Matte
Roughness
25
Example Lambertian BRDF
Ramamoorthi and Hanrahan, JOSA 2001
26
Second-Order Approximation

• Lambertian 9 parameters only
• order 2 approx. suffices
• Quadratic polynomial

Similar to Basri Jacobs 01
27
Dual Representation

• Practical Representation
• Diffuse localized in frequency space
• Specular localized in angular space
• Dual Angular, Frequency-Space representation

Frequency 9 param.
Frequency 9 param.
28
Outline

• Motivation
• Reflection as Convolution
• Efficient Rendering Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary

29
Video
Ramamoorthi and Hanrahan, SIGGRAPH 2001b
30
Outline

• Motivation
• Reflection as Convolution
• Efficient Rendering Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary

31
Lighting effects in recognition

• Space of Images (Lighting) is Infinite
  Dimensional
• Prior empirical work 5D subspace captures
  variability
• We explain empirical data, subspace methods

Peter Belhumeur Yale Face Database A
32
Face Basis Functions

• 5 basis functions capture 95 of image
  variability
• Linear combinations of spherical harmonics
• Complex illumination not much harder than points

Frontal Lighting
Side
Above/Below
Extreme Side
Corner
33
Inverse Lighting

• Well-posed unless r equals zero in denominator
• Cannot recover radiance from irradiance
  contradicts theorem in Preisendorfer 76
• Well-conditioned unless r small
• BRDF should contain high frequencies Sharp
  highlights
• Diffuse reflectors are ill-conditioned Low pass
  filters

34
Inverse Lambertian
Sum l2
Sum l4
True Lighting
Mirror
Teflon
35
Outline

• Motivation
• Reflection as Convolution
• Efficient Rendering Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary

36
Inverse Rendering Goals

• Formal study Well-posedness, conditioning
• General Complex (Unknown) Illumination

Quantitative Pixel error approximately 5
37
Factoring the Light Field

• The light field may be factored to estimate both
  the BRDF and the lighting
• Knowns B (4D)
• Unknowns L (2D)
• r (½ 3D) -- Make use of reciprocity

38
Algorithms Validation
Photograph
Rendering
Known Lighting
Unknown Lighting
Recovered Light
Marschner
Estimate s by ratio of intensity and total energy
39
Complex Geometry
3 photographs of a sculpture Complex unknown
illumination Geometry KNOWN Estimate BRDF and
Lighting
40
Flowchart
41
Comparison
42
New View, Lighting
Photograph
Computer rendering
43
Textured Objects
Real
Rendering
Complex, Known Lighting
44
Outline

• Motivation
• Reflection as Convolution
• Efficient Rendering Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
• Conclusions
• Pointers

45
Summary

• Reflection as Convolution
• Signal-Processing Framework
• Frequency-space analysis yields insights
• Lambertian approximated with 9 parameters
• Phong/Microfacet acts like Gaussian filter
• Inverse Rendering
• Formal Study Well-posedness, conditioning
• Dual Representations
• Practical Algorithms Complex Lighting,
  Factorization
• Efficient Forward Rendering (Environment Maps)
• Lighting Variability in Object Recognition

46
Papers

• http//graphics.stanford.edu/ravir/research.html
• Theory
• Flatland or 2D using Fourier analysis SPIE
  01
• Lambertian radiance from irradiance JOSA 01
• General 3D, Isotropic BRDFs SIGGRAPH 01a
• Applications
• Inverse Rendering SIGGRAPH
  01a
• Forward Rendering SIGGRAPH
  01b
• Lighting variability In
  preparation

47
Acknowledgements

• Marc Levoy
• Szymon Rusinkiewicz
• Steve Marschner
• John Parissenti
• Jean Gleason
• Scanned cat sculpture is Serenity by Sue Dawes
• Hodgson-Reed Stanford Graduate Fellowship
• NSF ITR grant 0085864 Interacting with the
  Visual World

48
The End
49
Related Work

• Graphics Prefiltering Environment Maps
• Qualitative observation of reflection as
  convolution
• Miller and Hoffman 84, Greene 86
• Cabral, Max, Springmeyer 87 (use spherical
  harmonics)
• Cabral et al. 99
• Vision, Perception
• DZmura 91 Reflection as frequency-space
  operator
• Basri and Jacobs 01 Lambertian reflection as
  convolution
• Recognition Appearance models e.g. Belhumeur et
  al.

50
Related Work

• Graphics Prefiltering Environment Maps
• Qualitative observation of reflection as
  convolution
• Miller and Hoffman 84, Greene 86
• Cabral, Max, Springmeyer 87 (use spherical
  harmonics)
• Cabral et al. 99
• Vision, Perception
• DZmura 91 Reflection as frequency-space
  operator
• Basri and Jacobs 01 Lambertian reflection as
  convolution
• Recognition Appearance models e.g. Belhumeur et
  al.
• Our Contributions
• Explicitly derive frequency-space convolution
  formula
• Formal Quantitative Analysis in General 3D Case

51
Example Directional Source

• Lighting is delta function
• Harmonic Transform is constant (infinite width)
• Reflected light field corresponds directly to
  BRDF
• Impulse response of BRDF filter

52
Practical Issues

• Incomplete sparse data Few views
• Use practical Dual Representation

B
Bs,fast fast specular (directional)
Bd diffuse
Bs,slow slow specular (area sources)
Frequency
Angular Space
53
Practical Issues

• Incomplete sparse data Few views
• Use practical Dual Representation
• Concavities Self Shadowing

B
Bs,fast fast specular (directional)
Bd diffuse
Bs,slow slow specular (area sources)
Integrate Lighting
Source Shadowed?
Reflected Ray Shadowed?
54
Practical Issues

• Incomplete sparse data Few views
• Use practical Dual Representation
• Concavities Self Shadowing
• Textures Spatially Varying Reflectance

Kd(x)
Ks(x)
B
Bs,fast fast specular (directional)
Bd diffuse
Bs,slow slow specular (area sources)
Integrate Lighting
Source Shadowed?
Reflected Ray Shadowed?
55
Inverse BRDF

• Well-conditioned unless L small
• Lighting should have sharp features (point
  sources, edges)
• Ill-conditioned for soft lighting

Area source Same BRDF
Directional Source
56
Comparison
Photograph
Rendering
Pixel Error Approximately 5
Known Lighting
Marschner
Our method