CharlesFelix Chabert Per Einarsson Andrew Jones WanChun Ma1

1
Relighting Human Locomotion with Flowed
Reflectance Fields
Charles-Felix Chabert Per Einarsson Andrew
Jones Wan-Chun Ma1 Bruce Lamond Tim Hawkins
Sebastian Sylwan Paul Debevec
USC Centers for Creative Technologies National
Taiwan University 1
2
Introduction

• Live action performance mixed with computer
  generated environment
• Live Actors
• inherently photoreal
• lack of postproduction control
• Virtual Actors
• Labor intensive
• Hard to achieve photoreal results, hair, skin,
  cloth...
• full postproduction control

3
Our Approach
4
Related Work

• View Interpolation and Light Field
  RenderingChen 93 Leveau Faugeras 94
  McMillan Bishop 95 Seitz Dyer 96 Levoy
  Hanrahan 96 Gortler 96
• Dynamic Light Field AcquisitionWilburn 2005
  Yang 2002 Yu 2002 Zhang Chen 2004
• Image-Based RelightingDorsey 95 Nimeroff 94
  Debevec 00 Wenger 05
• Free-Viewpoint VideoMoezzi 96 Rander 97
  Matsusik 00 Carranza 03 Vedula
  05 Zitnik 04 Theobalt 05

5
Related Work
Wenger et al. SIGGRAPH 2005
6
Related Work
1/30th of a second
7
Related Work
8
Light Stage 6
9
Light Stage 6
2/3 of a 8m geodesic dome 931 dome lights 140
floor lights
2 meter wide working volume 3 high-speed cameras
10
Capture

• Performance capture using 3 high speed cameras
• 36 locomotion cycles in 360 degrees rotation
• Metronome and tactile grooves keep motion
  repeatable

Top
Middle
Bottom
11
Capture

• Performance capture using 3 high speed cameras
• 36 locomotion cycles in 360 degrees rotation
• Metronome and tactile grooves keep motion
  repeatable

Top
Middle
Bottom
12
Capture
13
Reflectance Field

• 26 directional lighting condition
• 40 lights per basis element

• 3 track frames
• 3 matte frames

1/30th sec
14
Relighting

• Image-Based Relighting

Grace Cathedral
26-element projection

Sum of weighted basis frames
Relit frame
Debevec et. al 2000
15
7D Data Set

• 2D images
• 2D lighting
• 2D viewpoint
• 1D time
• Walking Data Set
• Horizontal image resolution 320
• Vertical image resolution 448
• Lighting Basis per walk cycle 32
• Lighting conditions per basis 33
• Horizontal viewpoint 36
• Vertical viewpoint 3
• Total frames acquired 114,048

1/30th sec
8GB of image data per camera
16
Optical Flow Fields
Image A
Image B
17
Data Registration

• Temporally align each reflectance field using
  optical flow BlackAnandan 91Wenger 05

Aligned
Reflectance field
18
Data Registration

• Temporally align each reflectance field using
  optical flow BlackAnandan 91Wenger 05

Aligned
Reflectance field
19
Data Registration
32 walk cycle frames
36x3 viewpoints
20
View Interpolation
turntable
360º
C0
Camera Array
C1
10º
C2
time
Ci
...
21
View interpolation

C1
C0
22
Flowed Reflectance Fields

For each locomotion frame 3 x 36 Reflectance
field data sets 108 Bidirectional horizontal
flow fields 72 Bidirectional vertical flow fields
23
Flowed Light Field Interpolation

24
Flowed Light Field Interpolation
25
Flowed Light Field Interpolation
top view

current camera
26
Flowed Light Field Interpolation
top view
p

current camera
27
Flowed Light Field Interpolation
top view
p

current camera
28
Flowed Light Field Interpolation

29
Flowed Light Field Interpolation

Traditional Light Field Interpolation
30
Flowed Light Field Interpolation

Flowed Light Field Interpolation
31
Flowed Light Field Interpolation

32
Matting

Matte frame
Track frame
33
Matting

Matte frame
Track frame
34
Relightable Shadow Maps

• For each locomotion step
• Voxel carved visual hull
• Model a 3x3m ground plane
• Trace ray toward each lighting basis element to
  determine the occlusion

Visual Hull from matte images
Relightable shadow map
35
Results
Captured Perfomance Data
Image Based Lighting Environment
36
Results
37
Results
38
Results
39
Conclusion

• 7D data set capture of a full human body
  performance
• Postproduction control over viewpoint and
  illumination
• Flowed Light Field Rendering algorithm
• Image-based approach produces photorealistic
  results

40
More Information

• Relighting Human Locomotion with Flowed
  Reflectance FieldsPer Einarsson, Charles-Felix
  Chabert, Andrew Jones, Wan-Chun Ma, Bruce Lamond,
  Marc Bolas, Sebastian Sylwan, Tim Hawkins, Paul
  Debevec EGSR 2006
• Relighting Human Locomotion with Flowed
  Reflectance FieldsSIGGRAPH 2006 Animation Theater

41
More Information

• Relighting Human Locomotion with Flowed
  Reflectance FieldsPer Einarsson, Charles-Felix
  Chabert, Andrew Jones, Wan-Chun Ma, Bruce Lamond,
  Marc Bolas, Sebastian Sylwan, Tim Hawkins, Paul
  DebevecEGSR 2006
• Relighting Human Locomotion with Flowed
  Reflectance FieldsSIGGRAPH 2006 Animation
  Theater
• NEW Light Stage 6 Data Sets Online!http//gl.ict.
  usc.edu/Data/LightStage/

42
Thank you.

• Relighting Human Locomotion with Flowed
  Reflectance FieldsPer Einarsson, Charles-Felix
  Chabert, Andrew Jones, Wan-Chun Ma, Bruce Lamond,
  Marc Bolas, Sebastian Sylwan, Tim Hawkins, Paul
  DebevecEGSR 2006
• Relighting Human Locomotion with Flowed
  Reflectance FieldsSIGGRAPH 2006 Animation
  Theater
• NEW Light Stage 6 Data Sets Online!http//gl.ict.
  usc.edu/Data/LightStage/