Presentaci - PowerPoint PPT Presentation

1 / 27
About This Presentation
Title:

Presentaci

Description:

COMS 6160 Real-Time High Quality Rendering Nov 3rd, 2004 Relighting Framework Sebastian Enrique Columbia University senrique_at_cs.columbia.edu COMS 6160 Relighting ... – PowerPoint PPT presentation

Number of Views:23
Avg rating:3.0/5.0
Slides: 28
Provided by: Sebastia105
Category:

less

Transcript and Presenter's Notes

Title: Presentaci


1
COMS 6160 Real-Time High Quality Rendering Nov
3rd, 2004
Relighting Framework
Sebastian Enrique Columbia University senrique_at_cs.
columbia.edu
2
COMS 6160
Relighting Framework
Nov 3rd, 2004
About Relighting
What is Relighting? Given a set of different
illuminated images from a scene, relighting is
the process of producing new images of the same
scene with new lighting conditions, composing in
some way the original data.
Why it is useful? Photorealistic real-time
rendering of complex scenes with complex
illumination is an open problem. One IBR approach
is to capture or render a set of original images
from a scene, and then relight it in real time to
produce the same scene but with novel
illumination!
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
2
3
COMS 6160
Nov 3rd, 2004
Relighting Framework
About Relighting (cont.)
What is one of the most challenging parts of
it? High quantity of images is needed to produce
good results, the challenge is to find/use a good
compression technique in order to relight fast
and using as less memory as possible.
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
3
4
COMS 6160
Nov 3rd, 2004
Relighting Framework
Some Previous Work
  • Doorsey et al. used a progressive radiosity
    method where they pre-rendered synthetic scenes
    to simulate lighting conditions superimposing
    single light source images- in opera stages in
    1991.
  • Many (Hallinan 94, Epstein 95, etc.) have
    pre-acquired real images changing the lighting
    direction.
  • Debevec compressed the pre-acquired images in
    JPG and processed in the compressed domain in
    2000.
  • Sloan uses low-frequency spherical harmonics on
    geometry in 2002.
  • In 2003 Sloan uses clustered or VQPCA on
    spherical harmonic coefficients.
  • Ng compressed data using wavelets in 2003.
  • Sloan (Local Deformable PRT) and Ramamoorthi
    (Triple Product Wavelets) are the most recent
    related techniques, 2004.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
4
5
COMS 6160
Nov 3rd, 2004
Relighting Framework
LSD Relighting Approach
  • The Lighting Sensitive Display (LSD) was
    developed by Shree Nayar, Peter Belhumeur, and
    Terry Boult.
  • Basically, it is a monitor with an attached
    camera that captures the lighting conditions of
    the environment. The monitor shows an scene,
    which changes (is relighted) as the illumination
    in the environment changes.
  • They adopted an image-based approach, using a
    large set of images.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
5
6
COMS 6160
Nov 3rd, 2004
Relighting Framework
LSD Relighting Approach (cont.)
  • They have developed a novel algorithm (using two
    stages of Principal Component Analysis or PCA)
    that compresses that large set of images and
    allows the relighting in real-time with complex
    lighting conditions.
  • They have reached compression ratios of 4761
    for colored images.
  • The algorithm simultaneously exploits
    correlations over the lighting domain as well as
    coherences over the spatial domain of the image.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
6
7
COMS 6160
Nov 3rd, 2004
Relighting Framework
LSD Relighting Approach (cont.)
  • We based our Relighting Framework on the LSD
    approach.
  • In the following slides we will explain
  • How the input images for the LSD algorithm
    should be taken.
  • First compression stage of the algorithm.
  • Second compression stage of the algorithm.
  • Real-Time rendering.
  • Then, we will get on
  • Extending the LSD approach with the use of
    cubemaps.
  • Problems found.
  • Future directions.
  • Discussion and Videos

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
7
8
COMS 6160
Nov 3rd, 2004
Relighting Framework
Setting Up Images
  • To take the images, a grid of light source
    positions is generated in the front face of an
    imaginary cube (a plane parallel to the image
    plane).
  • For each position on the grid, place the distant
    light with maximum intensity and generate (render
    or capture) an image.

Scene to Render / Capture
Distant Light Source For Single Image
Fixed Viewpoint
Grid of Positions for Distant Light Sources, its
in the front face of an imaginary cube.
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
8
9
COMS 6160
Nov 3rd, 2004
Relighting Framework
First Stage SVD
  • The first SVD (Singular Value Decomposition)
    exploits the fact that locally (within small
    regions of the images) the variation due to
    changes in illumination can be approximated by a
    small number of bases.
  • The image is divided into m square blocks, each
    containing p pixels, and bases are computed for
    each.
  • Each image Ii is an image of the scene
    illuminated by a single distant point light
    source.

Image divided in m square blocks
Each block has p pixels
Image Ii
IMAGES
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
9
10
COMS 6160
Nov 3rd, 2004
Relighting Framework
First Stage SVD (cont.)
  • Iij denote the jth block of the ith image.
  • For each block in the scene a low-dimensional
    approximation is computed as follows
  • Ij is a p x n matrix representing a collection
    of image blocks. The I column of Ij is formed by
    p pixels of the jth block.
  • Using SVD, a rank b approximation to Ij is found
    as
  • Ej is a p x b column-orthogonal matrix, called
    block bases.
  • Sj is a b x b diagonal matrix.
  • Cj is an n x b column-orthogonal matrix.
  • The singular values from Sj are absorbed into
    CjT, getting a b x n matrix, Lj, called block
    lighting coefficients.

Each Ij represents the pixels for the same block
on all images
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
10
11
COMS 6160
Nov 3rd, 2004
Relighting Framework
First Stage SVD (cont.)
  • SVD

Ij
Sj
Ej

x
x
p x n
p x n
p x p
CjT
diagonal matrix (singular values)
n x n
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
11
12
COMS 6160
Nov 3rd, 2004
Relighting Framework
First Stage SVD (cont.)
  • Then, stacking all of the m Lj block matrices,
    we get the lighting coefficient matrix L.
  • The image bases m Ej are also stacked in the
    matrix E.
  • The collection of submatrices within E and L
    contain all the information needed to approximate
    the collection of images corresponding to the n
    lighting directions.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
12
13
COMS 6160
Nov 3rd, 2004
Relighting Framework
First Stage SVD (cont.)
Image 640x480 n64x644096 m40x301200
p16x16256 b10
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
13
14
COMS 6160
Nov 3rd, 2004
Relighting Framework
Second Stage SVD
  • The fact that there is much coherence in image
    blocks is exploited using a second SVD.
  • It is applied to the light coefficient matrix L
  • U is an (m x b) x q column-orthogonal matrix,
    called lighting coefficient bases.
  • V is a q x n matrix, called compressed
    coefficient matrix.
  • Rank q denotes the number of linear bases kept
    to approximate L.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
14
15
COMS 6160
Nov 3rd, 2004
Relighting Framework
Second Stage SVD (cont.)
Image 640x480 n64x644096 m40x301200
p16x16256 b10 q200
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
15
16
COMS 6160
Nov 3rd, 2004
Relighting Framework
Examples
Bad Rank Election
(showing only red color component)
Relighted Image SVD First Stage Rank 3
Original Image
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
16
17
COMS 6160
Nov 3rd, 2004
Relighting Framework
Examples (cont.)
Higher Ranks
Only small differences in brightness are
noticeable, but there are no visible
block discontinuities.
Relighted Image SVD First Stage Rank 10 SVD
Second Stage Rank 20
Original Image (in fact, two combined original
images)
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
17
18
COMS 6160
Nov 3rd, 2004
Relighting Framework
Real-Time Rendering
  • Real-Time illumination is represented by the
    illumination field vector s.
  • Each element si corresponds to a point light,
    and its value represents the intensity that that
    point light is contributing with to the scene.
  • To render the relighted scene in real-time,
    preprocessed matrices E, U, and V should be used
  1. Compute a compressed coefficient vector as the
    product (q elements)
  2. Compute a lighting coefficient vector I as
    (mb elements)
  3. To render the j block, a subvector Ij from I must
    be multiplied with the corresponding stacked Ej
    matrix. This must be done each frame for the m
    blocks.

(b elements) (p
elements)
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
18
19
COMS 6160
Nov 3rd, 2004
Relighting Framework
Real-Time Rendering (cont.)
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
19
20
COMS 6160
Nov 3rd, 2004
Relighting Framework
Demo Point Lights / David
Image 480x640 n6x16x161536 m32x321024
p16x16256 b10 q20
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
20
21
COMS 6160
Nov 3rd, 2004
Relighting Framework
Extending LSD Cube Lighting
  • In place of using only the front face of the
    cube to capture images with different lighting
    condition, use all of the 6 faces.
  • The parameters that change are the quantity of
    input images and the length of the s light field
    vector (both now 6 n).
  • In this way, the scene can be relighted with
    illumination coming for all around.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
21
22
COMS 6160
Nov 3rd, 2004
Relighting Framework
High-Dynamic Range Cubic Environment Map
  • Having the relighting scheme for the full cube,
    we can add cubic environment maps to relight the
    scene.
  • To have more accuracy, High-Dynamic Range (HDR)
    cubic environment maps are used (floating point
    values in place of 0..255).
  • The CubicMap could be rotated, and each element
    in the s light field vector is contributed by the
    value of the corresponding texel, weighted by the
    solid angle.
  • HDR(i) is the corresponding texel value N is
    the corresponding cube face normal R is the
    vector from the origin to the center of the
    element i in the cube grid m is the face cube
    grid resolution (squared is how many images per
    face exist) each cube face has a size of 2 x 2.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
22
23
COMS 6160
Nov 3rd, 2004
Relighting Framework
Demo CubeMap Point Lights / Nicole
Image 512x512 n6x16x161536 m32x321024
p16x16256 b10 q20
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
23
24
COMS 6160
Nov 3rd, 2004
Relighting Framework
Framework Summary
Generate Scene-Matrices For Images Set Of Each
Scene to Relight
Relight in Real-Time the Given Scene-Matrices
with HDR CubeMap and Point Lights
Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
24
25
COMS 6160
Nov 3rd, 2004
Relighting Framework
Problems Found
  • Preprocess takes TOO MUCH TIME.
  • Preprocess uses TOO MUCH MEMORY.
  • Few things could be done in current graphics
    hardware (HDR cubemap processing specially).

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
25
26
COMS 6160
Nov 3rd, 2004
Relighting Framework
Future Directions
  • Optimize preprocessing stages (distributed
    computations?)
  • Use error metrics to automatically select
    adequate ranks.
  • Extend user interface to allow the relighting
    using lower ranks than that given in the input
    matrices.
  • Allow viewpoint changes.

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
26
27
COMS 6160
Nov 3rd, 2004
Relighting Framework
End of Talk
  • Ready to hear
  • Comments
  • Suggestions
  • Discussions
  • Questions
  • More VIDEOS to show while chatting

Sebastian Enrique - Columbia University -
senrique_at_cs.columbia.edu
27
Write a Comment
User Comments (0)
About PowerShow.com