Eyes for Relighting

1
Eyes for Relighting

• Extracting environment maps for use in
  integrating and relighting scenes
• (Noshino and Nayar)

2
Previous Work

• Global illumination image based lighting
• Mirror Ball environment maps
• Eye Shape studies

3
Idea

• Find Eye in image by hand
• Find best fit ellipse
• Construct ellipsoid in 3 space from ellipse,
  focal length, distance to camera and previous
  knowledge of eyes
• Perform inverse ray tracing from camera to
  ellipsoid
• Use law of reflection to project ray onto unit
  sphere and mark color.

4
Idea
Environment Map
Cornea
Camera and Picture
5
Implementation

• Mark points around eye, find best fit ellipse.
• Paper discusses using gradients to refine initial
  guess for ellipse parameters, though in most
  cases the results are satisfactory without.

6
Implementation

• Assume limited perspective, so that the ratio
  between major and minor axis represents the angle
  of the plane it lies in with respect to the
  camera.
• Assumes basically level camera.

7
Implementation

• Use a (t,theta) parameterization for the
  ellipsoid, where t is the height, and theta the
  radial angle.
• Send rays to focal point on camera and sample
  image (focal point in pixels)
• Find reflection along normal

8
Implementation Issues

• (t,theta) is not uniformly distributed, need far
  more rays for small t (where the ellipsoid is
  shallow) than large t
• No guarantee that whole of sphere is covered,
  there may be holes.

9
Implementation Issues

• To solve these ideally compute from sphere to
  camera and sample image.
• However this is not trivial since rays are
  dependent upon camera position and one would need
  to try several camera to eye rays to reach a
  certain point on the sphere.
• The solution used just keeps adding rays until
  there are no gaps in the sphere image

10
Implementation

• Once sphere map is calculated, use in global
  illumination as light source
• Assumes the map approximates higher light level
  HDR since the reflection reduced small light to
  negligible levels. Still need to be careful to
  pick low levels for illumination
• Additionally no attempt is made to remove iris
  texture. Need to pick dark eyes.

11
Implementation

• Rendered as a sphere map in Cinema 4d 7.0 with
  new objects added.

12
Blade Runner

• Suppose we want to add a car to the background of
  this image

13
Blade Runner

• First we find the best fit ellipse

14
Blade Runner

• Next we extract the correct environment map.
• (here rendered on a sphere)

15
Blade Runner

• Finally we render the object using the computed
  environment map
• (note that it is untextured)

16
Blade Runner

• Finally we composite (fiddle by hand!)

17
Blade Runner
18
Sections not implemented

• The map can also be used to unlight a model.
• Wave a light using the light distribution found
  on the eyes, and assumption that skin is
  lambertian (dot product of normal and light
  direction) find a normal map.

19
Sections not implemented

• Since the lighting is known and normals can be
  found, computing the base color is also possible.
• All computations take place over set of images
  with different lighting so best fit matches of
  both normals and base color can be found.

20
Sections not implemented

• Integration of the normals can be done as in
  previous work to find a 3d model which fits the
  lighting description.
• This can also be texture mapped with the base
  texture, and lit as a lambertian surface with
  shadows.

21
Sections not implemented

• The surface can be inserted into a new scene and
  lit according to the global illumination of that
  scene.
• This illumination can of course come from eyes!

22
Sections not implemented