Analysis of Lighting Effects

1
Analysis of Lighting Effects

• Outline
• The problem
• Lighting models
• Shape from shading
• Photometric stereo
• Harmonic analysis of lighting

2
(No Transcript)
3
(No Transcript)
4
(No Transcript)
5
Applications

• Modeling the effect of lighting can be used for
• Recognition particularly face recognition
• Shape reconstruction
• Motion estimation
• Re-rendering

6
Lighting is Complex

• Lighting can come from any direction and at any
  strength
• Infinite degree of freedom

7
Issues in Lighting

• Single light source (point, extended) vs.
  multiple light sources
• Far light vs. near light
• Matt surfaces vs. specular surfaces
• Cast shadows
• Inter-reflections

8
Lighting

• From a source travels in straight lines
• Energy decreases with r2 (r distance from
  source)
• When light rays reach an object
• Part of the energy is absorbed
• Part is reflected (possibly different amounts
  in different directions)
• Part may continue traveling into the object,
  if object is transparent / translucent

9
Specular Reflectance

• When a surface is smooth light reflects in the
  opposite direction of the surface normal

10
Specular Reflectance

• When a surface is slightly rough the reflected
  light will fall off around the specular direction

11
Lambertian Reflectance

• When the surface is very rough light may be
  reflected equally in all directions

12
Lambertian Reflectance

• When the surface is very rough light may be
  reflected equally in all directions

13
Lambertian Reflectance
14
Lambert Law
q
or
15
BRDF

• A general description of how
  opaque objects reflect light is
  given by the
  Bidirectional
  Reflectance Distribution Function (BRDF)
• BRDF specifies for a unit of incoming light in a
  direction (?i,Fi) how much light will be
  reflected in a direction (?e,Fe) . BRDF is a
  function of 4 variables f(?i,Fi?e,Fe).
• (0,0) denotes the direction of the surface
  normal.
• Most surfaces are isotropic, i.e., reflectance in
  any direction depends on the relative direction
  with respect to the incoming direction (leaving 3
  parameters)

16
Why BRDF is Needed?
Light from front Light from back
17
Most Existing Algorithms

• Assume a single, distant point source
• All normals visible to the source (?lt90)
• Plus, maybe, ambient light (constant lighting
  from all directions)

18
Shape from Shading

• Input a single image
• Output 3D shape
• Problem is ill-posed, many different shapes can
  give rise to same image
• Common assumptions
• Lighting is known
• Reflectance properties are completely known For
  Lambertian surfaces albedo is known (usually
  uniform)

19
convex
concave
20
convex
concave
21
HVS Assumes Light from Above
22
HVS Assumes Light from Above
23
Lambertian Shape from Shading (SFS)

• Image irradiance equation
• Image intensity depends on surface orientation
• It also depends on lighting and albedo, but those
  assumed to be known

24
Surface Normal

• A surface z(x,y)
• A point on the surface (x,y,z(x,y))T
• Tangent directions tx(1,0,p)T, ty(0,1,q)T
  with pzx, qzy

25
Lambertian SFS

• We obtain
• Proportionality because albedo is known up to
  scale
• For each point one differential equation in two
  unknowns, p and q
• But both come from an integrable surface z(x,y)
• Thus, py qx (zxyzyx).
• Therefore, one differential equation in one
  unknowns

26
Lambertian SFS
27
SFS with Fast Marching

• Suppose lighting coincides with viewing direction
  l(0,0,1)T, then
• Therefore
• For general l we can rotate the camera

28
Distance Transform

• is called Eikonal equation
• Consider d(x) s.t. dx1
• Assume x00

d
x
x0
29
Distance Transform

• is called Eikonal equation
• Consider d(x) s.t. dx1
• Assume x00 and x01

d
x
x1
x0
30
SFS with Fast Marching

• - Some places are more
  difficult to walk than others
• Solution to Eikonal equations using a variation
  of Dijkstras algorithm
• Initial condition we need to know z at extrema
• Starting from lowest points, we propagate a wave
  front, where we gradually compute new values of z
  from old ones

31
Results
32
Photometric Stereo

• Fewer assumptions are needed if we have several
  images of the same object under different
  lightings
• In this case we can solve for both lighting,
  albedo, and shape
• This can be done by Factorization
• Recall that
• Ignore the case ?gt90

33
Photometric Stereo - Factorization
Goal given M, find L and S
What should rank(M) be?
34
Photometric Stereo - Factorization

• Use SVD to find a rank 3 approximation
• Define
• So
• Factorization is not unique, since
• , A
  invertible
• To reduce ambiguity we impose integrability

35
Reducing Ambiguity

• Assume
• We want to enforce integrability
• Notice that
• Denote by the three rows of A,
  then
• From which we obtain

36
Reducing Ambiguity

• Linear transformations of a surface
• It can be shown that this is the only
  transformation that maintains integrability
• Such transformations are called generalized bas
  relief transformations (GBR)
• Thus, by imposing integrability the surface is
  reconstructed up to GBR

37
Relief Sculptures
38
Illumination Cone

• Due to additivity, the set of images of an object
  under different lighting forms a convex cone in
  RN
• This characterization is generic, holds also with
  specularities, shadows and inter-reflections
• Unfortunately, representing the cone is
  complicated (infinite degree of freedom)

0.5
0.2
0.3
39
Eigenfaces
Photobook/Eigenfaces (MIT Media Lab)
40
Recognition with PCA

• Amano, Hiura, Yamaguti, and Inokuchi Atick and
  Redlich Bakry, Abo-Elsoud, and Kamel
  Belhumeur, Hespanha, and Kriegman Bhatnagar,
  Shaw, and Williams Black and Jepson Brennan
  and Principe Campbell and Flynn Casasent, Sipe
  and Talukder Chan, Nasrabadi and Torrieri
  Chung, Kee and Kim Cootes, Taylor, Cooper and
  Graham Covell Cui and Weng Daily and
  Cottrell Demir, Akarun, and Alpaydin Duta,
  Jain and Dubuisson-Jolly Hallinan Han and
  Tewfik Jebara and Pentland Kagesawa, Ueno,
  Kasushi, and Kashiwagi King and Xu Kalocsai,
  Zhao, and Elagin Lee, Jung, Kwon and Hong Liu
  and Wechsler Menser and Muller Moghaddam
  Moon and Philips Murase and Nayar Nishino,
  Sato, and Ikeuchi Novak, and Owirka Nishino,
  Sato, and Ikeuchi Ohta, Kohtaro and Ikeuchi
  Ong and Gong Penev and Atick Penev and
  Sirivitch Lorente and Torres Pentland,
  Moghaddam, and Starner Ramanathan, Sum, and
  Soon Reiter and Matas Romdhani, Gong and
  Psarrou Shan, Gao, Chen, and Ma Shen, Fu, Xu,
  Hsu, Chang, and Meng Sirivitch and Kirby
  Song, Chang, and Shaowei Torres, Reutter, and
  Lorente Turk and Pentland Watta, Gandhi, and
  Lakshmanan Weng and Chen Yuela, Dai, and
  Feng Yuille, Snow, Epstein, and Belhumeur
  Zhao, Chellappa, and Krishnaswamy Zhao and
  Yang

41
Empirical Study
Ball Face Phone Parrot
1 48.2 53.7 67.9 42.8
2 84.4 75.2 83.2 69.7
3 94.4 90.2 88.2 76.3
4 96.5 92.1 92.0 81.5
5 97.9 93.5 94.1 84.7
6 98.9 94.5 95.2 87.2
7 99.1 95.3 96.3 88.5
8 99.3 95.8 96.8 89.7
9 99.5 96.3 97.2 90.7
10 99.6 96.6 97.5 91.7
(Yuille et al.)
42
Intuition
lighting
reflectance
43
(Light ? Reflectance) Convolution
44
(Light ? Reflectance) Convolution
45
Spherical Harmonics
1
Z
Y
X
XZ
YZ
XY
46
Harmonic Transform of Kernel
n
47
Cumulative Energy
(percents)
N
48
Second Order Approximation
49
Reflectance Near 9D
Yields 9D linear subspace. 4D approximation
(first order) can also be used point source
ambient
50
Harmonic Representations
? Albedo n Surface normal
Positive values Negative values
r
51
Photometric Stereo
S
L
M
r
rnz
rnz
rny
r(3nz2-1)
r(nx2-ny2)
rnxny
rnxnz
rnynz

SVD recovers L and S up to an ambiguity
52
Photometric Stereo
53
Photometric Stereo
54
Summary

• Lighting effects are complex
• Algorithms for SFS and photometric stereo for
  Lambertian object illuminated by a single light
  source
• Harmonic analysis extends this to multiple light
  sources
• Handling specularities, shadows, and
  inter-reflections is difficult

55
Mutual Information
Camera Rotation