CS 395/495-25: Spring 2003

1
CS 395/495-25 Spring 2003

• IBMR Week 7B
• Chapter 6.1, 6.2 Chapter 7 More Single-Camera
  Details
• Jack Tumblin
• jet_at_cs.northwestern.edu

2
IBMR-Related Seminars

• 3D Scanning for Cultural Heritage Applications
  Holly Rushmeier, IBM TJ Watson
• Friday May 16 300pm, Rm 381, CS Dept.
• Light Scattering Models for Rendering Human Hair
• Steve Marschner, Cornell University Friday May
  23 300pm, Rm 381, CS Dept.

3
Reminders

• ProjA graded Good Job! 90,95, 110
• ProjB graded Good! minor H confusions...
• MidTerm graded
• ProjC posted, due Friday, May 16
• ProjD tomorrow, Friday May 16, due Friday May 30
• Start Watsons Late Policy? Grade -(3n)
  points n of class meetings late
• Take-Home Final Exam Thurs June 5, due June 11

4
Mirror Spheres Why?

• Traditional CG Rendering
• To make an image,
• Compute radiance arriving at novel camera
  position
• Specify Incoming light Irradiance function
• at each (x,y,z) point from every direction
  (?,?)
• Specify Shape, Texture, Reflectance, BRDF,
  BRSSDF
• at each surface point (xs,ys,zs)
• Compute Outgoing light exitance function
• (after incoming light bounces around the
  scene) at any camera point (x,y,z) from any
  pixel direction (?,?)

5
Mirror Spheres Why?

• IBMR
• input is far less defined! images,
  (usually) no depth only
• To make an image,
• Compute radiance arriving at novel camera
  position
• Specify Radiance from images, and perhaps
• Specify Radiance from images
• Compute Outgoing light exitance function
• (after incoming light bounces around the
  scene) at any camera point (x,y,z) from any
  pixel direction (?,?)

6
Rendering from a camera image?

• Conventional external camera reads light field
• (after rendering)

Camera
Shape, Position, Movement,
Emitted Light
zc
xc
yc
Reflected, Scattered, Light
BRDF, Texture, Scattering
7
Rendering from a camera image?

• IBMR Let camera measure light inside scene

Shape, Position, Movement,
Emitted Light
Camera
x2
x1
Reflected, Scattered, Light
BRDF, Texture, Scattering
x3
8
Rendering from a camera image?

• IBMR Camera measures light inside scene

TROUBLE!Camera is an object reflects light,
changes scene.
Shape, Position, Movement,
Emitted Light
Camera
x2
x1
Reflected, Scattered, Light
BRDF, Texture, Scattering
x3
WANTED tiny, point-like panoramic cameraa
light probe
9
One Answer Light Probe

• Photograph a small mirror sphere

Shape, Position, Movement,
Emitted Light
Camera
xc
yc
Mirror Sphere
zc
Reflected, Scattered, Light
BRDF, Texture, Scattering
10
Light Probes How?

• Tele-photo a mirror sphere (narrow FOV)
• warp image to find irradiance .vs. direction
• High contrast?
• Higher resol.?
• More positions?
• More Pictures!

Paul Debevec, SIGGRAPH2001 course Image Based
Lighting
11
High Contrasts too!

• .

Paul Debevec, SIGGRAPH2001 short course Image
Based Lighting
12
One Answer Light Probe

• Light probes can measure irradiance
• the incoming light at a point.
• Can use them as panoramic cameras,
• --OR
• as local light maps
• they define intensity of incoming light .vs.
  direction, as if local neighborhood was lit by
  lights at infinity.
• Caution! may not be valid at nearby locations!
• Caution! high dynamic range! (gtgt 1255)
• Can use them to render synthetic objects...

13
A Mirror Sphere is...
A light probe to measure ALL incoming light
at a point.How can we use it?
14
Image-Based Actual Re-lighting
15
Image-Based Actual Re-lighting
Light the actress in Los Angeles
Debevec et al., SIGG2001
Film the background in Milan, Measure incoming
light,
Matched LA and Milan lighting.
Matte the background
16
Measure REAL light in a REAL scene...
Debevec et al., SIGG1998
17
Render FAKE objects with REAL light,
And combine with REAL image
Debevec et al., SIGG1998
18
The Grand Challenges

• Controlled Lights Controlled Camerassuggest we
  CAN recover arbitrary BRDF/ BSSSDFand enough
  shape.
• Is any method PRACTICAL?
• Can we avoid/reduce corrupting interreflections?
• Can we understand the shape/texture tradeoff?

19
The Grand Rewards

• Controlled Lights Controlled Camerassuggest we
  CAN recover arbitrary BRDF/ BSSSDFand enough
  shape.
• Holodeck? CAVE uncorrupted by interreflections
• Historical Preservation? complete optical records
• Fake Materials? a BRDF/BSSRDF display ...
• Shader Lamps? exchange reflectance for
  illumination
• IBMR invisibility?

20
Image-Based Synthetic Re-lighting
Masselus et al., 2002
21
Image-Based Shape Refinement
Fine geometric Details ?? Fine Texture/Normal
Details
Rushmeier, 2001
22
Image-Based Shape Approximation
Matusik 2002
Matusik 2002
23
Image-Based Shape Approximation
Matusik 2002
24
Why all this Projective Tedium?

• So you have the tools to try IBMR
• (and because Im struggling, slowly, to it boil
  down to the essentials in this course)
• Its almost over last 3 weeks of class will be
  reading good recent research papers, and will
• Begin exploring some open research questions...

25
Camera Matrix P Summary

• Basic camera
• x P0 X where P0 K 0
• World-space camera
• translate world origin to camera location C, then
  rotate x PX (P0RT) X
• Rewrite as
• P K R -RC
• Redundant notationP M p4M RKp4 -K
  R C

26
Chapter 6 In Just One Slide

• Given point correspondence sets (xi?? Xi), How
  do you find camera matrix P ? (full 11 DOF)
• Surprise! You already know how !
• DLT method -rewrite H x x as Hx ? x
  0 -rewrite P X x as PX ? x 0
• -vectorize, stack, solve Ah 0 for h
  vector -vectorize, stack, solve Ap 0 for p
  vector
• -Normalizing step removes origin dependence
• More data ? better results (at least 28 point
  pairs)(why so many? rule-of-thumb constraints
  5x DOF 55 27.5 point pairs)
• Algebraic Geometric Error, Sampson Error

27
Chapter 7 More One-Camera Details

• Full 3x4 camera matrix P maps P3world to P2 image
• ? What does it do to basic 3D world shapes?
• Planes
• Given any point X? on a plane in P3,
• Change worlds coord. system let plane be z0
• Matrix P reduces to 3x3 matrix H in P2
• x? PX?
• THUS P2 can do any, all P2 plane transforms

x?y?0t?
x?y?t?
h11 h12 h13 h21 h22 h21 h31 h32 h33
p11 p12 p13 p14 p21 p12 p23 p24 p31 p32
p33 p34

28
Chapter 7 More One-Camera Details

• Full 3x4 camera matrix P maps P3world to P2 image
• ? What does it do to basic 3D world shapes?
• Points, Directions
• World-space P3 direction D ? image space point
  xd
• Recall direction D (x,y,z,0) (a point at
  infinity)
• sets a R3 finite point d (x,y,z).
• xd PD M p4 D M d
• p4 column has no effect, because of Ds
  zero Recall M KR
• xd M d M-1xd d

D
xd
yc

d
p
C
f
zc
X (world space)
xc
29
Chapter 7 More One-Camera Details

• Full 3x4 camera matrix P maps P3world to P2 image
• ? What does it do to basic 3D world shapes?
• Lines Forward Projection
• Line / Ray in world ? Line/Ray in image
• Ray in P3 is X(?) A ?B
• Camera changes to P2 x(?) PA ?PB

yc
A
PA
?B
p
C
f
zc
xc
30
Chapter 7 More One-Camera Details

• Full 3x4 camera matrix P maps P3world to P2 image
• ? What does it do to basic 3D world shapes?
• Lines Back Projection
• Line L in image ? Plane ?L in world
• Recall Line L in P2 (a 3-vector) L x1 x2
  x3T
• Plane ?L in P3 (a 4-vector)
• ?L PTL
• (SKIP Plucker Matrix lines)

?L
p11 p21 p31 p12 p22 p32p13 p23 p33p14
p24 p34
?1?2?3
yc
L
zc
p
C
f
..
xc
31
Chapter 7 More One-Camera Details

• Full 3x4 camera matrix P maps P3world to P2 image
• ? What does it do to basic 3D world shapes?
• Conics 1
• Conic C in image ? Cone Quadric Qco in world
• Qco PTCP
• (Tip of cone is camera center V)

C
yc
p
V
f
zc
xc
32
Chapter 7 More One-Camera Details

• Full 3x4 camera matrix P maps P3world to P2 image
• ? What does it do to basic 3D world shapes?
• Conics 2
• Dual (line) Quadric Q in world ? Dual (line)
  Conic C silhouette in image
• C PTQP
• Works for ANY world quadric!sphere, cylinder,
  ellipsoid,paraboloid, hyperboloid, line, disk

C
Q
yc
p
V
zc
f
xc
33
Chapter 7 More One-Camera Details

• Full 3x4 camera matrix P maps P3world to P2 image
• ? What does it do to basic 3D world shapes?
• Conics 3
• World-space quadric Q ? World-space view cone
  Qco, a degenerate quadric
• Qco (VT QV)Q (QV)(QV)T

Qco
yc
Q
p
V
zc
f
xc
34
Chapter 7 More One-Camera Details

• Full 3x4 camera matrix P maps P3world to P2 image
• ? What if the image plane moves?
• A) Translation
• Given internal camera calibration K
• In (xc, yc)? changes px,py. In zc? focal length
  f let k (ftz)/f, then
• K
• Define effect of K on image points x,x x K
  0X x K 0X
• x K K-1 x

yc
p
c
f
zc
xc
35
Chapter 7 More One-Camera Details

• Full 3x4 camera matrix P maps P3world to P2 image
• ? What if the image plane moves?
• B) Rotation
• Given internal camera calibration K
• Rotate basic cameras output about its center
  C using 3D rotation matrix R (3x3) x K
  0X x KR 0 X x KR(K-1K) 0 X
  (KRK-1) K0 X
• Get new points x from old image pts x (KRK-1)
  x x
• aka conjugate rotation use this to construct
  planar panoramas

yc
R
p
c
f
zc
xc
36
Chapter 7 More One-Camera Details

• Full 3x4 camera matrix P maps P3world to P2 image
• THUS if the image plane moves
• just rearranges the image points
• a) Translations
• b) Rotations

(KRK-1) x x
(K K-1) x x
37
Movement Detection?

• Can we do it from images only?
• 2D projective transforms often LOOK like 3-D
• External cam. calib. affects all elements of P
• YES. Camera moved if--only-ifCamera-ray points
  (C?x?X1,X2,) will
• map to LINE (not a point) in the other image
• Epipolar Line l image of L
• Parallax x1?x2 vector

X2
X1
x2
x1
L
x
C
l
C
38
Cameras as Protractors

• Define world-space direction d
• From a P3 infinity point D xd yd zd 0T
  define d xd yd zd
• Use Basic Camera P0,
• (e.g. C(0,0,0,1), R0, P P0)
• (Danger! now mixing P2, P3)
• Link direction D to image-space pt.
  xd(xc,yc,zc) P0 Xd KIXd K d xd
• Ray thru image pt. x has direction d K-1xd

39
Cameras as Protractors

• Angle between C and 2 image points x1,x2(see
  book pg 199)
• cos ? x1T (K-TK-1) x2 (x1T (K-TK-1)
  x1)(x2T (K-TK-1) x2)
• Image line L defines a plane ?L
• (Careful! P3 world P2 camera axes here!)
• Plane normal direction n KT L

d2
n
x2
?
d1
x1
C
L
40
Cameras as Protractors

• Angle between C and 2 image points x1,x2(see
  book pg 199)
• cos ? x1T (K-TK-1) x2 (x1T (K-TK-1)
  x1)(x2T (K-TK-1) x2)
• Image line L defines a plane ?L
• (Careful! P3 world P2 camera axes here!)
• Plane normal direction n KT L

d2
n
Something Special here? Yes!
x2
?
d1
x1
C
L
41
Cameras as Protractors

• What is (K-TK-1) ?
• Recall P3 Conic Weirdness
• Plane at infinity ?? holds all horizon points d
  (universe wrapper)
• Absolute Conic ?? is imaginary outermost circle
  of ??
• for ANY camera,Translation wont change Horizon
  point images
• P Xd x KRd (pg200)
• Absolute conic is inside ?? its all horizon
  points
• for ANY camera, P ?? (K-TK-1) Image of
  Absolute Conic

42
Why do we care?

• P ?? (K-TK-1) Image of Absolute Conic
• IAC is a magic tool for camera calibration K
• Recall ?? let us find H from perp. lines.
• Much better than vanishing pt. methods
• With IAC, find P matrix from an image of just 3
  (non-coplanar) squares

43
Cameras as Protractors

• Image Direction d xc, yc, zc, 0T
• Image Direction from a point x d K-1x
• Angle ? between C and 2 image points
  x1,x2 (pg 199)
• Simplify with absolute conic ??
• P ?? (K-TK-1) ? Image of Absolute Conic

d2
x2
?
x1
d1
C
L
44
Cameras as Protractors

• P ?? (K-TK-1). OK. Now what was ?? again?
• Recall P3 Conic Weirdness (pg. 63-67)
• Plane at infinity ?? holds all horizon points d
  (universe wrapper)
• Absolute Conic ?? imaginary points in outermost
  circle of ??
• Satisfies BOTH x12 x22 x32 0 AND x42 0
• Can rewrite equations to look like a quadric (but
  isnt no x4)
• AHA! points on it are (complex conjugate)
  directions d !
• Finds right angles-- if d1 ? d2, then d1T
  ??d2 0

1 0 0 00 1 0 00 0 1 00 0 0 0
dT??d
45
Cameras as Protractors

• P ?? (K-TK-1). OK. Now what was ?? again?
• Dual of Absolute Conic ?? is Dual Quadric Q?
  (?!?!)
• More compact notation for imaginary planes ?
• Same matrix, but different use
• --find a plane ? for every possible direction d
  --? is ? to ??, and tangent to the quadric Q?
• ?? is circle in ?? where tangent planes ? are ?
  to ??
• Finds right angles-- if ?1 ? ?2, then ?1T
  Q??2 0

Inconsistent notation!
1 0 0 00 1 0 00 0 1 00 0 0 0
?TQ??
46
Cameras as Protractors

• P ?? (K-TK-1) ? Image of Absolute Conic
• Just ??as has a dual Q?, ? has dual ?
• ? ?-1 K KT
• The dual conic ? is the image of Q? , so
• ? P Q? P

(first 3 columns of P?)
47
Cameras as Protractors

• P ?? (K-TK-1) ? Image of Absolute Conic
• Just ??as has a dual Q?, ? has dual ?
• ? ?-1 K KT
• The dual conic ? is the image of Q? , so
• ? P Q? P
• Vanishing points v1,v2 of 2 ? world-space
  lines v1T? v2 0
• Vanishing lines L1, L2 of 2 ? world-space
  planes L1T? L2 0

(first 3 columns of P)
48
Cameras as Protractors

• Clever vanishing point trick
• Perpendicular lines in image?
• Find their vanishing pts. by construction
• Use v1T? v2 0, stack, solve for ? (K-TK-1)

v3
v2
v1
49
END