Range data

1
Range data

• Marc Pollefeys
• COMP 256

Some slides and illustrations from J. Ponce,
2
Tentative class schedule
Jan 16/18 - Introduction
Jan 23/25 Cameras Radiometry
Jan 30/Feb1 Sources Shadows Color
Feb 6/8 Linear filters edges Texture
Feb 13/15 Multi-View Geometry Stereo
Feb 20/22 Optical flow Project proposals
Feb27/Mar1 Affine SfM Projective SfM
Mar 6/8 Camera Calibration Segmentation
Mar 13/15 Springbreak Springbreak
Mar 20/22 Fitting Prob. Segmentation
Mar 27/29 Silhouettes and Photoconsistency Linear tracking
Apr 3/5 Project Update Non-linear Tracking
Apr 10/12 Object Recognition Object Recognition
Apr 17/19 Range data ?
Apr 24/26 Final project Final project
3
Final project

• Presentation
• 10 minute demo or presentation
• make arrangements with Talha for planning Monday
  or Wednesday (demos priority on Wed.)
• Papers
• due before my 35th birthday!

4
RANGE DATA

• Active Range Sensors
• Segmentation
• Elements of Analytical Differential Geometry
• Registration and Model Acquisition
• Quaternions
• Object Recognition

Reading Chapter 21.
5
Active Range Sensors

• Triangulation-based sensors
• Time-of-flight sensors
• New Technologies

Courtesy of D. Huber and M. Hebert.
6
Structured light

• Single grid projection
• Binary code
• A desktop scanner

7
Principle
deformationconnectivity of pattern
3D Shape
Proesmans and Van Gool, ICPR96
8
Acquisition setup
Proesmans and Van Gool, ICPR96
9
Calibration
Proesmans and Van Gool, ICPR96
10
Image with projected grid
Proesmans and Van Gool, ICPR96
11
Line detectors
Proesmans and Van Gool, ICPR96
12
Linking
Proesmans and Van Gool, ICPR96
13
Initial grid
Proesmans and Van Gool, ICPR96
14
Corrected grid
sub-pixel refinement of grid
Proesmans and Van Gool, ICPR96
15
Depth computation
Proesmans and Van Gool, ICPR96
16
Removing lines
Proesmans and Van Gool, ICPR96
17
Texture estimation
Proesmans and Van Gool, ICPR96
18
3D Reconstruction
Proesmans and Van Gool, ICPR96
19
Dionysos
Proesmans and Van Gool, ICPR96
20
Theatre mask
Proesmans and Van Gool, ICPR96
21
Capital
Proesmans and Van Gool, ICPR96
22
Coded planes

• Structured light
• Use projector as a camera
• Figure out correspondences by coding light
  pattern
• Only need to code 1D
• (but not parallel with epipolar lines!)

B. Curless
23
A desktop scanner
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
(No Transcript)
33
More range sensors
DeltaSphere
Z-cam
34
Real-time system
Koninckx and Van Gool
35
Elements of Analytical Differential Geometry

• Parametric surface x U? R2 ? E3

• Normal and Gaussian curvatures

36
Example Monge Patches
x ( u, v ) (u, v, h( u, v ))
37
Example Local Surface Parameterization

• u,v axes principal directions
• h axis surface normal

• In this case
• h(0,0)hu(0,0)hv(0,0)0
• N(0,0,1)T
• huv(0,0)0, ?1 huu(0,0), ?2 hvv(0,0)

38
Finding Step and Roof Edges in Range Images
39
Step Model
And, since z?0 in x?
40
Roof Model
And ?? has a maximum value inversely
proportional to ? in a point x? located at a
distance proportional to ? from the origin.
41
Computing the Principal Directions and Curvatures

• Adaptive smoothing

• Finite-difference masks

Reprinted from Describing Surfaces, by J.M.
Brady, J. Ponce, A. Yuille and H. Asada, Proc.
International Symposium on Robotics Research, H.
Hanafusa and H. Inoue (eds.), MIT Press (1985). ?
1985 MIT.
42
Scale-Space Matching
Reprinted from Toward a Surface Primal
Sketch, By J. Ponce and J.M. Brady, in
Three-Dimensional Machine Vision, T. Kanade
(ed.), Kluwer Academic Publishers (1987). ? 1987
Kluwer Academic Publishers.
43
Segmentation into Planes via Region
Growing (Faugeras Hebert, 1986)
Idea Iteratively merge the pair of planar
regions minimizing the average distance to the
plane best fitting them.
Reprinted from The Representation, Recognition
and Locating of 3D Objects, by O.D. Faugeras and
M. Hebert, the International Journal of Robotics
Research, 5(3)27-52 (1986). ? 1986 Sage
Publications. Reprinted by permission of Sage
Publications.
44
Quaternions
q is a quaternion, a 2 R is its real part, and ?
2 R3 is its imaginary part.
q a ?
Operations on quaternions

• Sum of quaternions ( a? ) ( b? ) ( ab
  ) (?? )

• Multiplication by a scalar ? ( a? ) (
  ?a?? )

• Quaternion product
• ( a? ) ( b? ) ( a b ? ? ) ( a
  ? b ? ? ? )

Note qq q q
45
Quaternions and Rotations

• Let R denote the rotation of angle ? about the
  unit vector u.
• Define q cos ?/2 sin ?/2 u.
• Then for any vector ?, R ? q ? q.

Reciprocally, if q a ( b, c, d )T is a
unit quaternion, the corresponding rotation
matrix is
46
The Iterative Closest Point Registration
Algorithm (Besl and McKay, 1992)

• Key points
• finding the closest-point pairs (k-d trees,
  caching)
• estimating the rigid transformation
  (quaternions).

47
Using Quaternions to Estimate a Rigid
Transformation
Problem Find the rotation matrix R and the
vector t that minimize
48
ICP Registration Results
Reprinted from A Method for Registration of 3D
Shapes, by P.J. Besl and N.D. McKay, IEEE Trans.
on Pattern Analysis and Machine Intelligence,
14(2)238-256 (1992). ? 1992 IEEE.
49
Initial alignment?

• Mostly open problem
• A possible approach using bitangents (Vanden
  Wyngaerd and Van Gool)

50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
Fusing Range Images (Curless Levoy, 1996)
Idea Construct watertight surfaces as level sets
of appropriate volumetric density functions.
Reprinted from A Volumetric Method for Building
Complex Models from Range Images, by B. Curless
and M. Levoy, Proc. SIGGRAPH (1996). ? 1996 ACM,
Inc. Included here by permission. Courtesy of M.
Levoy.
54
Fusing Range Images (Curless Levoy, 1996)
Idea Construct watertight surfaces as level sets
of appropriate volumetric density functions.
Reprinted from A Volumetric Method for Building
Complex Models from Range Images, by B. Curless
and M. Levoy, Proc. SIGGRAPH (1996). ? 1996 ACM,
Inc. Included here by permission. Courtesy of M.
Levoy.
55
Volumetric integration
(Curless and Levoy, Siggraph96)
range surfaces
signed distance to surface
volume
weight (accuracy)
distance
depth
sensor
surface1

• use voxel space
• new surface as zero-crossing
• (find using marching cubes)
• least-squares estimate
• (zero derivativeminimum)

surface2
combined estimate
56
Fusing Range Images (Curless Levoy, 1996)
Idea Construct watertight surfaces as level sets
of appropriate volumetric density functions.
Reprinted from A Volumetric Method for Building
Complex Models from Range Images, by B. Curless
and M. Levoy, Proc. SIGGRAPH (1996). ? 1996 ACM,
Inc. Included here by permission. Courtesy of M.
Levoy.
57
From volume to meshMarching Cubes

• First 2D, Marching Squares

Marching Cubes A High Resolution 3D Surface
Construction Algorithm,William E. Lorensen and
Harvey E. Cline,Computer Graphics (Proceedings
of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.
58
From volume to meshMarching Cubes
Marching Cubes A High Resolution 3D Surface
Construction Algorithm,William E. Lorensen and
Harvey E. Cline,Computer Graphics (Proceedings
of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.
59
From volume to meshMarching Cubes

• Improvement

                           
Marching Cubes A High Resolution 3D Surface
Construction Algorithm,William E. Lorensen and
Harvey E. Cline,Computer Graphics (Proceedings
of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.
60
The Faugeras-Hebert Plane Matching Algorithm
(1986)

• Key points
• finding initial matches (area comparisons,
  binning)
• estimating the rigid transformation
  (quaternions).

61
Finding all the vectors v making an angle between
?-? And ?? with a vector u.
62
Using Quaternions to Estimate a Rigid
Transformation
? n x d 0 ! ? n x d 0
where n R n and d n t d.
63
Recognition Results (Faugeras Hebert, 1986)
Reprinted from The Representation, Recognition
and Locating of 3D Objects, by O.D. Faugeras and
M. Hebert, the International Journal of Robotics
Research, 5(3)27-52 (1986). ? 1986 Sage
Publications. Reprinted by permission of Sage
Publications.
64
Spin Images (Johnson Hebert, 1998)
SP(Q)(PQ n, PQ n)
? ?
65
Sample Spin Images
Reprinted from Using Spin Images for Efficient
Object Recognition from Cluttered 3D Scenes, by
A.E. Johnson and M. Hebert, IEEE Trans. on
Pattern Analysis and Machine Intelligence,
21(5)433-449 (1999). ? 1999 IEEE.
Matching Criterion
66
Recognition Results
Reprinted from Using Spin Images for Efficient
Object Recognition from Cluttered 3D Scenes, by
A.E. Johnson and M. Hebert, IEEE Trans. on
Pattern Analysis and Machine Intelligence,
21(5)433-449 (1999). ? 1999 IEEE.
67
Computer Vision

• What next?
• Related courses
• Comp 254 Image Analysis
• Comp 255 Recent Advances in Image Analysis (Odd
  Falls)
• Comp 290

68
The future is bright

• Computation is cheap
• Lots of pix
• cameras are cheap, many pix are digital
• Lots of demand for slicing and dicing pix
• generate models
• new movies from old
• search
• Lots of hidden value
• cant do data mining for collections with pix in
  them
• e.g. mortgage papers, cheques, etc.
• e.g. filtering

69
There are lots of cameras!
surveillance cameras 1500/sq.mile in Manhattan
70
Recent flowering of vision

• can do (sort of!)
• structure from motion
• segmentation
• video representation
• model building
• tracking
• face finding

• will be able to do (sort of!)
• face recognition
• inference about people
• character recognition
• perhaps more

71
Big open problems

• Next step in structure from motion
• Really good missing variable formalism
• Decent understanding of illumination, materials
  and shading
• Segmentation

• Representation for recognition
• Efficient management of relations
• Recognition processes for lots of objects
• A lot of this looks like applied statistics

72
Next week Final project presentations