Augmented Reality

1
Augmented Reality

• Camera(s)

• Real scene, scene coordinates

yC

• Real table

xC
R, t
C

• Augmented plant

zC
R, t
Z
yV
xV
Y
zV
X

• Visualization (screen, HMD)

2
Example 1 ARToolkit
3
Example 2 Structure Motion Schweighofer
4
Pinhole Camera
y
x
principal point (u0,v0)
p(u,v)
z
optical axis
C
f
v
P(x,y,z)
u
P(x,y,z)
image plane pi (u,v) z -f
focal length f

• real camera

• Pinhole C center of projection

• 2D projection ? 3D scene

• p(u,v) ? line of sight viewing direction

5
Projective Geometry
P(x,y,z)
P
p(u,v)
!!!
y
x
y
x
z
P
(u0,v0)
C
v
u
p0 ... z 0
pi ... z f
projective camera, normalized camera f 1 1
stationary camera ? 1 coordinate system
(x,y,z) camera-centered coordinate system scene
coordinate system Only points in p0 are not
projected to pi
6
Projective Images Examples, Properties

• Impression of depth in images
• Parallel lines meet at infinity
• infinity is projected to finite
• location in the image
• horizon
• points at infinity,

Triggs and Mohr
7
Projective Images Scaling
8
Projective Images Foreshortening
9
Projective Images Parallel Lines Meet
Sonka, Hlavac, Boyle
10
Projective Geometry vs. Computer Vision

• points in p1
• straight lines of sight
• projective reconstruction
• geometry, precise
• known correspondences

• discrete pixels in pi
• sampling theorem
• lens distortion, aperture,
• depth of field
• oriented projective rec.
• in front of camera
• inherently imprecise ?
• estimation, minimization
• outliers ? robustness

11
Example Stereo Reconstruction

P
P
C1

• projective geometry
• computer vision

C2
12
Algebraic Projective Geometry (1)

• A unified geometric algebraic framework
• Point
• Line

13
Algebraic Projective Geometry (2)

• Duality point ? line
• Unified approach projective n-space Pn
• point (n1)
  - vector

14
Homogeneous Coordinates in Pn
Equivalence class of vectors
forms P2 projective plane
Homogeneous coordinates , but only 2
DoF inhomogeneous
15
Equivalence Class of Vectors
Without further knowledge, such situations cannot
be distingushed !
A further example Equivalence of a toy car,
closeup shot, and real car, distant shot
16
The Projective Plane (1)

• Point
• Line
• ideal points treated like any point x3?0
• ?
  Fluchtpunkte
• line at infinity ? the planes
  horizon

intersection of parallel lines !
17
The Projective Plane (2)

• Adding the ideal points to R2 leads to the
  projective plane P2
• Covers all homo-geneous coordinates

HartleyZisserman
18
The Projective Plane (3)
Image of the horizon of p, line at infinity
of p
vanishing point, Fluchtpunkt Bild eines
Fernpunktes
p

• Projective geometry can map infinitely far
  points / lines to finite ones
• No difference between finite and infinite
• e.g. hyperbola is one continuous conic

19
There is also Projective Space P3

• Points
• Planes

• Lines 4 DoF
• Dual line L

duality L ? L
duality point ? plane
20
Projective Transformations in Pn

• projective transformation collineation
  projectivity homography H
• Invertible mapping Pn ?Pn
• ? geradentreue Abbildung
• (n1) x (n1) matrix
• In P2
• H has (n1)2-1 DoF, H is non-singular

21
Projective Transformations in P2

• Translation
• Rotation
• Scaling
• Any combination, e.g.

22
A Remark on Conics

• 2nd degree equation in the plane
• Homog. coord
• Conic C
• Five DoF, 5 points define a conic

23
Back to Homographies Examples (1)
Mapping between planes
HartleyZisserman
central projection may be expressed by xHx
24
Back to Homographies Examples (2)
Removing projective distortion
HartleyZisserman
25
Back to Homographies Examples (3)
HartleyZisserman
26
Transformation for Points, Lines, Conics

• Point
• Line
• Conic

27
A Hierarchy of Transformations / Geometries (1)

• Isometric / Euclidean
• Invariants length, angle, area
• Similarity
• Invariants ratios of length / areas, angle,
  parallel lines

28
A Hierarchy of Transformations / Geometries (2)

• Affine
• 6 DoF 2 x scale ?1,?2 2 x rot. ?,? 2 x
  translation
• Invariants parallel lines, ratios of parallel
  lengths, ratios of areas

29
A Hierarchy of Transformations / Geometries (3)

• Projective
• 8 DoF 2 x scale ?1,?2 2 x rot. ?,? 2 x
  translation
• 2 x line at infinity
• Invariant Cross-ratio CR of 4 collinear points

A
B
C
D
30
A Hierarchy of Transformations / Geometries (4)
Projective 8dof
Affine 6dof
In 2D, a square transforms to
Similarity 4dof
Euclidean 3dof
31
A Hierarchy of Transformations / Geometries (5)
Projective 15dof
Affine 12dof
In 3D, a cube transforms to
Similarity 7dof
Euclidean 6dof
32
Stratification

• In AR, we take perspective images,
• but we require metric (Euclidean) reconstruction!

• How?
• The stratification of 3D geometry Pollefeys 2.2

33
Stratification of 2D / 3D Geometry

• Many possibilities, many approaches
• Examples
• Known directions
• Known points, lines, planes at 8
• Known lengths in the scene
• IAC (self-calibration)
• Known camera intrinsics
• Camera calibration relative orientation
• Multiview geometry, structuremotion

unknown scenes
34
Stratification Examples (1)

• Known points, line at infinity

l8
v1
v2
l1
l3
l2
l4
perspective
affine
35
Stratification Examples (1)

• Known directions

affine
metric (similarity, unknown scale)
36
Stratification Examples (2)

• Known plane at infinity

perspective
affine
37
Stratification Examples (2)

• Known directions

affine
metric (similarity, unknown scale)
38
Stratification Examples (3)

• Known lengths

Pollefeys IJCV99
metric (similarity, unknown scale)
metric (Euclidean, known scale)
39
Stratification Examples (4)

• ARToolkit

metric (Euclidean, known scale)
perspective
40
Video AR is (rather) simple

• Known artificial targets / markers
• Uncalibrated perspective camera
• But collineation required
• Problems when e.g. strong lens distortions
• Augmentation of the video frames
• Examples
• Artoolkit
• Kutulakos
• Can be related to scene coordinates, but requires
  ground truth for markers

41
ARToolkit Demo ISAR 2000
immersive view
observers view
42
Kutulakos Calibration-Free AR IEEE Trans.
Visualization and Graphics 1998
43
Field Maintenance Support ARVIKA
44
Scene Structure Camera Motion(the harder, but
more general approach to AR)

• Many possible approaches
• Monocular, calibrated, known natural landmarks
  Ribo
• Stereo, calibrated Schweighofer
• Monocular, calibrated Murray
• Monocular, uncalibrated Pollefeys
• not (yet?) in real time !

unknown scene, unknown natural landmarks
? calibration !