The Mathematics of ImageBased Modeling

1
The Mathematics of Image-Based Modeling

• Eleanor Rieffel
• Senior Research Scientist, FXPAL
• 14 October 2009

2
Will we model the world,and why?

• Yes. Because we can.
• Digital cameras everywhere
• Cheap to take pictures, even video
• Computational power improvements
• Rendering of 3D is fast, even for detailed models
• High bandwidth communications
• Photos, videos, and models easily shared

3
Weve already started

• Building Rome in a Day
• University of Washington (S. Agarwal. Y.
  Furukawa, et al.)
• St. Peters Basilica
• 1,294 images, 530,076 points
• http//grail.cs.washington.edu/rome/
• Photosynth spatial image browser
• 296 photos
• http//photosynth.net/view.aspx?cid17c617f7-dfc8-
  45f9-b2d3-e24f64c3b0aa

4
FXPALs DOTS Surveillance System
A. Girgensohn, et al., DOTS Support for
Effective Video Surveillance , ACM Multimedia 2007
5
FXPAL Collaboration with Tcho
M. Back, et al., High-tech Chocolate exploring
mobile and 3D applications for factories,
Siggraph 2009
6
Other application areas

• Training
• Emergency response, military exercises
• Immersive telepresence
• Facilities
• Testing disaster plans, e.g. for hospital
• Virtual tourism and virtual tours
• Psychiatric treatment
• Post-traumatic stress, phobias

7
The Mathematics behind Image- Based 3D model
reconstruction
8
Locating correspondence points
9
Might look like correspondence points, but
10
Might look like correspondence points, but
11
Obtaining correspondence points

• Handmarked by user
• Slow
• Error prone
• Automatic
• Sophisticated approaches
• Local feature matching
• Difficulties with surfaces all same color, esp.
  black
• Place markers with distinctive patterns
• Known size

12
FXPALs Pantheia system overview

• Goal make it quick and easy for a person to
  create a rough model of a space or object by
  placing semantic markers and taking images
• What the system does
• Estimates all marker and camera positions
• Deduces model from these estimates and the
  semantics associated with the markers
• Renders the object, and provides various ways of
  viewing and interacting with the models

13
What we can mark data types

• Vertices
• Edges
• Point-plane data
• point ,and the plane it is in
• Point normal data
• Point outward facing normal vector
• Goal reconstruct spaces and objects from small
  amounts of marked data

14
Definitions
http//bligblug.blogspot.com/
15
Motivation for the definitions

• We want to be able to describe the sets of
  objects that we can reconstruct
• We are only going to consider polygons and
  polyhedra
• It is easier to reconstruct shapes with nice
  properties such as
• Convex
• Orthogonal
• Simply Connected

16
Polygons

• There isnt a standard definition!
• There are many!
• Definition A polygon is a closed, connected,
  two-dimensional region of a plane whose boundary
  consists of a finite set of segments e1, e2, ,
  en called edges with endpoints v1, v2, , vn
  called vertices (singular vertex). Furthermore,
  every vertex vi is the endpoint of two edges, ei
  and ei1, where the indexing is modulo n, and
  nonconsecutive edges do not intersect.

17
Whenever you read a definition

• think up a rich set of examples that fit the
  definition, and a rich set of examples that do
  not.

18
Examples of Polygons and not

• Polygons

• Not polygons

• Draw
• Simplest examples you can think of
• Borderline cases that might confuse someone
• A complicated polygon

• Draw
• Simplest examples you can think of
• Borderline cases that might confuse someone
• Shapes that meet parts of the definition but not
  others

19
Polygons as I have defined them

• What are the simplest polygons you can think of?
• What complicated polygons can you think of?
• Can polygons have holes?
• What are some planar shapes that are not
  polygons?
• Can you think of a shape where there might be
  debate as to whether it is one polygon or two?
• What non-polygonal planar shapes might someone
  think are polygons?

20
Alternative definition of a polygon

• How would you define polygon if you wanted the
  definition to polygons with holes?
• Definition A polygon is a closed, connected,
  two-dimensional planar region whose boundary
  consists of a finite set of segments e1, e2, ,
  en called edges with endpoints v1, v2, , vn
  called vertices. Every vertex is the endpoint of
  exactly two edges, and two edges can only
  intersect in their endpoint vertices.

21
Polyhedra

• There isnt a standard definition!
• There are many!
• Definition A polyhedron (pl. polyhedra, adj.
  polyhedral) is a closed, connected,
  three-dimensional volume of three-dimensional
  Euclidean space whose boundary consists of a
  finite set of polygons called faces such that
  every edge of every polygon is shared with
  exactly one other polygon and two faces may
  intersect only at edges or vertices shared by the
  two polygons.

22
Polyhedra as I have defined them

• What are the simplest polyhedra you can think of?
• What complicated polyhedra can you think of?
• Describe or draw some shapes that are not
  polyhedra?
• What non-polyhedral shapes might someone think
  are polyhedra?

23
Shapes that are not polygons or polyhedra
No
Yes
24
Convex and nonconvex shapes

• Convex the line segment connecting every pair of
  points is contained in the shape

• Nonconvex there exists a pair of points such
  that the line segment between them goes outside
  the shape

Is this room convex?
http//bligblug.blogspot.com/
25
Orthogonal

• Orthogonal polygon
• All edges are aligned with one of the coordinate
  axes
• Orthogonal polyhedron
• All faces are aligned with one of the three
  coordinate axis planes, the xy- , yz- , r
  xz-plane.

26
Connected and simply connected

• Connected
• Every pair of points is connected by some path
  that stays within the shape
• Are there shapes that are convex but not
  connected?
• Are there shapes that are connected but not
  convex?

• Simply connected polygon or polyhedron
• has no holes
• Any loop within the shape can be shrunk to a
  point
• Not simply connected

27
Vertex DATA

• Reconstruction from

28
ORourkes Theorem

• Any orthogonal polygon can be unambiguously
  reconstructed from its set of vertices as long as
  it doesnt have any degenerate (180) vertices
• Properties of such polygons
• From each vertex exactly one vertical edge
  extends, and
• Exactly one horizontal edge extends

29
When you read ORourkes Theorem, ask yourself

• How would I reconstruct a polygon from the vertex
  data?
• Can I reconstruct in some simple cases?
• Some more complex cases?
• Do I see a general algorithm?
• Why is orthogonal required?
• Can I think of a counterexample?
• Why does he rule out 180 vertices?
• Can I think of a counterexample?
• What about the 3D case?

30
ORourkes Algorithm
31
ORourkes algorithm

• How would you go about reconstructing a polygon
  from this set of points?
• His algorithm is very simple (once you have seen
  it)
• While it is simple, it is not an ancient result
• ORourke published his paper in 1988
• There are all sorts of wonderful things waiting
  for you to discover them!

32
ORourkes Algorithm
33
ORourkes Algorithm
34
ORourkes Algorithm
35
ORourkes Algorithm
36
When you read ORourkes Theorem, ask yourself

• How would I reconstruct a polygon from the vertex
  data?
• Can I reconstruct in some simple cases?
• Some more complex cases?
• Do I see a general algorithm?
• Why is orthogonal required?
• Can I think of a counterexample?
• Why does he rule out 180 vertices?
• Can I think of a counterexample?
• What about the 3D case?

37
Why is orthogonal required?
38
Why is orthogonal required?
39
Why is orthogonal required?
40
Why are 180 vertices ruled out?
41
Why are 180 vertices ruled out?
42
Why are 180 vertices ruled out?
43
Why are 180 vertices ruled out?
44
Some comments on the 3D case

• ORourkes Theorem and algorithm can be extended
  to three dimensions
• No 180 degree vertices rules out many shapes

45
Point-normal data

• Reconstruction from

46
Marker-per-face markup strategy

• Marker-per-face markup strategy
• User must place at least one marker per face
• Or otherwise obtain position and normal for at
  least one point in each face
• Works for any convex polyhedron
• How would you figure out the polygon from the
  marker information?
• Ambiguities exist for nonconvex polyhedra
• Can you think of an example?
• Ambiguities exist for multiple convex polyhedra
• Can you think of an example?

47
Single Convex Polygon

• For every edge of the polygon, the entire polygon
  is contained with in one of the half planes
  defined by the edge
• To reconstruct the polygon, take the intersection
  of all of these half planes
• 3D case?

48
Ambiguity nonconvex case
49
Ambiguity nonconvex case
50
Ambiguity nonconvex case
51
Ambiguity nonconvex case (3D)
Can extend to 3D by extruding orthogonally
52
Ambiguity same topology
53
Ambiguity same topology
54
Ambiguity same topology
55
Ambiguity for multiple convex polygons
56
Ambiguity for multiple convex polygons
57
Elaboration metadata
58
Elaboration

• Use marker semantics to indicate which faces are
  the main faces of a convex shape
• Elaborations are indicated by marker semantics
  that say which face is being elaborated
• Orthogonal extrusions (and intrusions)
• Can obtain unambiguous reconstruction for a wide
  variety of shapes and scenes

59
Kumo model
60
Lab Room
Laboratory Room
Laboratory Model
61
Pantheia model of an IKEA Cabinet
Actual Bookcase
Bookcase VRML model
62
Thank you!
http//bligblug.blogspot.com/
63
Any Questions?
For more information http//blog.fxpal.com/ http
//www.fxpal.com/?prieffel rieffel_at_fxpal.com
Slide background a recoloring of a quasiperiodic
tiling John Baez created using Greg Egans
DeBruijn applet http//math.ucr.edu/home/baez/tili
ngs Cartoons courtesy of http//bligblug.blogspot
.com/