Real-time Acquisition and Rendering of Large 3D Models

1
Real-time Acquisition and Rendering of Large 3D
Models

• Szymon Rusinkiewicz

2
Computer Graphics Pipeline
Shape
Motion
Lighting and Reflectance

• Human time expensive
• Sensors cheap
• Computer graphics increasingly relies
  onmeasurements of the real world

3
3D Scanning Applications

• Computer graphics
• Product inspection
• Robot navigation
• As-built floorplans

• Product design
• Archaeology
• Clothes fitting
• Art history

4
The Digital Michelangelo Project

• Push state of the art in range scanning and
  demonstrate applications in art and art history

Working in the museum
Scanning geometry
Scanning color
5
Traditional Range Scanning Pipeline

• High-quality, robust pipeline for producing 3D
  models
• Scan object with laser triangulation scanner
  many views from different angles
• Align pieces into single coordinate
  frameinitial manual alignment, refined with ICP
• Merge overlapping regions compute average
  surface using VRIP Curless Levoy 96
• Display resulting model

6
3D Scan of David Statistics

• Over 5 meters tall
• 1/4 mm resolution
• 22 people
• 30 nights of scanning
• Efficiency max min 8 1
• Needed view planning
• Weight of gantry 800 kg
• Putting model together1000 man-hours and
  counting

7
New 3D Scanning Pipeline

• Need for a fast, inexpensive,easy-to-use 3D
  scanning system

• Wave a (small, rigid) object by hand in front of
  the scanner
• Automatically align data asit is acquired
• Let user see partial model as itis being built
  fill holes

8
Real-Time 3D Model Acquisition

• Prototype real-time model acquisition system
• 3D scanning of moving objects
• Fast alignment
• Real-time merging and display

9
Applications of Easy-to-Use3D Model Acquisition

• Advertising
• More capabilities in Photoshop
• Movie sets
• Augmented reality
• User interfaces

10
3D Scanning Technologies

• Contact-based touch probes
• Passive shape from stereo, motion, shading
• Active time-of-flight, defocus, photometric
  stereo, triangulation
• Triangulation systems are inexpensive, robust,
  and flexible
• Take advantage of trends in DLP projectors

11
Laser Triangulation
Object

• Project laser stripe onto object

12
Laser Triangulation
Object
(x,y)

• Depth from ray-plane triangulation

13
Triangulation

• Faster acquisition project multiple stripes
• Correspondence problem which stripeis which?

14
Triangulation
Slow, robust
Fast, fragile
15
Time-Coded Light Patterns

• Assign each stripe a unique illumination
  codeover time Posdamer 82

Time
Space
16
Gray-Code Patterns

• To minimize effects of quantization erroreach
  point may be a boundary only once

Time
Space
17
Structured-Light Assumptions

• Structured-light systems make certain assumptions
  about the scene
• Spatial continuity assumption
• Assume scene is one object
• Project a grid, pattern of dots, etc.
• Temporal continuity assumption
• Assume scene is static
• Assign stripes a code over time

18
Codes for Moving Scenes

• We make a different assumption
• Object may move
• Velocity low enough to permit tracking
• Spatio-temporal continuity

19
Codes for Moving Scenes

• Code stripe boundariesinstead of stripes
• Perform frame-to-frametracking of
  correspondingboundaries
• Propagate illumination history
• Hall-Holt Rusinkiewicz, ICCV 2001

20
New Scanning Pipeline
Project Code
Capture Images
Find Boundaries
Match Boundaries
Decode
Compute Range
21
Designing a Code
Project Code
Capture Images
Find Boundaries
Match Boundaries
Decode
Compute Range

• Biggest problem is ghosts WW or BB boundaries
  that cant be seen directly

22
Designing a Code

• Design a code to make tracking possible
• Do not allow two spatially adjacent ghosts
• Do not allow two temporally adjacent ghosts

t
23
Designing a Code

• Graph (for 4 frames)

24
Designing a Code

• Graph (for 4 frames)

• Edges boundaries (over time)

• Path with alternating colors55 edges in graph
  ?maximal-length traversal has 110 boundaries
  (111 stripes)

25
Image Capture
Project Code
Capture Images
Find Boundaries
Match Boundaries
Decode
Compute Range

• Standard video camera fields at 60 Hz
• Genlock camera to projector

26
Finding Boundaries
Project Code
Capture Images
Find Boundaries
Match Boundaries
Decode
Compute Range

• Standard edge detection problem
• Current solution find minima and maxima of
  intensity, boundary is between them

27
Matching Stripe Boundaries
Project Code
Capture Images
Find Boundaries
Match Boundaries
Decode
Compute Range

• Even if number of ghosts isminimized, matching
  is not easy

?
28
Matching Stripe Boundaries

• Resolve ambiguity by constraining maximum stripe
  velocity
• Could accommodate higher speeds by estimating
  velocities
• Could take advantage of methods intracking
  literature (e.g., Kalman filters)

29
Decoding Boundaries
Project Code
Capture Images
Find Boundaries
Match Boundaries
Decode
Compute Range

• Propagate illumination history
• Table lookup based on illumination history and
  position in four-frame sequence
• Once a stripe has been tracked for at least four
  frames,it contributes useful data on every
  subsequent frame

30
Computing 3D Position
Project Code
Capture Images
Find Boundaries
Match Boundaries
Decode
Compute Range

• Ray-plane intersection
• Requires calibration of
• Camera, projector intrinsics
• Relative position and orientation

31
Results
Video frames
Stripe boundaries
unknown known ghosts
32
Results

• Single range image of moving object

Top View
Top View
Front View
Front View
Boundary codes and tracking
Gray codes, no tracking
33
Aligning 3D Data

• This range scanner can be used for any moving
  objects
• For rigid objects, range images can be aligned to
  each other as object moves

34
Aligning 3D Data

• If correct correspondences are known,it is
  possible to find correct relative
  rotation/translation

35
Aligning 3D Data

• How to find corresponding points?
• Previous systems based on user input,feature
  matching, surface signatures, etc.

36
Aligning 3D Data

• Alternative assume closest points correspond to
  each other, compute the best transform

37
Aligning 3D Data

• and iterate to find alignment
• Iterated Closest Points (ICP) Besl McKay 92
• Converges if starting position close enough

38
ICP Variants

• Classic ICP algorithm not real-time
• To improve speed examine stages of ICP and
  evaluate proposed variants
• Rusinkiewicz Levoy, 3DIM 2001

• Selecting source points (from one or both meshes)
• Matching to points in the other mesh
• Weighting the correspondences
• Rejecting certain (outlier) point pairs
• Assigning an error metric to the current
  transform
• Minimizing the error metric

39
ICP Variant Point-to-Plane Error Metric

• Using point-to-plane distance instead of
  point-to-point lets flat regions slide along each
  other more easily Chen Medioni 91

40
Finding Corresponding Points

• Finding closest point is most expensive stage of
  ICP
• Brute force search O(n)
• Spatial data structure (e.g., k-d tree) O(log
  n)
• Voxel grid O(1), but large constant, slow
  preprocessing

41
Finding Corresponding Points

• For range images, simply project point Blais 95
• Constant-time, fast
• Does not require precomputing a spatial data
  structure

42
High-Speed ICP Algorithm

• ICP algorithm with projection-based
  correspondences, point-to-plane matchingcan
  align meshes in a few tens of ms.(cf. over 1
  sec. with closest-point)

43
Anchor Scans

• Alignment of consecutive scans leads to
  accumulation of ICP errors
• Alternative align all scans to an anchor scan,
  only switch anchor when overlap low
• Given anchor scans, restart after failed ICP
  becomes easier

44
Merging and Rendering

• Goal visualize the model well enoughto be able
  to see holes
• Cannot display all the scanned data accumulates
  linearly with time
• Standard high-quality merging methodsprocessing
  time 1 minute per scan

45
Merging and Rendering

• Real-time incremental merging and rendering
• Quantize samples to a 3D grid
• Maintain average normal of all pointsat a grid
  cell
• Point (splat) rendering
• Can be made hierarchical to conserve memory

46
Photograph
47
Real-time Scanning Demo

48
Postprocessing

• Goal of real-time display is to let user evaluate
  coverage, fill holes
• Quality/speed tradeoff
• Offline postprocessing for high-quality models

49
Merged Result
Photograph
Aligned scans
Merged
50
Future Work

• Technological improvements
• Use full resolution of projector
• Higher-resolution cameras
• Ideas from design of single-stripe 3D scanners
• Pipeline improvements
• Better detection of failed alignment
• Better handling of object texture combine with
  stereo?
• Global registration to eliminate drift
• More sophisticated merging
• Improve user interaction during scanning

51
Future Work

• Faster scanning
• Better stripe boundary matching
• Multiple cameras, projectors
• High-speed cameras
• Application in different contexts
• Small, hand-held
• Cart- or shoulder-mounted for digitizing rooms
• Infrared for imperceptibility

52
Rendering of Large Models

• Range scanners increasingly capable of producing
  very large models
• DMich models are 100 million to 1 billion samples
• Challenge how to allow viewing in real time
• Fast startup, progressive loading
• Traditional answer triangle meshes,
  simplification, hardware-accelerated rendering
• Impractical for such large models
• Alternative revisit basic data structure

53
QSplat

• Key observation a single bounding sphere
  hierarchy can be used for
• Hierarchical frustum and backface culling
• Level of detail control
• Splat rendering Westover 89

54
QSplat Node Structure
Position and Radius
Width of Cone of Normals
Tree Structure
Color (Optional)
Normal
13 bits
3 bits
14 bits
2 bits
16 bits
6 bytes
55
QSplat Node Structure
Position and Radius
Width of Cone of Normals
Tree Structure
Color (Optional)
Normal
13 bits
3 bits
14 bits
16 bits
2 bits

• Position and radius encoded relative to parent
  node
• Hierarchical coding vs. delta coding along a path
  for vertex positions

56
QSplat Node Structure
Position and Radius
Width of Cone of Normals
Tree Structure
Color (Optional)
Normal
13 bits
3 bits
14 bits
16 bits
2 bits
57
QSplat Node Structure
Position and Radius
Width of Cone of Normals
Tree Structure
Color (Optional)
Normal
13 bits
3 bits
14 bits
16 bits
2 bits
58
QSplat Node Structure
Position and Radius
Width of Cone of Normals
Tree Structure
Color (Optional)
Normal
13 bits
3 bits
14 bits
16 bits
2 bits
59
QSplat Rendering Algorithm

• Traverse hierarchy recursively

if (node not visible) Skip this branch else if
(leaf node) Draw a splat else if (size on screen
lt threshold) Draw a splat else Traverse children
60
Demo St. Matthew

• 3D scan of 2.7 meter statue at 0.25 mm
• 102,868,637 points
• File size 644 MB
• Preprocessing time 1 hour

61
Future Work

• Splats as primitive
• Unify rendering of meshes, volumes, point clouds
• Compatible with shading after rasterization
• Hybrid point/polygon systems
• High-level visibility / LOD frameworks
• Store different kinds of data at each node
  alpha, BRDF, scattering function, etc.
• Potentially could be used to unify
  image-based-rendering (IBR) techniques

62
Contributions

• Real-time 3D model acquisition system
• Video-rate 3D scanner for moving objects
• Analysis of ICP variants real-time algorithm
• Real-time merging and rendering
• Allows user to see model and fill holes
• QSplat interactive rendering of large 3D meshes
• Single data structure used for visibility
  culling,level-of-detail control, point
  rendering, compression
• Extension to network streaming I3D 2001

63
Acknowledgments

• Olaf Hall-Holt
• Lucas Pereira
• The Original DMich Gang Dave Koller, Sean
  Anderson, James Davis, Kari Pulli, Matt Ginzton,
  Jon Shade
• DMich, the next generation Gary King, Steve
  Marschner
• Graphics lab
• Advisor Marc Levoy
• Committee Pat Hanrahan, Leo Guibas, Mark
  Horowitz, Bernd Girod
• Family, friends
• Sponsors NSF, Interval, Honda, Sony, Intel