Automated 3D Model Construction for Urban Environments

1

Next Generation 4D Distributed Modeling and
Visualization
Automated 3D Model Construction for Urban
Environments
Christian Frueh Avideh Zakhor
University of California, Berkeley
June 13, 2002
2
Presentation Overview

• Introduction
• Ground based modeling
• Mesh processing
• Airborne modeling
• Aerial photos
• Airborne laser scans
• 3D Model Fusion
• Rendering
• Conclusion and Future Work

3
Introduction
Goal Generate 3D model of a city for virtual
walk/drive/fly-thrus and simulations

• Fast
• Automated
• Photorealistic

4
Introduction
3D model of building façades
3D Model of terrain and building tops
5
Airborne Modeling
Acquisition of terrain shape and top-view
building geometry
Goal

• Available Data
• Aerial Photos
• Airborne laser scans

Texture from aerial photos
Geometry 2 approaches
I) stereo matching of photos
II) airborne laser scans
6
Airborne Modeling
Approach I Stereo Matching
(last year)

• Stereo photo pairs from city/urban areas, 60
  overlap

Semi-Automatic
Manual
Automated

• Segmentation

• Camera parameter computation,
• Matching,
• Distortion reduction,
• Model generation

7
Stereo Matching
Stereo pair from downtown Berkeley and the
estimated disparity after removing perspective
distortions
8
Stereo Matching Results
Downtown Oakland
9
Airborne Modeling
Approach II Airborne Laser Scans (LiDAR)
Scanning city from plane

• Resolution 1 scan point/m2
• Berkeley 40 million scan points

point cloud
10
Airborne Laser Scans

• Re-sampling point cloud
• Sorting into grid
• Filling holes

Map-like height field
usable for

• Mesh Generation

• Monte Carlo Localization

11
Textured Mesh Generation for LiDAR
1. Connecting grid vertices to mesh
2. Applying Q-slim simplification
3. Texture mapping

• Semi-automatic
• Manual selection of few correspondence points 10
  mins/entire Berkeley
• Automated camera pose estimation
• Automated computation of texture for mesh

12
Airborne Model from LiDAR
Downtown Berkeley
http//www-video.eecs.berkeley.edu/frueh/3d/airbo
rne/
13
Airborne Model from LiDAR
East Berkeley campus with campanile
14
Airborne Model from LiDAR
15
Ground Based Modeling
Acquisition of highly detailed 3D building façade
models
Goal

• Acquisition vehicle
• Truck with rack
• 2 fast 2D laser scanners
• digital camera

• Scanning setup
• vertical 2D laser scanner for geometry capture
• horizontal scanner for pose estimation

16
Scan Matching Initial Path Computation
Horizontal laser scans

• Continuously captured during vehicle motion

• Overlap

Relative position estimation by scan-to-scan
matching
Translation (?u,?v) Rotation ??
Adding relative steps (?ui, ?vi, ??i)
t t0
??
t t1
(?u, ?v)
path (xi,yi,?i)
Scan matching
3 DOF pose (x, y, yaw)
17
6 DOF Pose Estimation From Images

• Scan matching cannot estimate vertical motion
• Small bumps and rolls
• Slopes in hill areas
• Extend initial 3 DOF pose by deriving missing 3
  DOF (z, pitch, roll) from images
• Full 6 DOF pose of the vehicle is important
  affects
• Future processing of the 3D and intensity data
• Texture mapping of the resulting 3D models

18
6 DOF Pose Estimation From Images

• Central idea photo-consistency

• Each 3D scan point can be projected into images
  using initial 3 DOF pose
• If pose estimate is correct, point should appear
  the same in all images
• Use discrepancies in projected position of 3D
  points within multiple images to solve for the
  full pose

19
6 DOF Pose Estimation Algorithm

• 3 DOF of laser as initial estimate
• Project scan points into both images
• If not consistent, use image correlation to find
  correct projection
• Solve for pose, using Ransac for robustness

20
6 DOF Pose Estimation Algorithm

• Use 3 DOF pose of laser scan matching as initial
  estimate
• Some scan points are taken simultaneously with
  each image, the exact projection of these scan
  points are know in this image
• Image correlation to find the projection of these
  scan points in other images
• Not always correct results
• Many outliers
• Reprojected error is minimized across many images
  to find the pose of each image
• RANSAC for robustness

21
6 DOF Pose Estimation Results
with 3 DOF pose
with 6 DOF pose
22
6 DOF Pose Estimation Results
 
23
Monte Carlo Localization (1)
Previously Global 3 DOF pose correction using
aerial photography
a) path before MCL correction
b) path after MCL correction
After correction, points fit to edges of aerial
image
24
Monte Carlo Localization (2)
Extend MCL to work with airborne laser data and 6
DOF pose
Now
No perspective shifts of building tops, no shadow
lines

• Fewer particles necessary, increased computation
  speed
• Significantly higher accuracy near high buildings
  and tree areas

Use terrain shape to estimate z coordinate of
truck

• Correct additional DOF for vehicle pose (z,
  pitch, roll)
• Modeling not restricted to flat areas

25
Monte Carlo Localization (3)
Track global 3D position of vehicle to correct
relative 6 DOF motion estimates
Resulting corrected path overlaid with airborne
laser height field
26
Path Segmentation
24 mins, 6769 meters
vertical scans 107,082
scan points 15 million
Too large to process as one block!
27
Path Segmentation
Resulting path segments overlaid with edges of
airborne laser height map
28
Simple Mesh Generation
29
Simple Mesh Generation
Triangulate
Point cloud
Mesh

• Problem
• Partially captured foreground objects
• erroneous scan points due to glass reflection

30
Façade Extraction and Processing (1)
1. Transform path segment into depth image
31
Façade Extraction and Processing (2)
3. Separate depth image into 2 layers
foreground trees, cars etc.
background building facades
32
Façade Extraction and Processing (3)
4. Process background layer

• Detect and remove invalid scan points

• Fill areas occluded by foreground objects by
  extending geometry from boundaries
• Horizontal, vertical, planar interpolation, RANSAC

• Apply segmentation

• Remove isolated segments

• Fill remaining holes in large segments

• Final result clean background layer

33
Façade Extraction Examples (1)
with processing
without processing
34
Façade Extraction Examples (2)
without processing
with processing
35
Façade Extraction Examples (3)
without processing
with processing
36
Statistics for Façade Processing
Visual inspection and comparison downtown path
segments
processed vs. original
Significantly better 35 56
Better 17 27
Same 11 17
Worse 0 0
Significantly worse 0 0
Total 63 100

• processing never resulted in degradation

37
Facade Processing
38
Foreground Removal
39
Mesh Generation
Downtown Berkeley
40
Automatic Texture Mapping (1)
Camera calibrated and synchronized with laser
scanners
Transformation matrix between camera image and
laser scan vertices can be computed
1. Project geometry into images
2. Mark occluding foreground objects in image
3. For each background triangle
Search pictures in which triangle is not
occluded, and texture with corresponding picture
area
41
Automatic Texture Mapping (2)
Efficient representation texture atlas
Copy texture of all triangles into collage
image
42
Automatic Texture Mapping (3)
Large foreground objects Some of the filled-in
triangles are not visible in any image!
texture holes in the atlas
43
Automatic Texture Mapping (4)
Texture holes marked
44
Automatic Texture Mapping (5)
45
Ground Based Modeling - Results
Façade models of downtown Berkeley
46
Ground Based Modeling - Results
Façade models of downtown Berkeley
47
Model Fusion
Fusion of ground based and airborne model to one
single model
Goal
Façade model
Airborne model
Model Fusion

1. Registration of models
2. Combining the registered meshes

48
Registration of Models
Models are already registered with each via
Monte-Carlo-Localization !
49
Preparing Ground Based Models
Intersect path segments with each other Remove
degenerated, redundant triangles in overlapping
areas
original mesh
redundant triangles removed
50
Preparing Airborne Model
Ground based model has 5-10 times higher
resolution

• Remove facades in airborne model where ground
  based geometry is available
• Add ground based façades
• Fill remaining gaps with a blend mesh to hide
  model transitions

51
Preparing Airborne Model
Initial airborne model
52
Preparing Airborne Model
Remove facades where ground based geometry is
available
53
Combining Models
Add ground based façade models
54
Combining Models
Fill remaining gaps with a blend mesh to hide
model transitions
55
Model Fusion - Results
56
Rendering
Ground based models

• Up to 270,000 triangles, 20 MB texture per path
  segment
• 4.28 million triangles, 348 MB texture for 4
  downtown blocks

• Difficult to render interactively!

• Subdivide model and create multiple
  level-of-details (LOD)
• Generate scene graph, decide which LOD
  to render when

57
Multiple LODs for façade meshes
Highest LOD

• Qslim mesh simplification
• Texture subsampling

Lower LOD

• Geometry 10

• Texture 25

of original mesh
58
Façade Model Subdivision for Rendering
Subdivide 2 highest LODs of façade meshes along
cut planes
Sub-scene
LOD 0
Submesh
LOD 1
Path segment
LOD 0
Submesh
LOD 1
Global scene
LOD 0
LOD 2
Submesh
LOD 1
59
Interactive Rendering
Downtown blocks Interactive rendering with
web-based browser!
60
Future Work

• Resolution enhancement and post-processing of
  LIDAR data
• Devise new data acquisition system and algorithms
  to capture
• both sides of street simultaneously
• texture for upper parts of tall buildings
• Include foreground objects in model
• Interactive rendering
• Add temporal component to dynamically update
  models
• Compact representation