MultiCamera Real Time 3D Reconstruction of Urban Environments

1
Multi-Camera Real Time 3D Reconstruction of
Urban Environments

• Goals of the PhD project
• Overview of current techniques
• Future developments

2
The Problem

• Given a vehicle with multiple onboard cameras, we
  want to construct a 3D map of the environment it
  travels through.
• We also want to
• avoid active scanning technologies, for example
  laser range finding,
• avoid dependency on GPS, and
• perform processing onboard in real time.

3
Project Goals

• Develop a multi-camera system capable of
• tracking its own location,
• mapping its surroundings, and
• reconstructing a 3D model.
• Develop improvements to urban mapping systems.
• Reliability over long runs
• Utility of generated data

4
Current Approaches

• Commercial applications
• Research systems

5
Commercial Systems

• Many companies are producing 3D city maps using
  laser range scanning
• Often without taking visual imagery
• Google StreetView
• Uses a multi-camera system
• Provides a panoramic view at locations along a
  recorded route

6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
Research SystemsGeneral Structure
Image stream
See Pollefeys08, Cornelis08 for examples.
15
Research SystemsGeneral Structure
Image stream
GPS readings
INS readings
Vehicle tracking
3D reconstruction
Fusion
3D model
Trajectory
16
ExamplePollefeys et al. system

• Video input http//www.youtube.com/watch?vKSAJkN6
  QH8Q
• Reconstruction 1 http//www.youtube.com/watch?v3R
  F26nWzxhc
• Reconstruction 2 http//www.youtube.com/watch?vUd
  YX9UZDjzY
• See Pollefeys08

17
Above 4 vehicle mounted cameras. Upper Right A
resulting image set (for one frame of
video) Lower Right An example of a trajectory is
shown in green, with feature points in blue. The
3D reconstruction is underlaid. Below Views of a
3D reconstructed model.
Source Mordohai07 and http//www.cs.unc.edu/Res
earch/urbanscape/
18
Research SystemsGeneral Structure
Image stream
GPS readings
INS readings
Vehicle tracking
3D reconstruction
Fusion
3D model
Trajectory
19
Vehicle TrackingThe Problem

• Estimation of the vehicles position and
  orientation
• Some visual-only methods have been tried
• These tend to accumulate inaccuracies
• These often will not recognise a location visited
  twice
• Other researchers fuse GPS and INS data with the
  output from the vision algorithm
• The model is georegistered
• System is dependent upon GPS to avoid drift

20
Vehicle TrackingCurrent approaches

• SLAM (Simultaneous Localisation And Mapping)
• Active area of research, especially in UK
• Visual odometry
• Designed to solve vehicle tracking problem
• Used in Pollefeys et al. system
• Both systems have a similar data flow

21
Vehicle TrackingTypical Structure
Image stream
Extract 2D feature tracks
Extrapolate 3D position of features and camera
22
Research SystemsGeneral Structure
Image stream
GPS readings
INS readings
Vehicle tracking
3D reconstruction
Fusion
3D model
Trajectory
23
3D Reconstruction

• Using the trajectory information, we can perform
  stereo reconstruction
• Plane-sweep algorithm is widely used for this
• Number of planes can be kept small to improve run
  time
• Urban scenes often contain approximately planar
  objects
• Effective for Lambertian (non-glossy) surfaces
  (due to photo-consistency constraint)

24
3D ReconstructionThe Plane-Sweeping Algorithm

• Photo-consistency constraint
• Assumes that surfaces reflect the same light in
  all directions Lambertian surfaces
• A series of planes are swept to find matching
  regions
• The model is formed as the set of regions found
• Planar regions approximating the real scene

25
The Plane-Sweeping AlgorithmBasic Approach
Camera 1
Camera 2
Scene object, e.g. building
Scene object, e.g. building
26
The Plane-Sweeping Algorithm Basic Approach
Camera 1
Camera 2
Scene object, e.g. building
Scene object, e.g. building
27
The Plane-Sweeping AlgorithmMultiple Sweeping
Directions
Camera 1
Camera 2
Scene object, e.g. building
Scene object, e.g. building
28
3D Reconstruction Multiple Sweeping Directions
Camera 1
Camera 2
Scene object, e.g. building
Scene object, e.g. building
29
Research SystemsGeneral Structure
Image stream
GPS readings
INS readings
Vehicle tracking
3D reconstruction
Fusion
3D model
Trajectory
30
Model

• 3D mesh of textured polygons
• Can be reprojected to provide visually accurate
  views near to the vehicle trajectory
• Disadvantages in current systems
• It has holes, especially on reflective objects
• It contains artefacts around moving objects such
  as cars, people and trees
• There is no semantic knowledge about what objects
  in the model represent no cartographic
  information is present

31
Model Example
Source Akbarzadeh06
32
Assessing Model Quality
Source Pollefeys08
Ground truth model
Reconstructed model
Accuracy
Completeness
Above figures blue represents error lt 15cm
red represents error gt 60cm
33
Future Developments
34
Multi-Camera Algorithms

• Many techniques have been developed for single
  cameras.
• Work needs to be done on extending algorithms for
  the multi-camera case.
• Other potential developments for multi-camera
  systems
• Increase in sampling
• Increase in reliability
• Improved triangulation

35
Large-Scale Loop Closing

• Modelling a large-scale area produces a huge
  amount of data
• Recognising previously visited locations requires
  that the system can identify the scene from the
  model

36
Lighting Invariance

• Recognising a previously visited location may be
  difficult if lighting conditions have changed
  significantly
• Techniques exist for removing shadows and other
  lighting effects
• Could compensate for weather changes
• Night-time scenes present a greater challenge

37
SourceTroccoli08
38
SourceTroccoli08
39
Recognising and ReconstructingCars, People and
Trees

• Cornelis08 uses template matching to identify
  cars in the image stream
• Car pose is estimated
• The car is not reconstructed
• (to avoid specularity artefacts)
• Instead, a virtual car is placed in the scene
• This idea could be extended to include other
  classes of object
• For example, trees and people can be
  reconstructed using more specialised algorithms
• Could also be used to monitor vehicle activity

40
Source Cornelis08

• Template matching has been used to locate
  vehicles in the scene.

41
Source Cornelis08

• Top-down view of the vehicle scene model
• The mapping vehicle is shown in black with its
  trajectory indicated
• The position and orientation of other vehicles
  has been estimated. These are shown in red.

42
Source Cornelis08

• Vehicles in the scene are replaced with virtual
  models in the reconstruction.

43
Integrating Model and Map

• If the model is georeferenced, it could be
  integrated with map data or aerial scans to
  identify buildings and roads
• The map data could then be used to infer features
  not visible from the route
• Example the backs of buildings would not be
  visible, but their position could be inferred
  from the map

44
Integrating Model and Map
45
Summary

• Real-time 3D reconstruction of a moderately sized
  urban scene can be demonstrated
• System split into a pipeline
• Multiple processors and graphics cards
• Such systems need further improvements such that
  they are accurate, flexible, reliable and useful

46
References

• Akbarzadeh06 A. Akbarzadeh, J.-M. Frahm, P.
  Mordohai, B. Clipp, C. Engels, D. Gallup, P.
  Merrell, M. Phelps, S. N. Sinha, B. Talton, L.
  Wang, Q. Yang, H. Stewénius, R. Yang, G. Welch,
  H. Towles, D. Nistér, M. Pollefeys, Towards Urban
  3D Reconstruction from Video. 3DPVT 2006 Third
  International Symposium on 3D Data Processing,
  Visualization and Transmission available online
  at http//www.vis.uky.edu/dnister/Publications/20
  06/Urban/Akbarzadeh_UrbanReconstruction06.pdf
• Cornelis08 N. Cornelis, B. Leibe, K. Cornelis,
  L. Gool, 3D Urban Scene Modeling Integrating
  Recognition and Reconstruction. International
  Journal of Computer Vision Vol 78 No 2-3 July
  2008 available online at http//www.vision.ee.eth
  z.ch/bleibe/papers/cornelis-3durbanscene-ijcv07f
  inal.pdf

47
References

• Mordohai07 P. Mordohai, J.-M. Frahm, A.
  Akbarzadeh, B. Clipp, C. Engels, D. Gallup, P.
  Merrell, C. Salmi, S. Sinha, B. Talton, L. Wang,
  Q. Yang, H. Stewénius, H. Towles, G. Welch, R.
  Yang, D. Nistér, M. Pollefeys, Real-Time
  Video-Based Reconstruction of Urban Environments.
  Proceedings of the 2nd ISPRS International
  Workshop 3D-ARCH 2007 3D Virtual Reconstruction
  and Visualization of Complex Architectures
  available online at http//www.cs.unc.edu/mordoha
  i/public/UNC-UKY_UrbanReconstuction07.pdf
• Pollefeys08 M. Pollefeys, D. Nistér, J. M.
  Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp, C.
  Engels, D. Gallup, S. J. Kim, P. Merrell,
  Detailed Real-Time Urban 3D Reconstruction from
  Video. International Journal of Computer Vision
  Vol 78 No 2-3 July 2008 available online at
  http//vision.ai.uiuc.edu/qyang6/publications/det
  ailed_urban3D_IJCV08.pdf
• Troccoli08 A. Troccoli, P. Allen, Building
  Illumination Coherent 3D Models of Large-Scale
  Outdoor Scenes. International Journal of Computer
  Vision Vol 78 No 2-3 July 2008 available
  online at http//www1.cs.columbia.edu/allen/PAPER
  S/ijcv08.pdf.