Automated DSM generation

1
Section 4 Automated DSM generation Image
Matching, Surface Reconstruction, and DSM
Analysis towards object extraction Nicolas
Paparoditis
2
Automatic Surface reconstruction from VHR
imagery Where are we now?

• Research has been improving in the last years
• Data sources and technologies have been
  improving
• New tools coming from computer vision and image
  processing are starting to get integrated
• What is working in production? How close are we
  to production?
• What is still research at short, medium and long
  term?
• I will show many case studies from the MATIS
  laboratory but similar work exists worlwide

3
Surface reconstruction from image matching how
does it work?

• A surface is generally described by
• a set of samples (obtained by total station,
  GPS, photogrammetry)
• a function to interpolate in between the samples
  (e.g. triangulation)
• Image matching algorithms work in the same way
• Matching image samples/features
• Reconstructing the surface covering the features
• Problems with VHR images
• discontinuities, steep slopes, and occlusions
• homogeneous areas, repetitive textures
• specular effects
• DIFFICULT IN URBAN AREAS

4
Our Road Map

• Image feature matching techniques and impact of
  multi-viewing
• - points, slopes
• - depth discontinuities
• - edges, segments
• Surface reconstruction strategy
• - raster-based optimisation techniques
• - DSM analysis and segmentation
• - feature-based interpolation techniques
• - model-based reconstruction

5
How do we match features from image space or
object space?
From object space vertical line locus constraint
6
How do we match features from image space or
object space?
From image space epipolar geometry constraint
O2
O1
P1 (i1,j1)
(imin,j1)
E
(imax,j1)
(imin,j1)
C1
C2
Zmin
Zmax
Epipolar Resampling
Image Matching a 1D problem? Yes, and
no. Cleaner in 2D due to no perfect estimation of
image poses All matches can be reinjected in the
bundle adjutment
7
Matching points
Image Right
Image Left
8
Matching points on (steep) slopes

• Least squares matching or adaptive window shape
  matching

Baltsavias 91
9
Matching points close to depth discontinuities

• Adaptive shape matching

10
Matching points close to depth discontinuities

• Adaptive shape matching

Paparoditis al 98
Cord al 99
1 metre satellite simulation
Classical template matching
Adaptive shape matching
13x13 window size
11
Matching (interest?) points from multiple views
3x3 window size
Correlation score
Robust filtering
12
Matching (interest?) points from multiple views
3D view of robust points
13
Matching segments

• From two views, to achieve robustness, we need a
  global matching technique relaxation, dynamic
  programming
• From more views, the problem is better-posed the
  geometry of the segment itself is self-sufficient

14
Matching segments
Taillandier al 02
15
Matching segments
Taillandier al 02
16
Matching edgels
Karner al 05
Jung al 02
17
Matching edgels
Karner al 05
Jung al 02
18
Matching edgels
Karner al 05
Jung al 02
19
Matching edgels
Jung al 02
Karner al 05
20
What Surface Reconstruction?
From a topographic to a cartographic
Reconstruction Degree of a priori knowledge

• Reality
• Raster-based
• Feature-based interpolation
• Model-based

21
Raster-based and energy minimisation-based
matching techniques

• Matching by the research of a field of
  parallax/surface G minimising E(G)
• 1. Differential approaches (e.g level sets)
• Precise and possibly quick
• - Very good initialisation, criteria and
  variables differentiable
• 2. Combinatory approaches (graph theory)
• No initialisation, no derivation
• - Discretisation necessary

Data attachment
Regularisation term
22
Data attachment term
Matching volume
23
Looking for a minima of Ea Energy Gradient
Data attachment
Regularisation term
24
Looking for a minimal Ea can be seen as a a
minimal length path problem

• Dynamic programming on a profile of the surface
• Matching conjugate epipolar lines
• Artefacts due to dissimetry in the processing of
  image lines and columns
• Figural continuity constraint as a solution to
  minimize problems

In object space
Baillard 97
In image space
25
z
RoyCox Optimal flow algorithm Roy Cox 98

• A 3D graph can be constructed such as
• the nodes are the possible values of Z (voxels
  in Fig)
• the edges are the pairs of neighboring Z (in xy
  or in z)
• the planimetric edges are assigned the
  regularization cost
• the altimetric edges are assigned the data
  attachment cost
• Roy Cox showed that
• the surfaces are the set of graph cuts between
  the set of nodes of max. Z and min. Z
• the weight of a cut associated to a surface is
  exactly equal to Ea
• Finding the surface minimizing Ea can be seen as
  finding a minimal cut in a graph
• solvable in polynomial time with classical
  minimal cut and maximal flow graph theory
  algorithms.

x
26
Integration in a multi-resolution scheme
Pierrot-Dessilligny Paparoditis 06

• To limit the combinatory, a multi-resolution
  approach
• At an initial step, of resolution 2N, we explore
  all possible disparities.... etc.
• At current step, of resolution 2K, initialisation
  with the previous one of résolution 2K1
• Morphological dilatation to define research space

Matching
Matching
Matching
Matching
27
Application to SPOT5-HRS

• 1200060000 image
• 3x3 window sizes
• 0.2
• 3 days on 1.8 GhZ PC
• 1 hour on a cluster

Omnidirectionnal shading
28
Application to Pléiädes simulations
tri-stereo forward-nadir-backward, 70 cm,
Toulouse
3x3 window sizes a 0.04
Omnidirectionnal shading
29
Application to aerial digital frame camera images
6 aerial images 20 cm, Amiens 20 cm, Marseille
3x3 window sizes a 0.02
Omnidirectionnal shading
30
Images Orientations
global
Scene analysis and Thematic interpolation
Image Matching
Stereo DSM
Scene Analysis and Segmentation
Mono thematic ROI Ground, Building, Vegetation
local
Thematic 3D Features Extraction
3D Points, 3D Line Segments, ...
Final, complete SURFACE Model of the observed
scene
Thematic 3D Interpolation
Thematic, local models.
Global Reconstruction
31
Scene analysis and DSM segmentation
Segmentation Classification (terrain,
vegetation, building, etc)
DSM
Images
32
Interpolation-based surface reconstruction
(buildings)
Paparoditis al 01

• Using raster-based DSM to reduce search space

Maillet al 02
Zhang al 05
33
Interpolation-based surface reconstruction
(buildings)
Jung al 03

• With edgels

Karner al 06
34
Interpolation-based surface reconstruction
(buildings)
Jung al 03

• With edgels

Karner al 06
35
Model-based surface reconstruction Building
reconstruction

• Heuristic methods
• Energy minimisation methods

Haala al 98
Vosselman Suveg 01
Rottensteiner 01
Taillandier 04
Jibrini al 02
36
Model-based surface reconstruction Building
reconstruction

• Using ground footprints databases and the DSM

37
Model-based surface reconstruction Building
reconstruction

• Finding footprints in the DSM
• Object approach using marked point processes
  energy minimization with a bayesian framework

Ortner 04 Lafarge al 06
Pléïades simulations Tri-stereo
forward-nadir-backward 70 cm, Amiens
38
Model-based surface reconstruction Terrain
reconstruction

• Robust Adaptive elastic grid interpolation
  methods
• Ground features/breaklines and uncertainties can
  be injected in the minimisation

Champion al 06
245m
Initial DSM (in white, points higher than 245m
Vegetation and Buildings Masks (in black) over
initial DSM
final DTM
165m
39
Model-based surface reconstruction 3D City Models
Bati3D Software

• Amiens
• 20 cm images
• 3000 ground footprints
• Automatic reconstruction
• 8 hour of editing

Kaartinen al 05
EuroSDR Building Extraction Comparison
40
Model-based surface reconstruction 3D City Models
Bati3D Software

• Marseille
• 6 camera-head
• 4 IR,R,G,B
• 2 VHR tilted panchro for façades

41
Conclusion

• Research in this area has significantly improved
  and is still improving thanks to the integration
  of computer vision and image processing tools
• Raster-based energy minimisation matching
  techniques are very efficient and operationnal at
  a production level
• Image quality and multi-viewing are capital to
  both increase signal to noise ratio and
  robustness
• Multi-viewing (along and across track) is the key
  solution to DSM generation automation agility of
  incoming satellites and future constellations
  should help
• DSMs are a key feature for scene understanding
  and object extraction
• Feature-based interpolation techniques ensure a
  duality between surface and object extraction
  which is very helpfull for scene management and
  3D GIS