CNES R

1
CNES RD studies in the field of information
extraction from EO images

• Alain Giros

2
Introduction

• Main objective of CNES RD activities in the
  field
• Image analysis and information extraction in
  order to promote the design of new products and
  applications.
• Status of these activities
• Done internally at CNES or with support of
  partners
• DLR and ENST within the CNES/DLR/ENST Competence
  Centre on Information Extraction and Image
  Understanding for Earth Observation
• Altamira, Alcatel, Magellium, DGA, ONERA, IGN,
  INRIA, CS-SI, BRGM, SCOT, SERTIT, ENSTB, Fresnel,
  Spikenet,
• Most are just finished, other are ongoing or just
  starting
• This presentation
• A short survey of these RD activities.
• Description of objectives and results, rarely
  with implementation details

3
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

4
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

5
Image Matching

• Goal
• Find the best geometric correspondance between an
  image and another object
• Another image
• A map
• A database of georeferenced objects
• Approaches
• Search for the optimal geometric transform which
  optimizes some similarity criterion
• Local or global optimization problem

6
Image Matching

• Some CNES actions in the field
• Study of similarity measures
• Radargrammetry for SAR images registration
• Image to map registration
• High resolution SAR/optical matching

7
Image Matching

• Some CNES actions in the field
• Study of similarity measures
• Radargrammetry for SAR images registration
• Image to map registration
• High resolution SAR/optical matching

8
Study of similarity measures
9
Study of similarity measures
10
Study of similarity measures
11
Image Matching

• Some CNES actions in the field
• Study of similarity measures
• Radargrammetry for SAR images registration
• Image to map registration
• High resolution SAR/optical matching

12
Radargrammetry for SAR images registration

• Standard Correlator
• ? ZNCC (Zero Normalized Cross Correlation)
• Contour based Correlators
• ? ZNCC on Roewa,
• ? ZNCC on Laplacian (ZNCC_lapla),
• ? Binary Overlap (BO)
• Radar specific Correlators
• ? Intensity Correlation (C_int),
• ? Reflectivity Correlation (C_ref)
• Mean Square Error Minimization Correlators
• ? Intensity Ratio (ratio)
• ? Variance Coefficient (C_var)
• Probabilistic Similarity Measures
• ? Mutual information
• ? Cluster Reward Algorithm

13
Radargrammetry for SAR images registration

• 1- ZNCC_lapla (Contours richness for fine mode
  Radarsat)
• 2- ZNCC
• 3- Corr_int - (Radiometric richness for fine mode
  Radarsat)
• 4- Binary overlap
• 5- Ratio (Objects resolution and constrasts)

14
Image Matching

• Some CNES actions in the field
• Study of similarity measures
• Radargrammetry for SAR images registration
• Image to map registration
• High resolution SAR/optical matching

15
Image to map registration
16
Image to map registration

• Segmentation
• contours (Canny-Deriche)
• Hysteresis thresholding
• Contours chaining
• Polygonal approximation

17
Image to map registration

• Significant segments (Hough transform)

18
Image to map registration
19
Image Matching

• Some CNES actions in the field
• Study of similarity measures
• Radargrammetry for SAR images registration
• Image to map registration
• High resolution SAR/optical matching

20
High Resolution SAR/Optical matching

• Match the objects above ground
• Compute the disparity map
• Transform the disparities in heights

21
High Resolution SAR/Optical matching

• No common primitives on the roof
• No matching patterns
• 3D very difficult

22
High Resolution SAR/Optical matching

• Common primitive features on the roof
• Very complex shapes
• 3D possible

23
High Resolution SAR/Optical matching
21
18
15
Height (m)
12
9
6
24
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

25
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

26
Altimetric measurements

• Goal
• Use interferometry and derived products in order
  to devise a 3D information

27
Altimetric measurements

• Some CNES actions in this field
• Urban DEM by interferometry on stable scatterers
• Interferogram unwrapping from a reference DTM

28
Altimetric measurements

• Some CNES actions in this field
• Urban DEM by interferometry on stable scatterers
• Interferogram unwrapping from a reference DTM

29
Ouputs of the interferometric chain
Urban DEM by interferometry on stable scatterers
Measured Subsidence
DTM error
Model Coherence
Mean Radiometry
Absolute displacement profiles
30
Urban DEM by interferometry on stable scatterers
FC Barcelona Stadium
Olympic Stadium
31
Altimetric measurements

• Some CNES actions in this field
• Urban DEM by interferometry on stable scatterers
• Interferogram unwrapping from a reference DTM

32
Interferogram unwrapping from a reference DTM

• Objectives
• Improve the altimetric accuracy of an existing
  DTM using interferometric SAR images

33
Interferogram unwrapping from a reference DTM
Ellipsoïd
SRTM 30 (horiz 700m - RMS 9 à 300m)
Differential interferograms for 3 ERS pairs
SRTM 3 (horiz 90m - RMS 16m)
IGN 50 (horiz 50m - RMS 2 to 40m)
SPOT PXS (horiz 40m - RMS 10 to 30 m)
34
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

35
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

36
3D reconstruction

• Goal
• Use of submetric resolution images for retrieving
  buildings in urban areas
• Preparation of Pléïades products
• DEM
• Orthoimages
• 3D views

37
3D reconstruction

• Some CNES actions in this field
• 3D building extraction without extra data
• 3D building extraction with exogeneous data
• Virtual flight over a 3D scene
• Methods for urban DEM quality assessment

38
3D reconstruction

• Some CNES actions in this field
• 3D building extraction without extra data
• 3D building extraction with exogeneous data
• Virtual flight over a 3D scene
• Methods for urban DEM quality assessment

39
3D Building extraction without extra data
tri-stereo disparities
Correlations at different altitudes
Disparity optimization within a correlation cube
40
3D Building extraction without extra data
Snakes from image gradients and shape
regularization constraints
3D Reconstruction
41
3D reconstruction

• Some CNES actions in this field
• 3D building extraction without extra data
• 3D building extraction with exogeneous data
• Virtual flight over a 3D scene
• Methods for urban DEM quality assessment

42
3D Building extraction with extra data
Cadastre superimposed to an approximated
orthoimage
Roof shape model
43
3D Building extraction with extra data
With cadastral maps
Without cadastral maps
44
3D reconstruction

• Some CNES actions in this field
• 3D building extraction without extra data
• 3D building extraction with exogeneous data
• Virtual flight over a 3D scene
• Methods for urban DEM quality assessment

45
Virtual flight over a 3D scene
Uniform walls
Ortho-image artifact
46
Virtual flight over a 3D scene
47
3D reconstruction

• Some CNES actions in this field
• 3D building extraction without extra data
• 3D building extraction with exogeneous data
• Virtual flight over a 3D scene
• Methods for urban DEM quality assessment

48
Methods for urban DEM quality assessment
Loss of small buildings
50 cm
70 cm
25 cm
DEM from subpixel correlation of tri-stereo data
at 3 resolutions
49
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

50
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

51
Image indexing

• Main problems
• Where is information in the images ?
• How to catch it efficiently ?
• How to structure and represent it ?
• How to handle its association with users
  semantics ?

52
Image indexing

• Some CNES actions in this field
• Relevant primitive image features
• Multiresolution range for primitive features
• Determination of object scale
• Separation of geometry and texture

53
Image indexing

• Some CNES actions in this field
• Relevant primitive image features
• Multiresolution range for primitive features
• Determination of object scale
• Separation of geometry and texture

54
Relevant primitive image features
Original
Haralick 1 Second horizontal angular moment
55
Relevant primitive image features
56
Relevant primitive image features
Original
QMF Decomposition
57
Relevant primitive image features
edges
lineaments
cloud
Numbers of linear segments (2)
sea
Lengths of linear segments (5)
desert
Pixel distribution (4)
city
Numbers of directions of linear segments (1)
forest
Frequencies of edge pixels (3)
field
58
Relevant primitive image features
Feature set Haralick Gabor QMF GMRF
Geometry
59
Image indexing

• Some CNES actions in this field
• Relevant primitive image features
• Multiresolution range for primitive features
• Determination of object scale
• Separation of geometry and texture

60
Multiresolution range for primitive features
61
Image indexing

• Some CNES actions in this field
• Relevant primitive image features
• Multiresolution range for primitive features
• Determination of object scale
• Separation of geometry and texture

62
Determination of object scale
63
Determination of object scale
64
Determination of object scale
65
Determination of object scale
Scale 4m
66
Determination of object scale
Scale 0,5 to 1,5 m
67
Determination of object scale
Scale 1 m
68
Determination of object scale
Scale 0,5 m
69
Image indexing

• Some CNES actions in this field
• Relevant primitive image features
• Multiresolution range for primitive features
• Determination of object scale
• Separation of geometry and texture

70
Separation of geometry and texture

• geometry

• texture

71
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

72
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

73
Land Cover

• Goal
• Detect and identify objects of interest
• Automatically build land cover or land use maps
• Approaches
• Via specific design of adapted models
• Through automatic learning
• Trends
• Towards very high resolution images gt objects
• Importance of multi-temporal information
• Source fusion

74
Land Cover

• Some CNES actions in the field
• Model based object recognition
• Machine learning based object recognition
• Pre-attentive visual recognition
• Automatic learning for Corine Land Cover maps
  production

75
Land Cover

• Some CNES actions in the field
• Model based object recognition
• Machine learning based object recognition
• Pre-attentive visual recognition
• Automatic learning for Corine Land Cover maps
  production

76
Model based object recognition
Alignments
Contours
Segmentation
Model
Adjacency Graphs
77
Model based object recognition
Rules based System
Neural Network
Detection
Primitive Extraction
78
Land Cover

• Some CNES actions in the field
• Model based object recognition
• Machine learning based object recognition
• Pre-attentive visual recognition
• Automatic learning for Corine Land Cover maps
  production

79
Machine learning based object recognition

• Learning step

Geometry Modelling
. .
Characterization
Highly-Dimensional Description Vector

• Complex geometric moments
• Fourier-Mellincoefficients

Gradient
80
Machine learning based object recognition

• Learning step

. .
. .
. .
. .
Examples From Class I
SVM IJ
. .
. .
. .
. .
Examples From Class J
81
Machine learning based object recognition

• Recognition Step

Geometry Modelling
. .
Characterization

• Complex geometric moments
• Fourier-Mellincoefficients

Gradient
82
Machine learning based object recognition

• Recognition Step

SVM 1N
Voting Strategy
SVM 1N-1
SVM 1j
SVM 13
SVM 12
. .
SVM IN
Class K
SVM Ij
SVM II1
SVM N-1N
83
Machine learning based object recognition
84
Land Cover

• Some CNES actions in the field
• Model based object recognition
• Machine learning based object recognition
• Pre-attentive visual recognition
• Automatic learning for Corine Land Cover maps
  production

85
Pre-attentive visual recognition
86
Pre-attentive visual recognition
87
Land Cover

• Some CNES actions in the field
• Model based object recognition
• Machine learning based object recognition
• Pre-attentive visual recognition
• Automatic learning of Corine Land Cover maps
  production

88
Learning of Corine Land Cover production

• CORINE production process
• Input data SPOT XS or LANDSAT image

89
Learning of Corine Land Cover production

• CORINE production process
• Photo-interpretation and drawing of a mask

90
Learning of Corine Land Cover production

• CORINE production process
• Digitalization

91
Learning of Corine Land Cover production

• CORINE production process
• Assembly

92
Learning of Corine Land Cover production

• CORINE production process
• Finalization

93
Learning of Corine Land Cover production

• Manual CORINE production process

94
Learning of Corine Land Cover production

• Learning of CORINE classification

95
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

96
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

97
Multitemporal Analysis

• Goal
• To detect and analyze temporal evolutions in an
  image time series
• Time series specificities
• Resolution range (from 0.5 m to 5000 m)
• Time interval (regular/irregular, from some
  minutes to several weeks)
• Temporal evolutions
• At low resolution
• Show moving objects (clouds, algae,)
• Vary rather smoothly
• At high resolution
• Show either steady objects or temporal outliers
• Vary abruptly (anthropic activities, snow,)

98
Multitemporal Analysis

• Some CNES actions in the field
• Abrupt change detection
• Multitemporal registration
• Multitemporal classification
• Multitemporal segmentation
• Multitemporal cloud assessment
• Multitemporal Information Mining

99
Multitemporal Analysis

• Some CNES actions in the field
• Abrupt change detection
• Multitemporal registration
• Multitemporal classification
• Multitemporal segmentation
• Multitemporal cloud assessment
• Multitemporal Information Mining

100
Abrupt Change Detection
101
Abrupt Change Detection
Change index
PIAO
Impact map
Damaged area
102
Multitemporal analysis
Multitemporal ADAM dataset
103
Multitemporal ADAM dataset
104
Multitemporal Analysis

• Some CNES actions in the field
• Abrupt change detection
• Multitemporal registration
• Multitemporal classification
• Multitemporal segmentation
• Multitemporal cloud assessment
• Multitemporal Information Mining

105
Multitemporal Registration

• Disparity measurements from image to image paths

Column displacement series
Line displacement series
106
Multitemporal Registration
107
Multitemporal Analysis

• Some CNES actions in the field
• Abrupt change detection
• Multitemporal registration
• Multitemporal classification
• Multitemporal segmentation
• Multitemporal cloud assessment
• Multitemporal Information Mining

108
Multitemporal Classification
f(1) B1 f(1) B2 f(1) B3 f(2) B1 f(2) B2 f(2)
B3 f(3) B1 f(3) B2 f(3) B3 r(1) B1 r(1) B2 r(1)
B3 r(2) B1 r(2) B2 r(2) B3 r(3) B1 r(3) B2 r(3)
B3 30 classes
109
Multitemporal Analysis

• Some CNES actions in the field
• Abrupt change detection
• Multitemporal registration
• Multitemporal classification
• Multitemporal segmentation
• Multitemporal cloud assessment
• Multitemporal Information Mining

110
Multitemporal Segmentation

• Individual segmentations
• Reference segmentation

111
Multitemporal Analysis

• Some CNES actions in the field
• Abrupt change detection
• Multitemporal registration
• Multitemporal classification
• Multitemporal segmentation
• Multitemporal cloud assessment
• Multitemporal Information Mining

112
Multitemporal Cloud Assessment
113
Multitemporal Analysis

• Some CNES actions in the field
• Abrupt change detection
• Multitemporal registration
• Multitemporal classification
• Multitemporal segmentation
• Multitemporal cloud assessment
• Multitemporal Information Mining

114
Multitemporal Information Mining
115
Multitemporal Information Mining
Wheat and pea agricultural practice
116
Main areas

• Image matching
• Altimetric measurements
• 3D reconstruction
• Image indexing
• Land cover
• Multitemporal analysis

117
Conclusion

• We just made a short and uncomplete survey of the
  CNES activities in the field.
• Information extraction from images is
  fundamental
• to feed other systems and processes with usefull
  data
• to make the images simpler to understand
• to help filling the gap between images and users
• Our motto
• Generic Approaches for Generic Information.