RealTime Motion and Structure Estimation from Moving Cameras

1
Real-Time Motion and Structure Estimation from
Moving Cameras
David Nistér and Andrew Davison
2
Very Rough Outline

• 830-1000 "Real-Time Robust Camera Motion
  Estimation"
• -Feature Tracking
• -RANSAC
• -Relative Orientation
• -Geometry Tools
• -1000-1030 Coffey break
• 1030-1200-"Real-Time SLAM with a Single
  Camera".
• - SLAM
• - Uncertainty propagation state, covariance and
  EKF
• - Motion model for an agile camera and EKF
  prediction/update
• - Active measurement using dynamically-calculated
  search regions
• - Real-time augmented reality and map management

3
(No Transcript)
4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
Sparse
Dense
8
Sparse Reconstruction
Structure and Motion
Feature Matching
Feature Detection
Original Video
Camera Motion
Dense Reconstruction
Window-based stereo
at multiple scales
Bayesian
Surface
Model Out
Median Fusion
framework
triangulation
driven by
of depth maps
and texturing
graph cuts
9
Geometry Tools
10
Structure and Motion
Feature Matching
Feature Detection
Original Video
3D Reconstruction
11
Feature Detection
Harris Corners
Structure and Motion
Feature Matching
Feature Detection
Original Video
3D Reconstruction
12
Feature Detection
Harris Corners
13
Feature Detection
Harris Corners
14
Feature Detection
Harris Corners
Second Moment Matrix
15
Feature Detection
5x5 Max
Image
k
Saturation
Features
16
Feature Detection
17
Structure and Motion
Feature Matching
Feature Detection
Original Video
3D Reconstruction
18
Feature Matching/Tracking
Structure and Motion
Feature Matching
Feature Detection
Original Video
3D Reconstruction
19
Feature Matching/Tracking
20
Feature Matching/Tracking
21
Feature Matching/Tracking
Only retain bidirectional matches No loops
because of symmetry d(a,b)d(b,a)
22
Feature Matching/Tracking
Normalized Correlation
23
Feature Matching/Tracking
24
(No Transcript)
25
Feature Matching/Tracking
26
Demonstration of live feature tracking and
MSERs(Matas et al.)
27
Homography
Mosaic
Structure and Motion
Feature Matching
Feature Detection
Original Video
3D Reconstruction
28
Structure and Motion
Feature Matching
Feature Detection
Original Video
3D Reconstruction
29
Estimate or posterior likelihood output
Hypothesis Generator
Probabilistic Formulation
Precise Formulation
Data Input
30
RANSAC- Random Sample Consensus
31
RANSAC- Random Sample Consensus
Line Hypotheses
Points
32
RANSAC
?
Hypotheses
500
Observations
1000
500 x 1000 500.000
33
Preemptive RANSAC
Depth-first Preemption
Hypotheses
500
Observations
1000
500 x ???? ???????
34
Preemptive RANSAC
Breadth-first Preemption
Hypotheses
500
Chunksize
100
Observations
1000
500 x 200 100.000
Overhead 100 microseconds
35
Preemptive RANSAC
Observed Tracks
36
Preemptive RANSAC
37
Preemptive RANSAC
38
Preemptive RANSAC
Total Time for RANSAC 500 Hypothesis
generator 100.000 Observation
likelihood (Iterative Refinement)
39
Relative Orientation
40
Calibrated vs Uncalibrated
41
Constraints
42
Constraints
SingularValues
43
2 Views
3 Views
6p Quan, 1994
8p von Sanden, 1908 Longuet-Higgins, 1981
4p Nister, Schaffalitzky, 2004
7p R. Sturm, 1869
5p Nister, 2003
6p Philip, 1996
5p Kruppa 1913 Nister 2003
44
(No Transcript)
45
The Epipoles and the Epipolar Line Homography
46
The Epipolar Constraint
h
47
The Kruppa Constraints
h
48
The Five Point Problem
What is R,t ?
E. Kruppa, Zur Ermittlung eines Objektes aus zwei
Perspektiven mit Innerer Orientierung, 1913.
49
O. Faugeras and S. Maybank, Motion from Point
Matches Multiplicity of Solutions, 1990.
J. Philip, A Non-Iterative Algorithm for
Determining all Essential Matrices Corresponding
to Five Point Pairs, 1996.
B. Triggs, Routines for Relative Pose of Two
Calibrated Cameras from 5 Points, 2000.
D. Nister, An Efficient Solution to the
Five-Point Relative Pose Problem, 2002.
50
The solution is minimal in two respects
51
Nr of Roots
Average 4.55
52
Nr of Solutions
Average 2.74
53
10 Solutions
0.067, 0.287 lt gt 0.329,1.297 0.254,
0.0646 lt gt 0.523,1.0807 0.239, -0.213 lt
gt 0.517,0.645 -0.710, -0.693 lt gt
-0.141,0.157 0.661, -0.307 lt gt 0.950,
0.773
54
The 5-point algorithm (Nistér PAMI 04)
55
The 5-point algorithm (Nistér CVPR 03)
E
R,t
56
The 5-point algorithm (Nistér PAMI 04)
R,t
E
57
The 5-point algorithm (Stewénius et al)
10 x 10 Action Matrix
Eigen-Decomposition
R,t
E
58
(No Transcript)
59
Noise
Minimal Cases, Sideways Motion
Depth 0.5 Baseline 0.1 Field of View 45 degrees
60
Direction
50 points
Depth 0.5 Baseline 0.1 Field of View 45 degrees
61
Baseline
Minimal Cases, Sideways Motion
Depth 0.5 Baseline 0.1 Field of View 45 degrees
62
Easy Conditions
Realistic Conditions
Correct Calibration
63
Focal Length Miscalibration
0.7
0.5
0.3
0.05
3.0
2.0
1.5
1.3
64
Planar Ambiguity, Uncalibrated
2Degrees of Freedom
65
Planar Ambiguity, Calibrated
2-Fold or Unique
66
Depth
67
Uncertainty in Epipolar Geometry work with Chris
Engels
Single Estimate often ill posed
Representation of posterior likelihood well
posed, but computationally challenging
68
Uncertainty in Epipolar Geometry work with Chris
Engels
Single Estimate often ill posed
Representation of posterior likelihood well
posed, but computationally challenging
69
Epipoloscope Demo
70
Triangulation
71
Triangulation

• 2 Stages Correction Ideal Triangulation

72
Triangulation

• Rays Intersect lt-gt Rays Coplanar

73
Triangulation

• One parameter family Balance the error

74
Triangulation

• One parameter family Balance the error

x
x
e
e
75
Triangulation

• One parameter family Balance the error

76
Triangulation

• One parameter family Balance the error

• Max-Norm -gt Quartic (Closed form, Nistér)

• L2-Norm -gt Sextic (Hartley Sturm)

• Directional Error -gt Quadratic (Oliensis)

77
Optimal 3 View Triangulation work with Henrik
Stewenius and Fred Schaffalitzky
47 Stationary Points
78
Sampson Approximation
Where
is the covariance matrix of detected image
features and
are the incidence function and its Jacobian
and
79
Sampson Approximation
For two views this leads to
For three views, an approximation of the distance
to trifocal Incidence can be found by tensor
contractions and Cramers rule in lt1 microsecond
Assuming Cauchy distribution
80
Bundle Adjustment
81
Trust Region Methods
x
82
Trust Region Methods
x
dx
83
Trust Region Methods
x
dx
84
Trust Region Methods
x
dx
85
Least squares with Gauss Newton Approximation
Results in cost function approximation
Identify Gradient
and Hessian Approximation
86
Bundle Adjustment
Block LU factorization
Multiply by
Multiply by
First order sparsity
87
Bundle Adjustment
Block LU factorization
Multiply by
Multiply by
88
Bundle Adjustment
Range
Bundle Adjuster
Domain
89
Reprojection constraint
World point
Image point
3DCamera
Radial Distortion
spaf
RC
RP
T
90
2D-3D Pose
91
The Generalized 3-Point Problem
92
The Generalized 3-Point Problem
93
(No Transcript)
94
(No Transcript)
95
(No Transcript)
96
(No Transcript)
97
(No Transcript)
98
(No Transcript)
99
(No Transcript)
100
(No Transcript)
101
(No Transcript)
102
(No Transcript)
103
(No Transcript)
104
(No Transcript)
105
(No Transcript)
106
(No Transcript)
107
(No Transcript)
108
(No Transcript)
109
(No Transcript)
110
(No Transcript)
111
(No Transcript)
112
(No Transcript)
113
(No Transcript)
114
(No Transcript)
115
(No Transcript)
116
(No Transcript)
117
(No Transcript)
118
(No Transcript)
119
(No Transcript)
120
(No Transcript)
121
(No Transcript)
122
(No Transcript)
123
(No Transcript)
124
(No Transcript)
125
(No Transcript)
126
(No Transcript)
127
(No Transcript)
128
(No Transcript)
129
(No Transcript)
130
(No Transcript)
131
(No Transcript)
132
(No Transcript)
133
(No Transcript)
134
(No Transcript)
135
(No Transcript)
136
(No Transcript)
137
(No Transcript)
138
(No Transcript)
139
(No Transcript)
140
(No Transcript)
141
(No Transcript)
142
(No Transcript)
143
(No Transcript)
144
(No Transcript)
145
(No Transcript)
146
(No Transcript)
147
(No Transcript)
148
(No Transcript)
149
(No Transcript)
150
(No Transcript)
151
(No Transcript)
152
(No Transcript)
153
(No Transcript)
154
(No Transcript)
155
(No Transcript)
156
(No Transcript)
157
(No Transcript)
158
(No Transcript)
159
(No Transcript)
160
(No Transcript)
161
(No Transcript)
162
(No Transcript)
163
(No Transcript)
164
(No Transcript)
165
(No Transcript)
166
(No Transcript)
167
(No Transcript)
168
(No Transcript)
169
(No Transcript)
170
(No Transcript)
171
(No Transcript)
172
(No Transcript)
173
(No Transcript)
174
(No Transcript)
175
(No Transcript)
176
(No Transcript)
177
(No Transcript)
178
(No Transcript)
179
(No Transcript)
180
(No Transcript)
181
(No Transcript)
182
(No Transcript)
183
(No Transcript)
184
(No Transcript)
185
(No Transcript)
186
(No Transcript)
187
(No Transcript)
188
(No Transcript)
189
(No Transcript)
190
(No Transcript)
191
(No Transcript)
192
(No Transcript)
193
(No Transcript)
194
(No Transcript)
195
(No Transcript)
196
(No Transcript)
197
(No Transcript)
198
(No Transcript)
199
(No Transcript)
200
(No Transcript)
201
(No Transcript)
202
(No Transcript)
203
(No Transcript)
204
(No Transcript)
205
(No Transcript)
206
(No Transcript)
207
(No Transcript)
208
(No Transcript)
209
(No Transcript)
210
(No Transcript)
211
(No Transcript)
212
(No Transcript)
213
(No Transcript)
214
(No Transcript)
215
(No Transcript)
216
(No Transcript)
217
(No Transcript)
218
(No Transcript)
219
(No Transcript)
220
(No Transcript)
221
(No Transcript)
222
(No Transcript)
223
(No Transcript)
224
(No Transcript)
225
(No Transcript)
226
(No Transcript)
227
(No Transcript)
228
(No Transcript)
229
(No Transcript)
230
(No Transcript)
231
(No Transcript)
232
(No Transcript)
233
(No Transcript)
234
(No Transcript)
235
(No Transcript)
236
(No Transcript)
237
(No Transcript)
238
(No Transcript)
239
(No Transcript)
240
(No Transcript)
241
(No Transcript)
242
(No Transcript)
243
(No Transcript)
244
(No Transcript)
245
(No Transcript)
246
(No Transcript)
247
(No Transcript)
248
(No Transcript)
249
(No Transcript)
250
(No Transcript)
251
(No Transcript)
252
(No Transcript)
253
(No Transcript)
254
(No Transcript)
255
(No Transcript)
256
(No Transcript)
257
(No Transcript)
258
(No Transcript)
259
(No Transcript)
260
(No Transcript)
261
(No Transcript)
262
(No Transcript)
263
(No Transcript)
264
Seamlessly into the classical case
265
6-point pose
Linear, stack 5 point constraints, results in
pencil of cameras
Projects world point onto image line
Correct point by perpendicular projection. Add
constraint and solve uniquely
266
Absolute OrientationStitching
B. Horn, Closed-Form Solution of Absolute
Orientation using Unit Quarternions
267
Absolute OrientationStitching
268
Absolute OrientationStitching
One camera overlap
Projective 4 points, Nistér 01 Calibrated 1
point
269
(No Transcript)
270
Timing
271
Demonstration of real-time single camera tracking
272
Computing vehicle pose
Vehicle pose
273
Kalman filter pose with Visual Odometry
No VisOdo
VisOdo
274
(No Transcript)
275
(No Transcript)
276
(No Transcript)
277
Moving Stereo Pair
278
Moving Stereo Pair
279
Moving Stereo Pair
280
Visual Odometrywork with Oleg Naroditsky and Jim
Bergen
281
Visual Odometrywork with Oleg Naroditsky and Jim
Bergen

• 365 m without loss of tracking
• 350 m ( 3.5 minutes) without GPS
• Error in distance traveled 1
• Accumulated error in position 3-5
• e.g. 10m over 350m

North
East
282
Visual Odometrywork with Oleg Naroditsky and Jim
Bergen
283
(No Transcript)
284
Visual Odometrywork with Oleg Naroditsky and Jim
Bergen
285
(No Transcript)
286
Dense Reconstruction
287
Discontinuity Energy
Dissimilarity Energy
288
Median Fusion