Model Refinement from Planar Parallax

1
Model Refinement from Planar Parallax

• Anthony Dick Roberto Cipolla
• Department of Engineering
• University of Cambridge

2
Model acquisition
Recovery of 3D scene structure from one or more
images Source Photobuilder The devils in the
detail
3
Model refinement

• We concentrate on approximately planar models of
  architectural scenes

4
Planar parallax

• Two projections of a planar scene are related by
  a 2D homography H
• Points not belonging to the plane do not obey
  this homography
• displaced by an additional parallax vector along
  epipolar line

Scene
Image 2
Image 1
5
Model refinement algorithm
Offset Images
Homography definition
Camera Calibration
Reconstruction
Correspondence
NO
OK?
Initial model Reference Image
YES
Prediction
Plane Fitting
Refined model
6
Homography Definition
Initial model Reference Image
Homography definition
Offset Images
Camera Calibration
4 correspondences required to estimate H (8
DOF) Use linear technique
7
Camera calibration
Homography definition
Camera calibration
Perspective camera model
Correspondence
where
is the same for all images
x
Reference camera coordinate system
z
x
z
World coordinate system
Optical axis
y
y
Model plane, n 0 0 1
Ref. image plane
Given n offset images, solve for
1 5n parameters
Given homographies
8n parameters
8
Correspondence

• Warp offset image
• Apply wavelet transform to each image
• Perform multi-resolution matching in wavelet
  domain
• sum squared difference of 6 filter outputs
• taking into account the epipolar constraint

Reconstruction
Camera Calibration
Correspondence
Prediction
9
Reconstruction
N
Reconstruction
Correspondence

• Triangulate to obtain dense depth map
• Robust fusion of several depth maps if necessary
• Use median at each point
• Special case - near depth discontinuity

OK?
Y
Plane Fitting
Choose camera 1
Choose camera 2
Model plane
Camera 1
Camera 2
10
Example - Wreath
Reference Image
Offset Image 1
Offset Image 2
Image 1 only
Image 2 only
Fused reconstruction
11
Example - Wreath
12
Example - Gateway
Image 1 only
Images 1 and 2
Images 1, 2 and 3
Images 1, 2, 3 and 4
13
Example - Gateway
14
Plane fitting prediction
Reconstruction
Correspondence
Outlier
OK?
Seed point
Prediction
Plane Fitting

• Grow planar regions
• RANSAC fit planes at each iteration
• Intensity edge fitting to refine region
  boundaries
• Use planes (refined model) to predict parallax at
  next offset image
• reduces correspondence search window in planar
  regions
• More accurate, requires fewer images

Refined model
15
Example - Church wall
Reference Image
Image 1
Image 2
Using image 1
Using images 1 and 2
16
Example - Church wall
17
Conclusion

• Model refinement avoids some problems of other
  methods
• little reliance on user
• low number of initial parameters
• Future directions
• simplify correspondence problem by modelling
  surface reflectance
• stitch together multiple façades
• model selection