High Resolution Surface Reconstruction from Overlapping MultipleViews

1
High Resolution Surface Reconstruction from
Overlapping Multiple-Views
Nader Salman nsalman_at_sophia.inria.fr
Mariette Yvinec myvinec_at_sophia.inria.fr
Geometrica Team-Project
JGA January 29th, 2009
2
Motivation

• Urban scenes are hard to model
• Complex geometry and topology
• Severe occlusions
• ? We target automatic surface reconstruction
• from stereo vision data

http//cvlab.epfl.ch/strecha/multiview/denseMVS.h
tml
3
Related Work

• Reconstructing 3D models of objects and scenes
  from N-views
• Image-based methods
• Image mosaics Szeliski et al. 97
• Lightfield Levoy et al. 96 and Lumigraph
  Cohen et al. 96
• Model-based methods
• Active sensing approach
• Turk et al. 94, Laurendeau et al. 95, Levoy
  et al. 96
• Passive sensing approach
• Silhouette based Niem et al. 97
• Structure from motion
• Simple planar scenes Zisserman et al. 96, 98
• Semi-automatic reconstruction Debevec et al. 96
• Automatic reconstruction
• Boissonnat et al. 88, Taylor et al. 03,
  Hilton 05,
• Pons et al. 07

4
System Overview
Image Capture
2D Constrained Triangulation
3D Triangle Soup
Triangle soupFiltering
Structure from Motion
Input Preprocessing
Reconstruction
5
System Overview
Image Capture
2D Constrained Triangulation
3D Triangle Soup
Triangle soupFiltering
Structure from Motion
Input Preprocessing
Reconstruction
6
Image Capture and 3D recovery (1)
Video sequences
captured images (100-3000 images)
10k points
Matchmoving (camera calibration point cloud
generation) MatchMover,Voodoo/VooCAT, or Boujou
7
Image Capture and 3D recovery (2)
High resolution localized still pictures
Features extraction ENPC/Certis CSTB
captured images (10-50 images)
300k points
Frames of tracked video MatchMover, Voodoo/VooCAT
, or Boujou
Hardware Gyroviz
8
Image Capture and 3D recovery (2)
High resolution localized still pictures
Features extraction ENPC/Certis CSTB
captured images (10-50 images)
300k points
Frames of tracked video MatchMover, Voodoo/VooCAT
, or Boujou
Hardware Gyroviz
9
Image Capture and 3D recovery (2)
Matchmoving Voodoo

• Output is data dependent
• - Movement while shooting video
• - Lighting conditions
• Sparse point clouds
• Outliers

10k points
Features extraction ENPC/Certis CSTB

• Dense point clouds (10-20x more)
• Requires calibrated images
• Many outliers
• Slow

10
System Overview
Image Capture
2D Constrained Triangulation
3D Triangle Soup
Triangle soupFiltering
Structure from Motion
Input Preprocessing
Reconstruction
11
Input Preprocessing (1)

• Outliers removal wrt average K-nearest neighbors
  squared distance

Detected outliers
Detected outliers
12
Input Preprocessing (2)

• Outliers removal wrt point error (number of
  cameras, cameras cone angle)

Detected outliers
Detected outliers
13
Input Preprocessing (3)

• 3D Points merging

• Point cloud smoothing (jet-fitting
  reprojection) Pouget-Cazals07

14
System Overview
Image Capture
2D Constrained Triangulation
3D Triangle Soup
Triangle soupFiltering
Structure from Motion
Input Preprocessing
Reconstruction
15
Reconstruction Algorithm

• Extract 3D triangles that lie on the surface of
  the scene.
• Extract a surface out of these triangles.

1
2
16
Triangle soup extraction (1)

• Idea
• Extract 3D triangles whose corresponding 2D
  does not intersect the image contour

N times N images
Step 3
2D constrained Tessellation Backprojecte
d into 3D space
Step 4
Filter triangle soup
Step 1
Constraints retrieval
Step 2
Contour extraction
Triangle soup
17
Triangle soup extraction (2)

• Idea Extract 3D triangles whose corresponding 2D
  does not intersect the image contour
• Step 1 Contour extraction
• Grayscale
• Effective luminance of a pixel
• Filter
• Gaussian smoothing mask
• Extract contours
• Frei-chen convolution masks

Survey www.2d3.com
18
Triangle soup extraction (3)

• Idea Extract 3D triangles whose corresponding
• 2D does not intersect the image contour
• Step 2 Constraints retrieval
• Build complete graph of feature points
• keep edges that respect the contour
• The edge weight depends on number of white pixels
• Adapt Needleman-Wunsch NW1970
• Filter collinear and intersecting edges
• Give higher priority to short edges

19
Triangle soup extraction (4)

• Idea Extract 3D triangles whose corresponding
• 2D does not intersect the image contour
• Step 3 2D constrained tessellation
• and backprojection
• Add constraints in 2D Delaunay tessellation
• Backproject into 3D space using corresponding 3D
  points

Triangle Soup
N times N images
20
System Overview
Image Capture
2D Constrained Triangulation
3D Triangle Soup
Triangle soupFiltering
Structure from Motion
Input Preprocessing
Reconstruction
21
Triangle soup filtering (1)

• Problem
• Triangle soup contains lots of incorrect
  triangles
• Filter triangles using size heuristic
• Compute bounding box of triangle soup
• Each triangle longest edge is compared to the
  half diagonal of the bounding box
• Filter triangles using visibility constraint
  Taylor03
• Empty ray in space between the camera location
  and 3D point position.

22
Triangle soup filtering (1)
Before size heuristic
After size heuristic
23
Triangle soup filtering (2)

• Problem
• Triangle soup contains lots of incorrect
  triangles
• Filter triangles using size heuristic
• Compute bounding box of triangle soup
• Each triangle longest edge is compared to the
  half diagonal of the bounding box
• Filter triangles using visibility constraint
  Taylor03
• Empty ray in space between the camera location
  and 3D point position.

24
Triangle soup filtering (2)
Before visibility constraint
After visibility constraint
25
System Overview
Image Capture
2D Constrained Triangulation
3D Triangle Soup
Triangle soupFiltering
Structure from Motion
Input Preprocessing
Reconstruction
26
Reconstruction (1)

• How do we extract a surface from this polygon
  soup?
• Restricted Delaunay
• Compute 3D Delaunay triangulation of triangle
  soup vertices
• Compute restricted Delaunay to the union of
  triangles

27
Restricted Delaunay reconstruction
28
Reconstruction (2)

• How do we extract a surface from this polygon
  soup?
• Restricted Delaunay
• ? Triangles respect empty circle condition
• Non watertight surface (holes)
• Extensions
• Remesh using restricted Delaunay
• OR
• Scatter a large number of points continuously
  across the surface of each triangle
• Create an implicit surface that approximates the
  triangle soup

29
Some results
LA_Aerial www.realviz.com
8k points
30
Some results
Lion www.realviz.com
20k points
31
Results
Furokawa et al. 2007
Strecha et al. 2006
Pons et al. 2009
Ours
32
Future work

• More data acquisition and testing for automatic
  parameter
• Improve and complete existing algorithms
• Triangles soup algorithm
• Step 1 Contour extraction
• Segmentation techniques Lévy06
• Histogram for automatic threshold Asano08
• Step 4 Triangle soup filtering
• Some big triangles are correct
• Use circumscribing circle to differentiate
  between detected triangles
• Project triangles on respective images
• Use cross correlation to detect incorrect
  triangles
• Sharp features detection and reconstruction

33
Thank you
Nader Salman http//www.gyroviz.org