Reconstructing 3D mesh from video image sequences

1
Reconstructing 3D mesh from video image sequences
specifications
Master thesis
by Martin Bujnák

• supervisor Mgr. Martin Samuelcik

MartinBujnak_at_post.sk
http//www.2B3D.host.sk
2
contents
Martin Bujnák Reconstructing 3D mesh from video
image sequences.

• aims
• requirements analysis
• software design
• algorithms selection (criteria)
• data description, I/O characteristics
  (assumptions)
• dependencies / independencies (HW/SW)
• function specification
• user model
• GUI

MartinBujnak_at_post.sk
http//www.2B3D.host.sk
3
Martin Bujnák Reconstructing 3D mesh from video
image sequences.
aims

• reconstruction of 3D mesh defined by continuous
  sequences of images, captured by standard (middle
  class) hand-held video camera
• reconstruction should be fully automatic or with
  very little user interactions
• output should by textured 3D model

MartinBujnak_at_post.sk
http//www.2B3D.host.sk
4
requirements analysis
Martin Bujnák Reconstructing 3D mesh from video
image sequences.

• find corresponding feature points in input images
• feature detectors (to slow)
• tracking (assumptions on input)
• enumerate relative transformation between spaces
  defined by neighboring input images
• camera trajectory reconstruction
• error correction (3D snapping in 2D)
• mesh and textures reconstruction

MartinBujnak_at_post.sk
http//www.2B3D.host.sk
5
image correspondence
Martin Bujnák Reconstructing 3D mesh from video
image sequences.
requirements analysis

• difficult process - big cpu/time consumption
• continuous image motion used as hint for
  correspondence problem
• brute-force comparison
• but nearest in motion direction first
• problems
• specular highlights (in feature tracking problem,
  in surface extraction helper)
• homomorphous filters (garage filter)
• bad features

MartinBujnak_at_post.sk
http//www.2B3D.host.sk
6
relative transformation
Martin Bujnák Reconstructing 3D mesh from video
image sequences.
requirements analysis

• assuming that at least 5 well detected
  corresponding feature points exist
• if not skip image
• epipolar geometry for computing projection
  matrices
• camera calibration
• enumerated from input
• 2 degree polynomial suffice to fix camera lens
  deformation
• focal point from image correspondence
• improving accuracy
• all camera positions lay on curve
• projection matrix basis vector are interpolated
  continuously

7
mesh and textures reconstruction
Martin Bujnák Reconstructing 3D mesh from video
image sequences.
requirements analysis

• direct volume reconstruction
• intersections in image space using normalized
  colors
• problems
• numerical stability

8
software design
Martin Bujnák Reconstructing 3D mesh from video
image sequences.

• pipe concept

previous results in fine structure
imagei-j
motion estimation
edges detector
homomorphous filter expected value trash holding
feature extraction
imagei
relative camera transformation
correspondence
error correction (cam. trajectory snapping)
surface textures extraction
filter
error correction
happy user )
9
algorithms selection
Martin Bujnák Reconstructing 3D mesh from video
image sequences.
software design

• main criteria is speed
• edges detected using differences
• edge reconstructed using curves (polynomial of
  low degree)
• feature point intersection of more curves (in
  some epsilon neighborhood)
• openCV (static) or own (dynamic)

10
algorithms selection
Martin Bujnák Reconstructing 3D mesh from video
image sequences.
software design

• not image-based (cross-correlation / invariant
  regions) correspondence technique
• slow
• inaccurate
• statistical
• image precision
• previous images not used
• many information ignored
• tracking technique combined with brute-force
  matching

11
algorithms selection
Martin Bujnák Reconstructing 3D mesh from video
image sequences.
software design

• surface reconstruction - similar to silhouette
  based techniques (on edged)
• clipping (sweep prune) extruded edge curves
• requires robust algorithm (booleans)
• still searching for it
• non-silhouette parts (inner points)
• intersect all extruded cones from every pixel
  (the same problems as meant above)
• idea restriction planes
• each image restricts move of its point leads to
  big equation
• point cloud or displacement map

12
data description, I/O characteristics ...
Martin Bujnák Reconstructing 3D mesh from video
image sequences.
software design

• input video file, of video stream
• handled by operating system and codecs
• internally every frame will be 24bit RGB 888
  bitmap
• float precision required in specular highlights
  removal algorithms (homomorphous operation -gt DCT
  -gt complex numbers)
• assuming continuous input (not time-step)

13
data description, I/O characteristics ...
Martin Bujnák Reconstructing 3D mesh from video
image sequences.
software design

• output mesh and textures
• mesh will be exported by supporting 3D
  environment (go3D) minor control (but wrml,
  3ds, scn)
• textures in some standardized image formats
  supported by go3D (bmp, tga, jpeg)
• internal data structures
• topological maps, 3D hashed grid,

14
dependencies / independencies (HW/SW)...
Martin Bujnák Reconstructing 3D mesh from video
image sequences.
software design

• will be implemented as plug-in into modeling 3D
  environment
• plug-in
• OS dependent
• interface to video devices (direct show)
• GUI using environment framework
• OS independent
• C
• open CV

15
function specification
Martin Bujnák Reconstructing 3D mesh from video
image sequences.
software design

• selecting input device
• process start
• exporting output
• subject of change

user model

• aim user not needed

GUI

• every function have its own button in some layout