Title: Simultaneous Point Matching and Recovery of Rigid and Nonrigid Shapes
1Simultaneous Point Matching andRecovery of Rigid
and Nonrigid Shapes
THESIS pROPOSAL
Thesis director Francesc Moreno
Noguer Tutor Alberto Sanfeliu Cortés
2Objective
Simultaneously solve the correspondence problem
and recover rigid and nonrigid shapes.
Given two point clouds extracted from different
views of the same object, the objective is to
simultaneously solve for point correspondence and
recover the mapping between the two rigid or
nonrigid model representations.
3Motivation
- Given um from a model point set and vt from a
target point set, find
model point set
target point set
4Motivation
- Given um from a model point set and vt from a
target point set, find
CORRESPONDENCES
um
vt
5Motivation
- Given um from a model point set and vt from a
target point set, find
CORRESPONDENCES
um
vt
TRANSFORM ESTIMATION
Hest
6Motivation
- Given um from a model point set and vt from a
target point set, find
CORRESPONDENCES
um
vt
TRANSFORM ESTIMATION
Hest
7Motivation
- Outliers
- Occlusions
- Clutteredbackground
- Observationnoise
- Repetitivepatterns
- Rigid / Nonrigidmodel
- 2D/3D or2D-2D / 3D-3D
um
vt
correspondences that do not fit the model
8Motivation
- Outliers
- Occlusions
- Clutteredbackground
- Observationnoise
- Repetitivepatterns
- Rigid / Nonrigidmodel
- 2D/3D or2D-2D / 3D-3D
um
vt
partial matching parts of the scene are occluded
9Motivation
- Outliers
- Occlusions
- Clutteredbackground
- Observationnoise
- Repetitivepatterns
- Rigid / Nonrigidmodel
- 2D/3D or2D-2D / 3D-3D
um
vt
points that do not belong to the model and
hinder the recognition
10Motivation
- Outliers
- Occlusions
- Clutteredbackground
- Observationnoise
- Repetitivepatterns
- Rigid / Nonrigidmodel
- 2D/3D or2D-2D / 3D-3D
um
vt
error in the position of points
11Motivation
- Outliers
- Occlusions
- Clutteredbackground
- Observationnoise
- Repetitivepatterns
- Rigid / Nonrigidmodel
- 2D/3D or2D-2D / 3D-3D
um
vt
regular structures are indistinguishable
algorithms fall into local minima
12Motivation
- Outliers
- Occlusions
- Clutteredbackground
- Observationnoise
- Repetitivepatterns
- Rigid / Nonrigidmodel
- 2D/3D or2D-2D / 3D-3D
um
vt
model shapes can undergo rigid or nonrigid
deformations
13Motivation
- Outliers
- Occlusions
- Clutteredbackground
- Observationnoise
- Repetitivepatterns
- Rigid / Nonrigidmodel
- 2D/3D or2D-2D / 3D-3D
um
vt
model shapes can undergo rigid or nonrigid
deformations
14Motivation
- Outliers
- Occlusions
- Clutteredbackground
- Observationnoise
- Repetitivepatterns
- Rigid / Nonrigidmodel
- 2D/3D or2D-2D / 3D-3D
um
vt
Coordinate System 1
Coordinate System 2
the transform can be embedded in 2D or 3D or
projective as in monocular view case (2D-3D)
15Motivation
- Outliers
- Occlusions
- Clutteredbackground
- Observationnoise
- Repetitivepatterns
- Rigid / Nonrigidmodel
- 2D/3D or2D-2D / 3D-3D
16Motivation
- Outliers
- Occlusions
- Clutteredbackground
- Observationnoise
- Repetitivepatterns
- Rigid / Nonrigidmodel
- 2D/3D or2D-2D / 3D-3D
Camera Coordinate System
World Coordinate System
the transform can be embedded in 2D or 3D or
projective as in monocular view case (2D-3D)
17Motivation
- Outliers
- Occlusions
- Clutteredbackground
- Observationnoise
- Repetitivepatterns
- Rigid / Nonrigidmodel
- 2D/3D or2D-2D / 3D-3D
Camera Coordinate System
World Coordinate System
xcam A R t xworld
18State-of-Art
2D-3D
2D-2D or 3D-3D
Rigid Model
Rigid 2D-2D / 3D-3D
Rigid 2D-3D
Nonrigid Model
Nonrigid 2D-3D
Nonrigid 2D-2D / 3D-3D
19State-of-Art
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
20State-of-Art
RIGID 2D-2D / 3D-3D MATCHING
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting with
Applications to Image Analysis and Automated
Cartography
RANSAC (global solution / high complexity)
21State-of-Art
RIGID 2D-2D / 3D-3D MATCHING
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting with
Applications to Image Analysis and Automated
Cartography
1992 Besl McKay, A Method for Registration of
3D Shapes
ICP Iterative Closest Point (requires good
initialization)
22State-of-Art
RIGID 2D-2D / 3D-3D MATCHING
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting with
Applications to Image Analysis and Automated
Cartography
1992 Besl McKay, A Method for Registration of
3D Shapes
2005 Chum Matas. Matching With PROSAC -
Progressive Sample Consensus
PROSAC RANSAC Appearance
23State-of-Art
RIGID 2D-2D / 3D-3D MATCHING
- Unsolved when
- Weak detection,
- outliers,
- occlusions,
- image noise,
- repetitive patterns,
- highly textured scenes,
- oblique angles
-
- PROSAC complexity similar to RANSAC (too high)
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting With
Applications to Image Analysis and Automated
Cartography
1992 Besl McKay, A Method for Registration of
3D Shapes
2005 Chum Matas. Matching With PROSAC -
Progressive Sample Consensus
24State-of-Art
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
25State-of-Art
RIGID 2D-3D MATCHING
1970
1980
1990
Known correspondences Perspective-n-Point
(PnP) (old problem)
2000
2010
2009 Moreno-Noguer et al, EPnP An Accurate
O(n) Solution to the PnP Problem
26State-of-Art
RIGID 2D-3D MATCHING
1970
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting With
Applications to Image Analysis and Automated
Cartography
1980
RANSAC Random Sampled Consensus DLT Direct
Linear Transform (high complexity)
1990
2000
2010
2009 Moreno-Noguer et al, EPnP An Accurate
O(n) Solution to the PnP Problem
27State-of-Art
RIGID 2D-3D MATCHING
1970
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting With
Applications to Image Analysis and Automated
Cartography
1980
1990
SoftPOSIT Unknown correspondences
2002 David et al,SoftPOSIT Simultaneous Pose
and Correspondence Determination
2000
2010
2009 Moreno-Noguer et al, EPnP An Accurate
O(n) Solution to the PnP Problem
28State-of-Art
RIGID 2D-3D MATCHING
1970
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting With
Applications to Image Analysis and Automated
Cartography
1980
Blind PnP Unknown correspondences Pose Priors
(geometrical consistency) Kalman Filter to
propagate pose uncertainty
1990
2002 David et al,SoftPOSIT Simultaneous Pose
and Correspondence Determination
2000
2008 Moreno-Noguer et al, Pose Priors for
Simultaneously Solving Alignment and
Correspondence.
2010
2009 Moreno-Noguer et al, EPnP An Accurate
O(n) Solution to the PnP Problem
29State-of-Art
RIGID 2D-3D MATCHING
1970
- Modeling the uncertainty
- Kalman Filter linearizes the uncertainty model
- Work with Bayesian non-parametric models
(Gaussian Processes)
1981 Fischler Bolles. Random Sample
Consensus A Paradigm for Model Fitting With
Applications to Image Analysis and Automated
Cartography
1980
1990
2002 David et al,SoftPOSIT Simultaneous Pose
and Correspondence Determination
2000
2008 Moreno-Noguer et al, Pose Priors for
Simultaneously Solving Alignment and
Correspondence.
2010
2009 Moreno-Noguer et al, EPnP An Accurate
O(n) Solution to the PnP Problem
30State-of-Art
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
31State-of-Art
NONRIGID 2D-2D / 3D-3D MATCHING
Soft-assign (requires good initialization)
1970
1980
1998 Gold et al.. New Algorithms for 2D and 3D
Point Matching Point Estimation and
Correspondence
1990
2003 Chui Rangarajan. A New Point Matching
Algorithm for Non-Rigid Registration
2000
Soft-assign Thin-plate splines (requires good
initialization) (smooth deformations)
2010
32State-of-Art
NONRIGID 2D-2D / 3D-3D MATCHING
1970
1980
1998 Gold et al.. New Algorithms for 2D and 3D
Point Matching Point Estimation and
Correspondence
1990
2003 Hannel et al.. An Extension of the ICP
Algorithm for Modeling Nonrigid Objects with
Mobile Robots
2003 Chui Rangarajan. A New Point Matching
Algorithm for Non-Rigid Registration
2000
2008 Li et al, Global Correspondence
Optimization for Non-Rigid Registration of
Depth Scans
2010
Nonrigid ICP Variants (Require a good
initialization)
33State-of-Art
NONRIGID 2D-2D / 3D-3D MATCHING
Graph Matching Thin-plate splines (Smooth
deformations)
1970
Shape appearance Thin-plate splines (Smooth
deformations)
1980
1998 Gold et al.. New Algorithms for 2D and 3D
Point Matching Point Estimation and
Correspondence
2002 Belongui, Shape matching and Object
Recognition Using Shape Contexts
1990
2003 Hannel et al.. An Extension of the ICP
Algorithm for Modeling Nonrigid Objects with
Mobile Robots
2003 Chui Rangarajan. A New Point Matching
Algorithm for Non-Rigid Registration
2000
2008 Li et al, Global Correspondence
Optimization for Non-Rigid Registration of
Depth Scans
2010
2010 Deng et al, Retinal Fundus Image
Registration via Vascular Structure Graph Matching
34State-of-Art
NONRIGID 2D-2D / 3D-3D MATCHING
1970
- Unsolved problems
- Harsh deformations (Gaussian Processes vs TPS)
- Avoid local minima (Global Search vs ICP et al.)
1980
1998 Gold et al.. New Algorithms for 2D and 3D
Point Matching Point Estimation and
Correspondence
2002 Belongui, Shape matching and Object
Recognition Using Shape Contexts
1990
2003 Hannel et al.. An Extension of the ICP
Algorithm for Modeling Nonrigid Objects with
Mobile Robots
2003 Chui Rangarajan. A New Point Matching
Algorithm for Non-Rigid Registration
2000
2008 Li et al, Global Correspondence
Optimization for Non-Rigid Registration of
Depth Scans
2010 Myronenko Song, Point-Set Registration
Coherent Point Drift
2010
2010 Deng et al, Retinal Fundus Image
Registration via Vascular Structure Graph Matching
35State-of-Art
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
36State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
1980
1980
1990
1990
2000
2000
2010
2010
37State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
2003 Shakhnarovich et al., Fast Pose Estimation
with Parameter Sensitive Hashing
2006 Sigal Black, Humaneva Synchronized
Video and Motion Capture Dataset for Evaluation
of Articulated Human Motion
1980
1980
1990
1990
Discriminative methods Database learning
Nearest Neigbour Selection
2000
2000
2010
2010
38State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
2003 Shakhnarovich et al., Fast Pose Estimation
with Parameter Sensitive Hashing
2006 Sigal Black, Humaneva Synchronized
Video and Motion Capture Dataset for Evaluation
of Articulated Human Motion
1980
1980
2007 Salzmann et al., Surface
Deformation Models for Non-Rigid 3D Shape Recovery
1990
1990
2000
2000
2010 Sanchez et al., Simultaneous Pose,
Correspondence and Non-Rigid Shape
Generative methods PCA model of the surface
2010
2010
39State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
2003 Shakhnarovich et al., Fast Pose Estimation
with Parameter Sensitive Hashing
2006 Sigal Black, Humaneva Synchronized
Video and Motion Capture Dataset for Evaluation
of Articulated Human Motion
1980
1980
2007 Salzmann et al., Surface
Deformation Models for Non-Rigid 3D Shape Recovery
1990
1990
Thin-plate Splines (Medical Imaging!)
2000
2000
2010 Sanchez et al., Simultaneous Pose,
Correspondence and Non-Rigid Shape
2010
2010
2009 Groher, Deformable 2D-3D Registration of
Vascular Structures in a One View Scenario
40State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
2003 Shakhnarovich et al., Fast Pose Estimation
with Parameter Sensitive Hashing
2006 Sigal Black, Humaneva Synchronized
Video and Motion Capture Dataset for Evaluation
of Articulated Human Motion
1980
1980
2007 Salzmann et al., Surface
Deformation Models for Non-Rigid 3D Shape Recovery
1990
1990
Combining Discriminative Generative methods
2000
2000
2010 Sanchez et al., Simultaneous Pose,
Correspondence and Non-Rigid Shape
2010 Salzmann Urtasun, Combining
Discriminative and Generative Methods for 3D
Deformable Surface and Articulated Pose
Reconstruction
2010
2010
2009 Groher, Deformable 2D-3D Registration of
Vascular Structures in a One View Scenario
41State-of-Art
NONRIGID 2D-3D MATCHING
Deformable surfaces
Articulated structures
1970
1970
2003 Shakhnarovich et al., Fast Pose Estimation
with Parameter Sensitive Hashing
2006 Sigal Black, Humaneva Synchronized
Video and Motion Capture Dataset for Evaluation
of Articulated Human Motion
- Potential improvements
- Better parameterization for articulated
structures - Harsh deformations (Gaussian Processes vs TPS)
1980
1980
2007 Salzmann et al., Surface
Deformation Models for Non-Rigid 3D Shape Recovery
1990
1990
2000
2000
2010 Sanchez et al., Simultaneous Pose,
Correspondence and Non-Rigid Shape
2010 Salzmann Urtasun, Combining
Discriminative and Generative Methods for 3D
Deformable Surface and Articulated Pose
Reconstruction
2010
2010
2009 Groher, Deformable 2D-3D Registration of
Vascular Structures in a One View Scenario
42Contributions
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
43Contributions
Homography (2D-to-2D) estimation
44Contributions
- Rigid registration
- Feature Point Detectors assigns multiple
correspondences - PROSAC picks just the one with better similarity
score !!!
Homography (2D-to-2D) estimation
45Contributions
- Rigid registration
- Feature Point Detectors assigns multiple
correspondences - Using Kalman Filter we can propagate geometry
priors,thus constraining the search regions for
each feature point
Homography (2D-to-2D) estimation
46Contributions
- Rigid registration
- Feature Point Detectors assigns multiple
correspondences - Using Kalman Filter we can propagate geometry
priors,thus constraining the search regions for
each feature point - Iterative approach
Homography (2D-to-2D) estimation
47Contributions
- Rigid registration
- Feature Point Detectors assigns multiple
correspondences - Using Kalman Filter we can propagate geometry
priors,thus constraining the search regions for
each feature point - Iterative approach
Homography (2D-to-2D) estimation
48Contributions
- Rigid registration
- Feature Point Detectors assigns multiple
correspondences - Using Kalman Filter we can propagate geometry
priors,thus constraining the search regions for
each feature point - Iterative approach
Homography (2D-to-2D) estimation
49Contributions
- Rigid registration
- Feature Point Detectors assigns multiple
correspondences - Using Kalman Filter we can propagate geometry
priors,thus constraining the search regions for
each feature point - Iterative approach backtracking when necessary
Homography (2D-to-2D) estimation
50Contributions
- Rigid registration
- Some results
PROSAC
Blind Homography
51Contributions
- Rigid registration
- E. Serradell, M. Ozuysal, V. Lepetit, P. Fua and
F. Moreno-Noguer Combining Geometric and
Appearance Priors for Robust Homography
Estimation . In ECCV 2010
Homography (2D-to-2D) estimation
52Contributions
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
53Contributions
- Nonrigid registration
- Nonrigid 2D-to-3D registration
- Project in collaboration with
- New parameterization for articulated models
2D X-ray Image
CT 3D Volume
54Contributions
- Nonrigid registration
- Recursive parameterization of the nodes of the
articulated structure
Camera (known)
CT 3D Volume
2D X-ray Image
55Contributions
- Nonrigid registration
- Recursive parameterization of the nodes of the
articulated structure - Generative model Probabilistic PCA
Camera (known)
synthetic samples
2D features
56Contributions
- Nonrigid registration
- Recursive parameterization of the nodes of the
articulated structure - Generative model Probabilistic PCA
Camera (known)
generative model
2D features
57Contributions
- Nonrigid registration
- Recursive parameterization of the nodes of the
articulated structure - Generative model Probabilistic PCA
- Iterative update
- 1.- Kalman Filter model projection
- 2.- Assign correspondences
Camera (known)
generative model
2D features
58Contributions
- Nonrigid registration
- E. Serradell, A. Romero, R. Leta, C. Gatta and F.
Moreno-Noguer Simultaneous Correspondence and
Non-Rigid 3D Reconstruction of the Coronary Tree
from Single X-ray Images. In ICCV 2011
shape prior
recoveredmodel
2D X-ray Image
CT 3D Volume
59Contributions
2D-3D
2D-2D or 3D-3D
Rigid Model
Nonrigid Model
60Contributions
- Nonrigid registration
- Nonrigid 2D-to-2D or 3D-to-3D registration
Partial matching
10-6 m
10-7 m
Optical Microscope Image Stack
Electron Microscope Image Stack
61Contributions
- Nonrigid registration
- Nonrigid 2D-to-2D or 3D-to-3D registration
- Extract neuronal tree ? Graph Matching
Optical Microscope Image Stack
Electron Microscope Image Stack
62Contributions
- Nonrigid registration
- Nonrigid 2D-to-2D or 3D-to-3D registration
- Extract neuronal tree ? Graph Matching
- Two step process
- 1.- Affine transform (Kalman Filter approach)
y A x b
Optical Microscope Image Stack
Electron Microscope Image Stack
63Contributions
- Nonrigid registration
- Nonrigid 2D-to-2D or 3D-to-3D registration
- Extract neuronal tree ? Graph Matching
- Two step process
- 1.- Affine transform (Kalman Filter approach)
- 2.- Nonrigid transform (Gaussian Processes for
regression)
y A x b f(x)
Optical Microscope Image Stack
Electron Microscope Image Stack
64Contributions
- Nonrigid registration
- Some results
original graphs
affinetransform
nonlinear refining
65Contributions
- Nonrigid registration
- Some results
original graphs
affinetransform
nonlinear refining
affinetransform
nonlinear refining
66Contributions
- Nonrigid registration
- E. Serradell, J. Kybic, F. Moreno-Noguer and P.
Fua Robust Elastic 2D/3D Geometric Graph
Matching, submitted to SPIE Medical Imaging
Optical Microscope Image Stack
Electron Microscope Image Stack
67Contributions
2D-3D
2D-2D or 3D-3D
Rigid Model
Rigid 2D-2D / 3D-3D
Rigid 2D-3D
Nonrigid Model
Nonrigid 2D-3D
Nonrigid 2D-2D / 3D-3D
68Contributions
- Global solution to point registration
- Valid for 2D-2D,3D-3D,2D-3D / rigid and nonrigid
models - Using Gaussian Processes some preliminary
results
initial shape
recovered shape
69Task Planning
- Master ARV
- Simultaneous Correspondence Robust Estimation
- Nonrigid Model Reconstruction from Single Images
- Elastic Graph Matching
- Write Thesis
70Task Planning
Done
On-going
To Do
- Master ARV
- Simultaneous Correspondence Robust Estimation
- Nonrigid Model Reconstruction from Single Images
- Elastic Graph Matching
- Write Thesis
71Task Planning
Done
On-going
To Do
- Master ARV
- Simultaneous Correspondence Robust Estimation
- Nonrigid Model Reconstruction from Single Images
- Elastic Graph Matching
- Write Thesis
Research Stay at CVLAB (EPFL)
72Task Planning
Done
On-going
To Do
- Master ARV
- Simultaneous Correspondence Robust Estimation
- Nonrigid Model Reconstruction from Single Images
- Elastic Graph Matching
- Write Thesis
Project in collaboration with CVC (UAB), MAiA
(UB) and Hospital Sant Pau
73Task Planning
Done
On-going
To Do
- Master ARV
- Simultaneous Correspondence Robust Estimation
- Nonrigid Model Reconstruction from Single Images
- Elastic Graph Matching
- Write Thesis
Research Stay at CVLAB (EPFL)
74Task Planning
Done
On-going
To Do
- Master ARV
- Simultaneous Correspondence Robust Estimation
- Nonrigid Model Reconstruction from Single Images
- Elastic Graph Matching
- Write Thesis
Research Stay at CVLAB (EPFL)
75Task Planning
Done
On-going
To Do
- Master ARV
- Simultaneous Correspondence Robust Estimation
- Nonrigid Model Reconstruction from Single Images
- Elastic Graph Matching
- Write Thesis
76Summary of achievements
- Published papers
- E. Serradell, M. Ozuysal, V. Lepetit, P. Fua and
F. Moreno-Noguer Combining Geometric and
Appearance Priors for Robust Homography
Estimation . In ECCV 2010 - E. Serradell, A. Romero, R. Leta, C. Gatta and F.
Moreno-Noguer Simultaneous Correspondence and
Non-Rigid 3D Reconstruction of the Coronary Tree
from Single X-ray Images. In ICCV 2011 - ICCV, ECCV acceptance rate lt 30
- Submitted papers
- E. Serradell, J. Kybic, F. Moreno-Noguer and P.
Fua Robust Elastic 2D/3D Geometric Graph
Matching, submitted to SPIE Medical Imaging
77THANKS!