Simultaneous Pose Estimation and Camera Calibration from Multiple Views

1
Simultaneous Pose Estimation and Camera
Calibration from Multiple Views

• Tomá Io
• W. Eric L. Grimson
• June 27, 2004

2
Motivation

• Easy problem for humans
• Take image of walking human
• Shape a 3D stick figure to correspond to the pose
  in image
• Applications
• Intelligent rooms
• Gait analysis

3
Problem

• Input
• Video sequences of walking person from multiple
  cameras
• Cameras are synchronized and have overlapping
  fields
• Cameras not calibrated
• Output
• Estimated camera pose (positions) with respect to
  direction of motion
• Estimated 3D body pose in each frame

4
Approach

• Model human walking motion
• Model appearance by clustering examples
• Model dynamics using an HMM
• Fit model to input
• Obtain silhouettes using background subtraction
• Find best matching appearance cluster for each
  silhouette
• Estimate camera position from best matching
  clusters
• Estimate pose using HMM

5
System Overview
6
Motion Appearance Model

• Camera Pose Space (L)
• Space of allowed camera positions around person
• Set of pairs (?,f)
• Body Pose Space (B)
• Walking cycle discretized into 20 phases
• Sample from space BxL
• 17,280 examples
• Examples consist of silhouette 2D joint
  locations
• Cluster examples using EM
• Cluster on 2D joint locations

7
Clustering Examples Using EM

• Assume gaussian cluster densities
• EM update equations

8
Clustering algorithm

• Select number of clusters N
• Initialize ? using k-means
• Randomly pick N examples to serve as initial
  cluster means µi
• Assign example xi to cluster j whenever
• Recompute cluster means and iterate until
  stopping criterion met
• Compute priors and covariance matrices
• Iterate using EM update equations until stopping
  criterion is met

9
Number of Clusters

• Scatter criteria
• Within-scatter
• Between-scatter
• Evaluating within-scatter

10
Clustering Results
Single cluster (EM with N100)
Cluster means in the camera pose space
11
Comparing binary silhouettes and continuous
cluster means

• Take cluster mean sc and binary silhouette sb
• Register silhouettes
• Fit ellipse to each and align major axes
• Isotropic scaling to achieve equal height
• Compute distance

12
Hidden Markov Models

• Markov Model
• Set of states with associated priors
• Transition probabilities
• Hidden Markov Model
• Added set of visible symbols
• States no longer directly observable hidden
  states
• Hidden states generate visible symbols with some
  probability density over the set of visible
  symbols

13
HMM for walking motion

• Input
• walking phase estimate per frame and camera
• ? observations
• Output
• sequence of phase estimates
• ? hidden states
• Structure
• Cyclical, feed-forward
• Problems
• Input is noisy
• Opposite phases have similar appearance

14
System Overview
15
Using the Appearance Model for Camera Calibration

• Use motion segmentation to obtain silhouettes
• Compare silhouettes with cluster means to find
  best matching clusters
• Estimate camera pose by robustly estimating the
  mean of the cloud formed by camera positions of
  examples in matched clusters over entire sequence

16
Body Pose Estimation with HMMs
View 1
View 2
Silhouettes
3D model rendered from estimated camera position
in view 2 using the computed sequence of body
pose estimates
Matched Clusters
HMM
Phase Estimates
17
Results
18
Results
19
Accuracy of CameraPose Estimation
20
Using the model to improve silhouettes

• Take input silhouettes
• Generate rendering of model from estimated
  viewpoint and given estimated walking phases
• Adjust priors on pixels being foreground or
  background using rendered model silhouettes

21
Conclusions

• Strengths
• No manual initialization
• Uncalibrated cameras
• Simple appearance and dynamics models
• Robust with respect to silhouette noise
• Weaknesses
• Estimated body pose is model-specific
• No explicit biometric information
• Use
• Good estimate of actual body pose, subject to
  further refinement
• Appearance model can serve as reference point for
  comparing input sequences from different views