Enforcing Constraints for Human Body Tracking

1
Enforcing Constraints for Human Body Tracking

• David Demirdjian
• Artificial Intelligence Laboratory, MIT

2
Goal

• Real-time articulated body tracking from stereo
  accounting for constraints on pose

3
Approach

• Differential tracking assuming the articulated
  body pose Pt-1 is known, estimate the pose Pt (or
  equivalently the set of limb rigid motions
  xi(ti,wi) between poses Pt-1 and Pt) that
  minimizes the distance between the articulated
  model and the observed 3D data
• ? tracking as a constrained optimization problem

4
Approach

• Differential tracking assuming the articulated
  body pose Pt-1 is known, estimate the pose Pt (or
  equivalently the set of limb rigid motions
  xi(ti,wi) between poses Pt-1 and Pt) that
  minimizes the distance between the articulated
  model and the observed 3D data
• ? tracking as a constrained optimization problem
• Solve unconstrained optimization problem
• Project solution on constraint surface

5
Projection-based approach
D (unconstrained optimum)
D
human motion manifold
6
Approach

• Estimate limb motions xi(ti,wi) independently
  using standard multi-object tracking algorithm
• Projection find the closest body motion D(xi)
  to D(xi) that satisfies human body constraints
• articulated constraints
• other constraints joint limit,

7
Previous work

• Particle sampling
• Sidenbladh al. ECCV00
• Sminchisescu Triggs CVPR01
• Differential tracking
• Plankers Fua ICCV99
• Jojic al. ICCV99
• Delamarre Faugeras ICCV99

8
Plan

• Unconstrained problem
• Articulated constraints enforcing
• Other constraints
• Tracking results
• Application (Multimodal interface)
• Conclusion

9
Multi-object tracking

• Assuming the articulated body pose Pt-1 is known,
  estimate the set of limb rigid motions xi(ti,wi)
  minimizes the distance between the (moved) limb
  and the observed 3D data
• Consists in estimating limb motions xi(ti,wi)
  independently
• Estimate visible 3D mesh of each limb
• Current implementation uses the ICP algorithm to
  register each limb w.r.t 3D data

10
Iterative Closest Point

• 3D registration
• find the rigid transformation x that maps shape
  St (limb model) to shape Sr (3D data)

x
St
Sr
11
Iterative Closest Point

• Matching points
• For all points in St, we search for the closest
  point in Sr by computing the distance and keep
  the closest

St
Sr
12
Iterative Closest Point

• Energy function minimization
• Estimate the rigid transformation that minimizes
  the sum of squared distances between
  corresponding matched points

St
Sr
13
Iterative Closest Point

• Energy function minimization
• Estimate the rigid transformation that minimizes
  the sum of squared distances between
  corresponding matched points

St
Sr
14
Iterative Closest Point

• Optimal rigid transformation x (and uncertainty
  Lx) found by combining all the elementary
  displacements

15
Plan

• Unconstrained problem
• Articulated constraints enforcing
• Other constraints
• Tracking results
• Application (Multimodal interface)
• Conclusion

16
Projection

• The unconstrained optimal body motion is given by
  
• D(x1, x2 xN)
• With uncertainty
• L(L1, L2 LN)

Articulated constraints enforcement find D
that minimizes the Mahalanobis distance
17
Articulated motion estimation
If Mij is a joint between objects i and j
Motion of point Mij on limb j
Motion of point Mij on limb i

(Ri,ti)
Mij joint
 
obj. i
(Rj,tj)
obj. j
18
Articulated motion estimation
If Mij is a joint between objects i and j
Motion of point Mij on limb j
Motion of point Mij on limb i

(Ri,ti)
Mij joint
 
obj. i
(Rj,tj)
obj. j
19
Articulated motion estimation
If Mij is a joint between objects i and j
Motion of point Mij on limb j
Motion of point Mij on limb i

(Ri,ti)
Mij joint
 
obj. i
(Rj,tj)
obj. j
.x denotes skew-symmetric matrix
20
Articulated motion estimation
If Mij is a joint between objects i and j
Motion of point Mij on limb j
Motion of point Mij on limb i

(Ri,ti)
Mij joint
 
obj. i
(Rj,tj)
obj. j
.x denotes skew-symmetric matrix
21
Articulated motion estimation
If Mij is a joint between objects i and j
Motion of point Mij on limb j
Motion of point Mij on limb i

(Ri,ti)
Mij joint
 
obj. i
(Rj,tj)
obj. j
22
Articulated motion estimation
If Mij is a joint between objects i and j
Motion of point Mij on limb j
Motion of point Mij on limb i

(Ri,ti)
Mij joint
 
obj. i
(Rj,tj)
obj. j
(Stack for all joints)
23
Articulated motion estimation
All the joint constraints can be written as a
linear constraint

• is a linear combination of vectors in the
• nullspace of F. Therefore there exists a matrix V
  such that

V can be estimated by SVD of F
24
Articulated motion estimation
unconstrained motion
articulated motion
 
Find minimum of E2 in nullspace of

(linear projection)
25
Plan

• Unconstrained problem
• Articulated constraints enforcing
• Other constraints
• Tracking results
• Application (Multimodal interface)
• Conclusion

26
Other constraints

• Constraints
• Static Joint angle bounds, gravity law,
• Dynamic Maximum velocity,
• Motivation
• Using more constraints to reduce local minima and
  therefore increase tracking robustness
• Application context can reduce tremendously the
  dimension of the pose space

27
Other constraints
Pose constraints modeled by a (user-defined)
function f, such that valid poses correspond to
f(P)gt0 ex f(P)min(g1(P), g2(P), gN(P))
with g1(P) angle(l-arm, l-forearm)
min_angle g2(P) max_angle - angle(l-arm,
l-forearm) . Constraints enforcement find
D that minimizes the Mahalanobis distance
with D satisfying FPt-1(D)f(D (Pt-1))gt0
28
Other constraints
articulated constrained motion
articulated motion
(local parameterization)
with
29
Constrained optimization algorithm
Alternate between binary and stochastic searches
d
30
Constrained optimization algorithm
Alternate between binary and stochastic searches
31
Constrained optimization algorithm
Alternate between binary and stochastic searches
E2 E0
32
Constrained optimization algorithm
Alternate between binary and stochastic searches
E2 E1


33
Constrained optimization algorithm
Alternate between binary and stochastic searches
d


34
TRACKING SEQUENCE
35
Future work

• Quantitative measurement
• (comparing results with tethered motion capture
  system)
• Appearance/Shape information
• (learning color distribution shape of limbs)
• Motion/gesture
• (including dynamic constraints)
• Learning human motion constraints
• (instead of giving them explicitly.. ICCV03)

36
Applications

• Multimodal Human-Computer Interaction (gesture
  speech)

37
(No Transcript)
38
Conclusion

• Projection-based approach for articulated body
  tracking
• articulated constraints enforced by (linearly)
  projecting unconstrained limb motion on
  articulated motion manifold
• other constraints enforced using a stochastic
  constrained optimization algorithm