Video Motion Capture

1
Video Motion Capture

• Christoph Bregler
• Jitendra Malik
• UC Berkley 1997

2
Overview

• Challenges
• Review
• Method
• Results
• Conclusions

3
Challenges

• High Accuracy
• Frequent Inter-part Occlusion
• Low Contrast

4
Review
5
Review

• Motion capture on synthetic images
• ORouke and Balder, 1980
• 1 DOF marker free tracking
• Hogg, 1983. Rohr, 1993
• Higher DOF full body tracking
• Gravrila and Davis, 1995

6
Review

• About the previous work
• All in controlled environments with high contrast
  and clear edge bounries
• Most use skintight suits or markers
• Camera calibration needed

7
Method
8
Method

• Basic Assumptions
• From frame to frame, all intensity pixel
  intensity changes are local
• u is motion model and is written as a matrix
  equation

9
Method

• Finding Gradients
• Gradient form of the first equation
• Find a least squares solution to f
• Warp image I(t1) based on f
• Find new gradients
• Repeat to minimize

10
Method

• Motion as twists
• Standard pose matrix to move from object space to
  camera space (3D)
• Scaled orthographic projection moves to image
  space
• Requires knowing something about the 3D model of
  the image. Approximated as ellipsoids.

11
Method

• Motion as twists
• Any motion can be represented as a rotation about
  an axis, and a translation about that axis
• For example,
• to make this motion

12
Method

• Motion as twists
• You make this motion

13
Method

• Motion as twists
• Twists can be represented as small vector or
  matrix
• Can be made to a pose by
• Encode the motion of a pixel between two frames

14
Method

• Motion as twists
• Linear algebra manipulation allows using the
  twist vector to write a motion equation for each
  pixel
• Those equations are put in a vector and used to
  find a global f parameter for that object

15
Method

• Kinematic chains
• Body parts represented as multiple connected
  objects
• Each object can be found by the top pose and an
  angle and twist for each object down the chain
• More linear algebra is used to find a f for each
  body part

16
Method

• Multiple cameras
• Adds accuracy because change of fully occluded
  parts drop with each view
• Normal motion equation is
• H is system of equations for each pixel
• f is global parameter vector for each object
• z is initial position of the pixel

17
Method

• Multiple cameras
• Adding synchronized cameras
• H becomes a matrix where each column represents a
  view
• The f vector gets a term W for each view that
  represents the pose seen from that view
• z becomes a vector with an initial position for
  each view.

18
Method

• Support maps
• Limits pixel search to area defined by map for
  speed
• Value for each pixel in range 0,1, where 1
  means pixel is in the region
• Method for finding starts as an elliptical guess,
  but refining it is not described

19
Method

• Algorithm review
• Input Image I(t), I(t1), pose and IK angles
• Output Pose and IK angles for I(t1)
• Find 3D points for each pixel in image
• Compute support map for each segment
• Set poses and IK angles for I(t1) I(t)
• Iterate
• Compute gradients
• Estimate f
• Update poses and IK angles
• Warp image based on the pose and support map

20
Method

• Initialization
• Algorithm depends on known positions for the
  first frame
• For multiple views, each first frame must be
  initialized
• User clicks joint positions, and 3D estimations
  and joint angles are computed
• Values like symmetry can be enforced

21
Results
22
Results
In Lab Movie

• Single angle
• 53 frames with decent results
• Upper leg hard to track, so IK chain compensates
  with lower leg and torso

23
Results
Oblique Lab Movie

• Oblique angle
• Tracking over 45 frames
• Algorithm could track change in scale due to
  perspective changes

24
Results

• Oldest known movie
• High noise and low contrast
• Low framerate
• Multiple views

Digital Muybridge
25
Conclusions

• Future Work/Shortcomings
• May break with large movements
• Fixed camera only
• Did not show tracking of back limbs
• No timing data
• Few results