Performance Evaluation of Vision-based Real-time Motion Capture

1
Performance Evaluation of Vision-based Real-time
Motion Capture

• Naoto Date, Hiromasa Yoshimoto, Daisaku Arita,
• Satoshi Yonemoto, Rin-ichiro Taniguchi
• Kyushu University, Japan

2
Background of Research

• Motion Capture System
• Interaction of human and machine in a virtual
  space
• Remote control of humanoid robots
• Creating character actions in 3D animations or
  video games
• Sensor-based Motion Capture System
• Using Special Sensors (Magnetic type, Infrared
  type etc.)
• Users action is restricted by attachment of
  sensors
• Vision-based Motion Capture System
• No sensor attachments
• Multiple cameras and PC cluster

3
Key Issue

• Available features acquired by vision process is
  limited.
• Head, faces and feet can be detected robustly.
• How to estimate human postures from the limited
  visual features
• Three kinds of estimation algorithms
• Comparative study of them

4
System Overview
camera
PC
PC
PC
camera
PC
camera
PC
camera
PC
camera
5
System Overview
Using 10 cameras for robust motion capture
camera
PC
camera
PC
camera
PC
camera
PC
camera
6
System Overview
1 top-view camera on the ceiling
camera
PC
camera
camera
PC
camera
PC
camera
7
System Overview
9 side-view cameras around the user
camera
camera
camera
camera
PC
camera
8
System Overview
Using PC cluster for real-time feature
PC
PC
PC
PC
camera
PC
camera
PC
camera
9
System Overview
First, take images with each camera
camera
PC
camera
PC
camera
PC
camera
PC
camera
10
System Overview
Extract image-features on the first stage PCs
camera
PC
PC
camera
PC
camera
PC
camera
PC
camera
11
System Overview
Reconstruct human CG model by feature parameters
in each image
PC
PC
PC
PC
camera
PC
camera
12
System Overview
Synchronous IEEE1394 cameras 15fps
camera
camera
PC
camera
PC
camera
PC
camera
13
System Overview
CPU Pentium?700MHz x 2 OS
Linux Network Gigabit LAN Myrinet
PC
PC
PC
camera
PC
camera
camera
PC
14
Top-view camera process

• Background subtraction
• Opening operation
• Inertia principal axis
• Detect body directionand transfer it

15
Top-view camera process

• Background subtraction
• Opening operation
• Inertia principal axis
• Detect body directionand transfer it

16
Top-view camera process

• Background subtraction
• Opening operation
• Inertia principal axis
• Detect body directionand transfer it

17
Top-view camera process

• Background subtraction
• Opening operation
• Feature extraction
• Inertia principal axis
• Body direction

18
Side-view camera process

• Background subtraction
• Calculate centroids of skin-color blobs

19
Side-view camera process

• Background subtraction
• Calculate centroids of skin-color blobs

20
Side-view camera process

• Background subtraction
• Calculate centroids of skin-color blobs

21
Estimate 3D position of skin-color blob

• From all the combination of cameras and blob
  centroids, we select all possible pairs of lines
  of sight. Then we calculate an intersection
  point of each line pair. Unless the distance of
  the two lines is smaller than a threshold, we
  decide there is no intersection point.

22
Estimate 3D position of skin-color blob

• The calculated points are clustered according to
  distances from the feature points (head, hands,
  feet) of the previous frame.
• Select points where feature points are dense as
  the 3D positions of the true feature points.

23
Estimate 3D position of torso
A method based on simple body model
Center point
24
Performance evaluation of right hand position
estimation
25
Estimate 3D positions of elbows and knees

• 3 estimation methods
• Inverse Kinematics (IK)
• Search by Reverse Projection (SRP)
• Estimation with Physical Restrictions (EPR)

26
Estimate 3D positions of elbows and knees

• IK
• f3 assumed to be a constant

27
Estimate 3D positions of elbows and knees

• SRP

28
Estimate 3D positions of elbows and knees

• EPR
• An arm is assumed to be the connected two spring
  model.
• The both ends of a spring are fixed to the
  position of the shoulder, and the position of a
  hand.
• The position of an elbow is converged to the
  position where a spring becomes natural length.
  (the natural length of springs is the length of
  the bottom arm and the upper arm which acquired
  beforehand.)

29
Accuracy of estimating right elbow position
30
Accuracy of posture parameters
31
Visual comparison of 3 methods
32
Computation time required in each algorithm

• Top-view camera processing 50ms
• Side-view camera processing 26ms
• 3D blob calculation 2ms
• IK calculation 9ms
• SRP calculation 34ms
• EPR calculation 22ms
  

33
Online demo movie (EPR)
34
Conclusions

• We have constructed a Vision-based Real-time
  Motion Capture System and evaluated its
  performance
• Future works
• Improvement of posture estimation algorithm
• Construction of various applications
• Man and machine interaction in a virtual space
• Humanoid robot remote control system