Eugene Hsu

1
Style Translationfor Human Motion

• Eugene Hsu
• Kari Pulli
• Jovan Popovic

2006. 9. 26 Sangwook Yoo
2
Outline

• Introduction
• Previous Work
• Correspondence
• Style Translation
• Results
• Conclusion

3
Introduction

• Human Motion Different Style

Preserve content
Rapid transform
Ex) Normal Walk Sneaky Walk
4
Introduction Get TM
Sneaky Walk motion example
Normal Walk motion example
Learning a Translation Model
5
Introduction Use TM
Translating Motions
6
Previous Work

• Motion Generation
• Concatenation, e.g., Kovar et al. 2002
• Interpolation, e.g., Rose et al. 1998
• Statistical, e.g., Brand and Hertzmann 2000
• Physical, e.g., Liu et al. 2005
• Motion Transformation
• Motion Warping, Witkin and Popovic 1995
• Frequency Filtering, Bruderlin and Williams 1995
• Emotional Transforms, Amaya et al. 1996

7
Correspondence

• DTW (Dynamic Time Warping)
• Motion warping approach Witkin and Popovic 1995
• Temporal warp
• Applied to correspondence,
• Stylistically different motion need spatial
  warp
• Input motion ? spatial/temporal warp ? Output
  motion as close as possible

8
Correspondence

• Iterative Motion Warping (IMW) algorithm
• Assumption Single-dimensional motion
• u input motion, y output motion
• U diag(u)
• a scale vector, b offset vector ? Space warp
• W warp matrix ? Time warp

9
Correspondence

• Fa, Gb first derivatives of a and b
• F a
• -2   2   0   0   0 a1 (a2-a1)
• 1 -1   0   1   0   0 a2 (a3-a1)/2
• -      0  -1   0   1   0 a3 (a4-a2)/2
• 2       0   0  -1   0   1 a4 (a5-a3)/2
•   0   0   0  -2   2 a5 (a5-a4)

10
Correspondence Full Algorithm

• Uniformly lengthening y
• After initializing a 1 1, b 0 0,
• Time-warp and space-warp stages are alternated
  (Coordinate descent)
• Optimize E with a, b fixed
• Optimize E with W fixed
• Convergence ?
• E only contains quadratic terms lower bounded
• Each iteration performs a global opt. E cannot
  increase

11
Correspondence Time Warp

• Approximate String Matching Problem
• Given two string p p1 pm, q q1 qn (n
  gt m)
• Fj(p) j-th character of the edited p ? only
  repetition
• Ex) a ? ar (1r k, k repetition limit)
• c cost function (0 if arguments are same, 1
  otherwise)
• Dynamic programming
• Ex) p a b c d e, q a a b c d e ? F(p)
  a a b c d e

12
Correspondence Time Warp

• Convert to Finding Correspondence Problem
• Given two motion p p1 pm, q q1 qn (n
  gt m)
• Cost function c ? squared difference
• Edit function F ? integer repetition vector r
• 1a column vector of a ones
• R repetition matrix
• Constraints
• ri ?1, , k, ?iri n

13
Correspondence Time Warp

• More general situation
• Algorithm lengthens p to compute the
  correspondence with q
• What if p is longer than q?
• First, lengthening q by s repetitions of each
  element

14
Correspondence Time Warp

• Optimize E with a, b fixed
• Lengthening y
• p Uab, q y
• After minimization, W repetition matrix R
• ?

IMW objective function
Time warp objective function
15
Correspondence Space Warp

• Optimize E with W fixed
• Zeroing the first derivative of E w.r.t. a and b
• Can use Gaussian elimination to solve efficiently

(dE/da 0, dE/db 0)
?
16
Correspondence Full Dimension

• So far, motions in single dimension
• Undesired
• the legs must be synchronized in the same time
  frame
• Solution
• Time warp replace by the squared norm of the
  frame difference
• Space warp unmodified and use independently for
  each DOF

17
Correspondence Usage

• a, b
• Auxiliary terms
• W (Warp matrix)
• Repetition vector w
• w w1 w2 wn (wi repetition number of
  input motion frame i)
• diag(WTW)
• But, w encodes correspondence of u to the
  s-lengthened
• version of y
• w is divided by the s

18
Style Translation

• LTI(Linear Time-Invariant) model
• approximate the relationship - input and output
  style
• its parameters are estimated from the training
  data using existing algorithm and implementation
• Given a input frames ut in input style,
• Translation system computes a sequence of frames
  yt in output style

19
Style Translation Representation

• Time
• encode the time warp so that it can be reversed
  during synthesis.
• Attach elements of w to their corresponding
  frames.
• w correspond to a monotonically increasing
  timeline
• No guarantee that the estimated LTI model produce
  positive w-values
• Store the logarithm
• Inversion exponential produces positive values.

20
Style Translation Representation

• Root joint
• Encode the root position and orientation relative
  to the previous frames root position and
  orientation.
• Joint orientations
• Exponential map Grassia 1998
• Normalize all DOFs to have 0-mean and
  unit-variance across all frames

21
Style Translation Translation Model

• Each joint is translated independently

Root
TM
Root
Hip
TM
Hip
Knee
TM
Knee
Elbow
TM
Elbow



Root
TM
Timing
22
Style Translation Translation Model

• Why linear time-invariant (LTI) model ?
• Relationship between two motions (ut input, yt
  output)
• How about linear Regression ?
• yt Dut

LTI model Translation Model
ut
yt
23
Style Translation Estimation

• Uses example motions to infer the model
  parameters
• Classic system identification task with many
  standard and robust algorithms Ljung 1999
• Used N4SID algorithm Van Overschee and De Moor
  1996
• MATLAB system identification toolbox
• 1) Estimate the optimal state sequence
• 2) System matrices are estimated

24
Style Translation Usage

• Set the initial condition x0 0 0
• As each input frame arrives, converted to the
  proper representation
• Above is inserted into the LTI model
• Produces output
• The steps of representational transformation are
  inverted

25
Style Translation Postprocessing

• May violate kinematic constraints imposed by the
  environment
• Footstake can be corrected by Kovar et al. 2002
• Heuristic Generalization
• Motion outside the training examples
• Artifacts
• Detect and blend
• original input motion and output of the LTI model

26
Results

• All motions
• 120 fps, 35 DOFs
• Walking examples
• Normal, Supermodel, Sneaky, Limping, Shuffling
• Translation Normal walk into others
• Fighting
• Weak, Aggressive
• Translation Weak style into Aggressive style

27
Conclusion

• IMW algorithm
• LTI model
• Not physically correct
• Biomechanics insights about human motion