Representing and animating human movement for dance

1
Representing and animating human movement for
dance

• Tom Calvert
• School of Interactive Arts and Technology
• Simon Fraser University

2
Summary

• What is this about? How is movement captured?
• History
• Compare and contrast notation, animation and
  motion capture
• Dance friendly software
• Research questions

3
How is movement captured?
Labanotation score
Human notator
Dance performance
Animation file - keyframes
Human animator
Mocap Instrumentation
Mocap file - .bvh, etc
4
Differences between notation, animation and mocap

• Notation created by human notator
• Notator uses knowledge of human movement and
  dance to analyse movement
• The notation score provides a high level
  representation
• Physical and temporal accuracy is low
• Animation score created by human animator
• Analysis is similar to that of a notator but
  output is a series of animation keyframes not
  as analytic as notation
• Accuracy is lower than mocap but higher than
  notation
• Motion Capture depends on instrumentation
• No analysis a description
• A smart program is needed to thin mocap to
  animation keyframes
• Spatial and temporal accuracy are high

5
Question

• Which representation is best
• For archival purposes?
• For reconstruction?
• For animated performance?

6
Notation

• Notation?
• Analogous to a musical score conveys the intent
  of the composer
• Very few can read and even fewer can write
  notation
• Can we translate notation into animation? See
  LabanEditor2 or LabanDancer (demo)
• Can we translate animation into notation? Not yet

7
Animation

• Animation?
• Shows the intent of the composer
• An enhanced score such as that used by DanceForms
  can add notes to the keyframes
• Easy to edit - DanceForms
• Very easy to visualize when played as an
  animation
• Can this be translated into notation? Not yet.

8
Motion Capture

• Mocap?
• Provides a dense series of frames that can be
  played as an animation. So the information is
  there
• Analogous to a musical recording
• Hard to edit without thinning
• Can this be translated into an animation score?
  Yes with a thinning program
• But this is different animation from that created
  by an animator

9
Analysis vs Accuracy
Analysis (intent)
notation
animation
mocap
Accuracy (what happened)
10
History

• Merce Cunningham 1966
• A.M. Noll (Bell Labs) 1967 Dance Magazine
• Zella Wolofsky 1973 - SFU
• Steve Smoliar 1978 Univ of Pennsylvania
• Norm Badler 1978 Penn
• Maxine Brown 1978 Penn
• Tom Calvert 1978 use of notation for animation
• Not so many others for some years

11
More recently

• Lucy Venable and colleagues LabanWriter 1987
  present
• Rhonda Ryman and colleagues MacBenesh 1985
• Tom Calvert colleagues DanceForms 1995
  present
• Kozaburo Hachimura, Minako Nakamura and
  colleagues LabanEditor 2001
• Rhonda Ryman DanceMoves 2002
• Ilene Fox, Rhonda Ryman, Tom Calvert and Lars
  Wilke LabanDancer 2004
• Kozaburo Hachimura, Minako Nakamura and
  colleagues LabanXML 2004

12
What is movement notation?

• At least 80 systems worldwide
• Three in more general use
• Labanotation
• Rudolf Laban - 1928
• Vertical staff more later
• Most common in North America also used in tm
  studies
• Effort-shape
• Benesh Notation
• Joan Rudolf Benesh 1940s
• Horizontal staff similar to a music score
• Most commonly used for ballet in Europe
• Editor developed at University of Waterloo
• Eshkol-Wachman Notation
• Noa Eshkol - 1980
• Easiest system to interpret but least used in
  dance
• Ilan Golani notated rat, mouse and seal flipper
  movement

13
Labanotation I

• Centre two columns represent support
• Outer columns represent gestures
• By using presigns every limb segment can be
  individually identified and notated

14
Labanotation II
Shading indicates level
Columns represent body parts
Symbols for direction
A score
15
Dance friendly software

• LabanWriter
• An editor for composing and editing Labanotation
  scores
• DanceForms
• A keyframe based human figure animation tool
  customized for dance
• Proposal to link LabanWriter DanceForms
  resulted in
• LabanDancer
• A prototype system for animating Labanotation
  scores

16
Notation Editors

• LabanWriter
• A Mac based Labanotation Editor
• Developed in the Dance Dept at the Ohio State
  University
• Team led by Professor Lucy Venable
• Others
• LabanEditor 2 Labanotation Editor developed in
  Japan (Nakamura and Hachimura) This provides
  animation of the score
• MacBenesh - A Benesh editor developed at
  University of Waterloo (Singh, Beatty, Booth,
  Ryman)
• LED Don Herbison-Evans, UTS, Sydney
• Several others

17
LabanWriter
18
LabanWriter Notes

• Only available for the Mac
• Version 4.6 has recently been released
• Project lacks support and PC version on hold
• LabanWriter is strictly a graphical editor
• The system records the symbol type, the location,
  orientation and size
• No explicit knowledge is stored for example,
  the system does not provide direct information on
  the limb to which a symbol refers. This is
  deduced from the position on the score.

19
Animation

• Created by an animator using a variety of tools
• Although procedural animation and dynamics tools
  are available, many animators stick to keyframe
  animation
• Keyframes are analogous to Labanotation symbols,
  but do not indicate dynamics
• Adding annotation to keyframes helps

20
Animation systems

• There are many, many animation software systems
  available commercially
• Some popular examples are 3D Studio Max, Maya,
  Motion Builder and Poser
• General purpose animation systems are hard to
  learn.
• DanceForms is a simple animation system optimized
  for articulated figures (e.g. humans) and
  designed to be dance friendly

21
DanceForms

• Derived from Life Forms human animation system
  developed at SFU (1985-95)
• Further developed and marketed by Credo
  Interactive Inc. (www.danceforms.com)
• Available for Mac and Windows
• Can import and export motion files in a wide
  variety of formats
• Less features than general purpose animation
  systems but easier for dancers to use
• Extensive content available Ballet Moves,
  PowerMoves, MegaMocap, etc

22
DanceForms
23
Studio window
24
Score (timeline)
Notes can be added to the score
25
Stage
26
Performance
27
DanceForms Demo
28
Motion Capture

• There are many systems available using different
  modalities
• Optical passive markers
• Optical active markers
• Electromagnetic
• Electromechanical
• All allow a numerical description of body
  movement for all or part of the body

29
Early motion capture
Goniometers - 1978
30
Applications to Dance

• As a tool to capture specific movements
• As a tool for composition Will Smith demo
• As a research tool to characterize different
  aspects of dance
• Increasing interest in using mocap as an archival
  tool Danny Grossman demo

31
Mocap demos

• Will Smith demo
• Danny Grossman demo
• Repository demo

32
Will Smith demo

• Will Smith is a professor at University of
  Lethbridge in Alberta, Canada. He uses mocap as a
  compositional tool.
• http//people.uleth.ca/aw.smith/isabelle.html
• Three Blue Rocks
• this was one single take that uses the same
  memory processes as a jazz musician. I had not
  prechoreographed it, rather, created the movement
  moment by moment, relying on my experience to
  assist me to 'get it right' and create a coherent
  structure. Yes, and I captured this motion when I
  was in my '50s.

33
Danny Grossman demo

• Danny Grossman is a well known figure in modern
  dance in Canada
• The Danny Grossman Dance Company (DGDC) wishes to
  document Grossmans key works for education and
  reconstruction.
• Curious Schools of Theatrical Dancing figures
  prominently in this
• Rhonda Ryman is leading a study to compare the
  use of animation and motion capture to represent
  these dances.
• See www.dannygrossman.com
• http//www.sfu.ca/tom/Interesting/RymanCorpsPaper
  July507.pdf

34
Danny Grossman
35
Danny Grossman Demo

• Figure in white Gregario was created in
  keyframe animation by Rhonda Ryman
• Animation took many months using video and
  consultation with choreographer and artists
• Figure in black mocap of Eddie Kastrau
  performing the same piece
• Created in one session at Western Michigan
  University

36
Translation between modalities

• Mocap to Animation Thinning program
• Notation to Animation LabanDancer and Laban
  Editor 2
• Mocap or Animation to Notation an open
  problem..

37
From Notation to Animation

• 1973 Zella Wolofskys program to interpret
  selected Labanotation commands
• 1994 Dance Notation Bureau leads a series of
  projects to develop a system to translate
  notation to animation and vice-versa
• Projects involved DNB, SFU, Waterloo, Ohio State
  and Credo Interactive Inc
• Funded by National Endowment for the Humanities,
  National Endowment for the Arts and National
  Initiative to Preserve Americas Dance (NIPAD)

38
LabanDancer I

• These projects have resulted in a prototype
  system - LabanDancer
• System reads LabanWriter (.lw4) files and creates
  animation
• System prototype developed for Mac and Windows
• Development is currently on hold for financial
  and intellectual reasons

39
LabanDancer Interface
40
LabanDancer II

• Create a composite score symbols are mapped to
  animation channels
• Keyframe animation channels control joint angles
  in the skeleton
• The priority was to adopt approaches with general
  applicability
• 3 command types gestures, support, other
• IKAN inverse kinematics used for 4 chains 2
  arms and 2 legs
• Support commands translated into footprints and
  animation created with van de Pannes algorithm
• System must handle numerous instances of implicit
  constraints that depend on context

41
Three steps in animating support
notation score
42
Principles of the Footprints Algorithm
Heuristic approach Optimize criterion function E
43
LabanDancer Demo

• See www.sfu.ca/tom/Papers for paper

44
Other developments

• New display environments
• Large screens
• Remote collaboration for rehearsal and
  performance
• Web based environments
• Interactive learning environment to support
  choreography
• Canadian National Arts Centre
• Dance site at ArtsAlive.ca
• Virtual worlds Second Life

45
Dance in Second Life

• Second Life Ballet Company
• Import animation clips using bvh format
• Limited to 30 seconds
• Costs 10 Linden dollars!

46
Revisit the Questions

• Which representation is best
• For archival purposes?
• For reconstruction?
• For animated performance?

47
Notation

• Notation?
• Analogous to a musical score conveys the intent
  of the composer
• Very few can read and even fewer can write
  notation
• Can we translate notation into animation?
  LabanEditor 2 prototype exists and we have seen
  the LabanDancer demo

48
Animation

• Animation?
• This is the result of analysis by the animator
• An enhanced score such as is used by DanceForms
  can add notes to the keyframes
• Easy to edit - DanceForms
• Very easy to visualize when played as an
  animation
• Can this be translated into notation? Not yet.

49
Motion Capture

• Mocap?
• Provides a series of keyframes that can be played
  as an animation. The information is there but it
  is not the same as an animation created by an
  animator
• Analogous to a musical recording
• Hard to edit without thinning
• Can this be translated into an animation score?
  Yes with a thinning program
• Can this be translated into notation? Not yet.

50
Analysis vs Accuracy
Analysis
notation
animation
mocap
Accuracy
51
Why is it difficult to move forward?

• There is a need for a unique, unambiguous way to
  represent human movement
• Both analysis and accuracy are important
• The LabanDancer composite score is a start
• Labanotation is helpful but even that does not
  provide an unambiguous representation
• There are many, many special cases
• It is fairly easy to develop prototypes for
  restricted application
• A truly comprehensive implementation is very,
  very hard

52
Major open questions include

• Can 3D animation replace notation for archival
  purposes?
• How to create an unambiguous, unique
  representation of human movement?
• A workshop on movement, dance and notation
  methods held at Ohio State in 2004 called it an
  interlingua and identified it as the top
  priority for the field
• See www.sfu.ca/tom/Interesting
• The working plan is to create an XML based
  representation

53
Analysis vs Accuracy
Analysis
notation
inter-lingua
animation
mocap
Accuracy
54
The need for an interlingua
55
LabanXML/DanceXML

• Analogous to MusicXML
• A start has been made with LabanXML by Minako
  Nakamura, Kozaburo Hachimura and their colleagues
  in Japan
• A student at SFU developed DanceXML but this was
  not comprehensive

56
LabanXML example
57
Conclusion

• In 1973 the challenge for me was to interpret and
  animate Labanotation
• Today this is still a challenge
• The more basic challenge is to develop an
  unambiguous inter-lingua representation
• This could allow easy exchange between all other
  representations

58
Interesting Websites

• Dance Notation Bureau www.dancenotation.org
• DanceForms www.danceforms.com
• LabanWriter
• http//www.dance.ohio-state.edu/3_research_gallery
  /laban_writer.html
• LabanEditor
• http//www.img.is.ritsumei.ac.jp/7Ehachi/hachi_e.
  html
• National Arts Centre ArtsAlive.ca
• Will Smith http//people.uleth.ca/aw.smith/isabe
  lle.html
• References
• www.sfu.ca/tom/Papers
• www.sfu.ca/tom/Interesting
• Contact tom_at_sfu.ca