Agent animation: capabilities, issues, and trends

1
Agent animation capabilities, issues, and trends

• Paolo Petta
• Austrian Research Institute for Artificial
  Intelligence, Vienna

2
Introduction

• Computer animation developments
• Geometry
• Resolution, detail
• Model-driven dynamics
• Ambient physics modeling, Behavioural modeling
• Control
• Interactivity, communication techniques,
  autonomy, learning
• Population
• Multiple actors, distributed systems

3
Typical Applications

• Synthetic characters,virtual Humans,visualisatio
  n/simulation
• Design choices
• Sparse top-down models vs.complete bottom-up
  models
• Application requirements
• deep-and-narrow vs.
• broad-and-shallow

4
Research topics
Artificial Intelligence
Robotics
User Interface
User interfaceforEmotion control Actor
behaviouremotion control
Animation
Vision-basedanimation Path planning
Kinematics Dynamics
Walkingmodels Objectgrasping
Behaviouralanimation Spatialrelationships
shape transformation Collision
detection Facial
animation
Clothanimation
Collisionresponses
Musclemodels
Geometric Modelling
Finite-element deforma-tions
Facedesign
Hair
Physics
ImageSynthesis
Skin texture
5
Auxilliary sciences

• Artificial Life
• Biology/Ethology
• Dramatic Arts
• Embodied Artificial Intelligence/Robotics
• Physics
• Psychology
• Sensor technology
• Vision

6
IMPROV (MRL, NYU)

• Artistic and commercial applications
• Animated staging
• Choreography
• Interactive multi-user environments
• ...
• Surface model of moodemotions
• Productivity tool
• API for laypersons(educators, historians,
  social scientists)

7
IMPROV

• Microlevel
• Procedural animation
• Accurate modeling of single actions and all
  permissible transitions
• Statistically controlled parameter randomization
  for variability and consistency

8
IMPROV

• Microlevel
• Behavioural layering
• Scripts are classified in a hierarchy according
  to level of behaviour
• User-defined connections between layers define
  the effective heterarchy
• Action selectiondeterministic linear scripts or
  stochastic selection from alternatives
• Exclusion of pursuit of conflicting goals at same
  level
• Parallelism across the hierarchy

9
IMPROV

• Macrolevel
• Blackboard architecture

Characters (attributes scripts) Avatars Story
agent (director)
Stage Manager
10
IMPROV

• Macrolevel
• Behaviour layers spanning across groups of agents
  forcoordinated action
• Distributed environment modeling Inverse
  Causality (gt MOO)
• information about interactions is attached to
  objects
• characters are contaminated by use (new/update
  of state variables competence learning)

11
Edge of Intention (Oz, CMU)

• Interactive drama
• Believable autonomous characters
• Goal-directed
• Emotional(folk theory of emotions, OCC)
• Simple appearance, emphasis on behaviours(-gt
  internal processing)
• Interaction modes
• Moving/gesturing, talking (typing)

12
TOK architecture

• Microlevel
• Hap
• Goal-oriented reactive action engine
• Static plan library
• Action behaviours
• Emotion behaviours
• Sensing behaviours
• Sensing of low-level actions of other Woggles
• Action blending

13
TOK architecture

• Microlevel
• Em
• Model of emotional and social aspects
• Explicit state variables for beliefs and
  standards of performance
• Variables are influenced by comparison of current
  goal states with events and perceived actions
  (thresholding)

14
TOK architecture

• Microlevel
• Behavioural features
• Mapping of emotional state to overt behaviour
• Manifestation of personality
• Tight integration of Hap and Em
• No need for arbitration

15
TOK architecture
behaviour featuresand raw emotions goal
successes,failures creation
standardsattitudesemotions Em
goalsbehaviours Hap
senselanguagequeries
senselanguagequeries
sensory routines andintegrated sense model
The world
16
TOK architecture

• Macrolevel
• Fixed plan library encodes all possible
  communications/interactions

17
ALIVE (MIT Media Lab)

• Entertainment
• Magic mirror metaphore
• Unincumbered immersive environment

18
ALIVE

• Microlevel
• Hamsterdam
• Behaviour system for action selection
• Based on ethological model
• Sensory inputs via release mechanism
• Loose hierarchy of behaviour groups
• Avalanche effect for persistent selection
• Inhibited behaviours can issue secondary and meta
  commands
• Motor skills layer for coordination of motions
• Geometry layer for animation rendering

19
ALIVE
External World World SensorySystem ReleasingMec
hanism
Goals/Motivations
InternalVariable
InternalVariable
Levelof Interest
Inhibition
Motor Commands
20
ALIVE
21
ALIVE

• Levels of control
• Motivations via variables of single behaviours
• You are hungry
• Directions via motor skills
• Go to that tree
• Tasks via sensory, release, and behaviour systems
• Wag your tail

22
ALIVE

• Increased situatedness
• Synthetic vision
• For navigation
• Generic interface
• Plasticity
• reinforcement learning (conditioning)

23
ALIVE

• Macrolevel
• Totally distributed control

24
Virtual Humans (Miralab/EPFL)

• Goal
• Simulation of existing people
• Real-time animation of virtual humans that are
  realistic and recognizable
• Inclusion of synthetic sensing capabilities
  allows simulation of (seemingly) complex
  capabilities,e.g. real-time tennis

25
Virtual Humans

• Issues requiring compromising
• Surface modeling
• Deformation
• Skeletal animation
• Locomotion
• Grasping
• Facial animation
• Shadows
• Clothes
• Skin
• Hair

26
Virtual Humans

• Methodology
• Modeling
• Prototype-based
• Head and hand sculpting
• Layered body definitionSkeleton, Volume, Skin
• Animation
• Skeleton motioncaptured, play-back, computed
• Body deformationfor realistic rendering of
  joints
• Detailled hand and facial animation

27
Virtual Humans

• Synthetic sensing as a main information channel
  between virtual environment and digital
  actor(since ca. 1990)
• Synthetic audition, vision and tactile
• Differs fundamentally from robotic
  sensingdirect access to semantic information

28
Virtual Humans

• Example synthetic vision
• Environment is perceived from a field-of-view
  that is rendered from the actors point of view
• Access to pixel attributescolor,
  distance,index to semantic information
• Simple case color coding of objectsgt
  perception of color recognition of object
• Object attributes areretrieved directly from the
  simulation

29
Virtual Humans

• Navigation
• Path planning obstace avoidance
• Global navigation
• Based on prelearned model
• Determines the global navigation goal
• Local navigation
• Purely indexical, based on sensinggt No need for
  model of environmentgt No need for current
  position
• Three modules
• synthetic vision, controller, performer

30
Virtual Humans

• Navigation controller
• Regularly invokes vision to retrieve updated
  state of environment
• Creates temporary local goals if an obstacle up
  front
• Local goals are determined by obstacle-specific
  Displacement local automata

31
Virtual Humans

• Interaction with the environmentSmart Objects
• Each modeled object includes detailled solutions
  for each possible interaction with the object
• Objects are modeled according to situated
  decomposition

32
Virtual Humans

• Smart Objects include
• Description of moving parts, physical properties,
  semantic index(purpose and design intent)
• Information for each possible interaction
  position of interaction part, position and
  gesture information for the actor (capacity
  limits!)
• Object behaviours with state variables (gt actor
  state info)
• Triggered agent behaviours

33
Virtual Humans

• Example virtual tennis
• Actor model based on stack machine of state
  automata
• Actor state can change according to currently
  active automaton and sensorial input

34
Virtual Humans
Architectureof behaviourcontrol
35
Virtual Humans
Tennisgameautomata sequence
36
JACK (UPenn)

• Ergonomic environment analysis
• Workplace assessment
• Product evaluation
• Device interfaces
• Logistics

37
JACK

• Microlevel
• Biomechanically correct model
• Synthetic sensors for high-level behaviours
• Three-level architecture realising truly
  situated low-level behaviour

38
JACK

• Microlevel

(learned sense-control-act loop parameters)
39
JACK

• Macrolevel
• Taskable virtual agent
• Global intentions and expectations of all
  characters are statically captured (explicitly
  anticipated)
• Parallel Transition networks

40
JACK

• Macrolevel PaT Net

41
Topics for Discussion

• Completeness of modeling
• True agent characteristics(WooldridgeJennings)
  
• Autonomy
• Social abilities
• Reactivity
• Pro-activeness

42
Topics for Discussion

• The TLA Debate
• Situatedness/synthetic sensing
• Variability/adaptiveness/plasticity
• Believability

43
Modelling completeness

• Sparse models
• Abstract, top down
• Based on explicit, reified design elements
• Bridging/obviating of full detail by careful
  selection of modeled elements
• Broader coverage at differing resolution
• Believability/impression over fidelity
• (Bound to) Lose in the long run?

44
Modelling completeness

• Complete models
• Situated, bottom up
• Depend on balanced design(including
  environmentcoupling)
• Limited coverage/complexity
• Allow for flexible action-selection
• Fidelity over believability/impression
• Win in the long run?

45
Autonomy (McFarland/Boesser)

• Automatonstate-dependent behaviour
• Autonomous agentself-controlling, motivated
• Motivationreversable internal processes that
  are responsible for changes in behaviour
• Multiple goals/actions are the rule!gt
  concurrency, transitioning
• Insights on own skillsconditions of applicability

46
Social abilities

• Deep agent modeling
• Of the self BDI and variants
• Of others (recursively)
• Of the society
• Coordination
• Communication
• Generationunderstanding of facial expressions,
  postures, gestures, task execution, text/speech,
• (social) Emotions(including display rules)

47
Social abilities

• From Action Selection to Action expression
• Sign management context-dependent behaviour
  sematics
• What should an agent do at any point in order to
  best communicate its goals and activities?
• Goal increase comprehensibility of behaviour

48
Believability

• Quality vs. correctness
• Self-motivation
• pursuit of multiple simultaneous goals
• gt entails requirement of broad capabilities
• Personality/Emotion
• Plasticity/change over time
• Situatedness
• social skills
• affordances

49
And then...

• Methodologies for assembly of architectures with
  understandable/predicatable (motivated,
  goal-directed,) behaviour
• Agent control systems
• Persistency, plasticity
• Agent animation as simulation