State of the Art in Emotion and Personality

1
State of the Art in Emotion and Personality

• Elisabeth André
• University of Augsburg, Augsburg, Germany
• DFKI GmbH, Saarbrücken, Germany

2
Why should we care about emotional agents?

• To increase believability
• Joe Bates The role of emotions in Believable
  Agents, CACM 1994
• ... emotion is one of the primary means to
  achieve this believability, this illusion of
  life, because it helps us know that characters
  really care about what happens in the world, that
  they truly have desires.
• To engage the user
• David Freeman Four Ways to Use Symbols to Add
  Emotional Depth to Games, Game Developer 2002
• ... shallow emotional experience in games
  featuring stories and characters will
  increasingly stand out negatively in consumers
  mind

3
Why should we care about emotional agents?

• Central to human experience
• Essential for movies, drama, games
• Social glue (affection, guilt)
• Motivators (fear)
• Readily manipulated (by storytellers, salesmen)
• Central to many theories of intelligence
• Emotion, cognition, behavior tightly coupled
• Focus of Attention (Matthews and Wells)
• Belief formation (Frijda, Manstead, and Bem)
• Decision Making (Damasio, Murnighan)
• Learning, Tutoring, Training (Lepper, Frasson,
  Lester)

4
Why should we care about emotional agents?

• Key to Pedagogy
• Example CARMEN
• Subject matter concerns highly emotionally
  charged situation
• Emotions and how people cope critical to subject
  matter
• Help users develop positive emotional attitude
• To influence the users behavior by showing
  empathy
• Example Computer Petz

5
Issues and Directions

• Sources of emotions
• Task-oriented emotions
• Social and organizational factors
• Consequences of Emotions
• Expressive behaviors
• Impact on speech generation and synthesis
• Impact on decision-making
• Impact on Human Participant
• Impact on believability, engagement
• Impact on learning

6
Schema for comparing different approaches

• Underlying theory of emotion and personality
• Purpose of emotions and personality
• Emotion and personality input
• Emotion and personality processing
• Emotion and personality output
• Implementation aspects

7
Models of Personality and Emotions

• Personality
• Duration long-term
• Focus factors that determine personality are
  diffuse and indirect
• Five Factor Model Extraversion, Agreeableness,
  Openness, Conscientiousness, Neurotism
• Emotions
• Duration Short-lived
• Focus influenced by particular events, agents or
  objects
• OCC Model Emotions as valenced reactions to
  events, agents or objects

8
OCEAN

• Open curious, broad interests, creative,
  original, imaginative, untraditional
• Conscientious organized, reliable, hard-working,
  self-disciplined, honest, clean
• Extravert sociable, active, talkative,
  optimistic, fun-loving affectionate
• Agreeable good-natured, trusting, helpful,
  forgiving, gullible, straightforward
• Neurotic worries, nervous, emotional, insecure,
  inadequate, hypochondriac

9
The OCC-Model

• Emotions as valenced reactions to
• The desirability of events with respect to goals
• The praiseworthiness of other agents (users) with
  respect to standards
• And the appealingness of objects with respect to
  attitudes

10
Global Structure of Emotion Types
VALENCED REACTION TO
ASPECTS OF OBJECTS
CONSEQUENCES OF EVENTS
ACTIONS OF AGENTS
pleased, displeased etc.
approving, disapproving etc.
FOCUSING ON
FOCUSING ON
CONSEQUENCES FOR SELF
CONSEQUENCES FOR OTHER
liking, disliking etc.
DESIRABLE FOR OTHER
UNDESIRABLE FOR OTHER
PROSPECTS IRRELEVANT
PROSPECTS RELEVANT
SELF AGENT
OTHER AGENT
hope fear
CONFIRMED
DISCONFIRMED
satisfaction fears-confirmed
relief disappointment
PROSPECT-BASED
11
Classification of Operationalized Models

• Communication-driven
• Models
• Selection of communicative actions and emotions
  based on an effect to be achieved on the user

• Appraisal-driven Models
• Simulation of dependencies between emotions and
  cognitive processes

12
Examples of Communication-driven Models

• André et al., Lester et al. Speech acts drive
  the selection and sequencing of emotive
  behaviors.
• Ball/Breese Convey empathy by expression styles
  that reflect the emotional state of the user
• Poggi/Pelachaud Use of facial expressions to
  express affect

13
Examples of Appraisal-Driven Models

• Implementations of the OCC-Modell
• Elliott The Affective Reasoner
• Reilly Believable Social and Emotional Agents
• Gratch EMiLE - Plan-based model of emotion
  appraisal
• Cognitive Architectures Sloman et al.
• Simulations
• Blumberg et al. Learning and Emotions
• El-Nasr et al. FLAME Fuzzy Logic Adaptive
  Model of Emotions

14
Emile Plan-based Model of Emotion Appraisal

• Intensity Variables
• Probability with which a goal can be achieved.
• Importance of a goal
• Examples
• IntensityHope(goal) Import(goal) P(goal)
• IntensityJoy(goal) Import(goal)
• IntensityFear(goal) Import(goal)1-P(goal)
• IntensityDistress(goal) Import(goal)
• IntensityAnger(goal) Import(goal)P(threat)

Literatur J. Gratch Emile Marshalling passions
in training and education. Proc. of the 4th
International Conference on Autonomous Agents,
Barcelona, Spain, 2000.
15
Dialogue between two Emile-Agents
16
Emotional Agents for Training

• Model communicative function of emotion
• Convey information about agents internal state
• Goal priorities
• Situation assessment
• Action tendencies
• Create drama and tension
• Demands motivational and
• Behavioral consistency

17
Cognitive Appraisal Theory (Lazarus, OCC, Frijda)
Goals, Beliefs
Environment
Emotion
Coping Behavior

• Give agent consistent internal state
• Assess state vis-à-vis the world
• Convey the assessment

18
Sources of Emotion Plan-based appraisal
Émile, Gratch2000
Troops_helping
TROOPS LEAVE
Anger
Troops_helping
U 40
Treat Child
Pr 0.1
fear
hope
Child Healthy
Coping Behavior
U 60
Goal
Pr 0.1
Gratch Marsella Tears and Fears, Autonomous
Agents 2001
19
Relationship between Personality and Emotions

• Problem Unified model that could be directly
  implemented seems to be missing?
• Basic Approaches
• Map emotions to behaviors in a personality-specifi
  c ways (e.g. see Bates 1994)
• Treat personality as a variable that determines
  the intensity of a certain emotion (e.g. see
  Allen 2000)

20
Relationship between Personality and Emotions I
positive influence, - negative influence
Source Allen, DFKI, 2000

21
Relationship between Personality and Emotions II
positive influence, - negative influence
Source Allen, DFKI, 2000
22
Presence
23
Affective Processing Examples

• (Un)Desirable Situation
• User continuously clicks the agents body. This
  leads to the emotional state distressed, because
  the event is undesirable for the agent.
• intensity (value of undesirability of lteventgt)
  x
• degree of neuroticism
• Praiseworthy Situation
• The agent gives additional information to an
  request. The user praises this behavior. The
  approving of the agents own action, leads to the
  emotional state pride.
• intensity (value of praiseworthiness) x
• degree of extraversion

24
Haptek Inc.
Emotional Behavior
Facial Expression Body
Language Action Variants
Boston Dynamics Inc.
25
How to Convey Personality An Example

• Extrovert Characters
• more expansive gestures
• speak louder
• more powerful phrases
• tend to take initiative in a dialogue
• talkative
• tend to look at conversational partner

• Introvert Characters
• tight gestures
• speak at low voice
• indirect, hesitant speech
• waits for others to take the initiative
• says only the minimum
• tend to avoid looking at conversational partner

26
Expression of Personality

• Talking Posture 1
• cautious, hesitant
• appeal for compliance
• avoids body-gestures

• Talking Posture 2
• active, self-confident
• uses body-gestures

27
Expression of Emotion
Im embarrassed.
Im angry.
Im pleased.
28
Carmens Bright IDEAS (Marsella, Johnson, LaBore)
29
Physical Focus Model Marsella2000

• Analysis of Emotion-predicting motion
• Analysis of movement in clinical settings
  (Freedman)
• Coping strategies (Lazarus)
• Organize behavior around distinct modes
• Bodyfocus inward directed
• Communicative outward directed
• Consistent interpretation of agent
• Internal state behavior
• Emotions relation to environment
• Coordinate different modalities

30
Physical Focus Modes

• Selected based on emotional state
• High distress ? Bodyfocus
• Inner-directed attention
• Increased self-touching
• Gaze aversion (limited perception)
• High anger or joy ? Communicative
• Outward directed attention
• Increased use of communicative gestures
• Directed gaze (unlimited perception)

31
Affective Speech
Source Cahn, MIT Media Lab.
32
Expression of Emotions via Speech

• Anger
• Disgust
• Fear
• Gladness
• Sadness
• Surprise

33
How to Represent the Input to the Surface
Realization Components?

• Virtual Human Markup Language (VHML) combines
• EML Emotion Markup Language
• GML Gesture Markup Language
• SML Speech Markup Language (based on SSML)
• FAML Facial Animation Markup Language
• BAML Body Animation Markup Language
• XHTML eXtensible Hypertext Markup Language
• DMML Dialogue Manager Markup Language

34
Example of a Sentence Annotated with APML

• ltAPMLgt
• ltturn-allocation type"take turn"gt
• ltperformative type"inform" affect"sorry-for"
  certainty"certain"gt
• I'm sorry to tell you that you have been
  diagnosed as suffering from what we call angina
  pectoris,
• lt/performativegt
• ltbelief-relation type"elaboration-object-attrib
  ute"gt which
• ltperformative type"inform" certainty"certain"gt
  
• appears to be ltadjectival type"small"gtmild
  lt/adjectivalgt
• lt/performativegtlt/belief-relationgt
• lt/turn-allocationgt
• lt/APMLgt

35
Examples of Communikative Functions in Greta
36
Hamlet Display Or Not Display

• Context1
• 3-year-old girl would like to make a present to
  her mother und cuts a dress. The mother comes in
  and sees the disaster.
• Oh, a present for me you are a very nice girl!
• But the ribbon is made from my suit that I need
• for my work. If you cut it, I cant wear it
  anymore.
• Regel IF (Feel Ag Anger) AND (Adoptive Ag I)
• AND NOT (Comprehension-I)
• THEN NOT(Display Ag Anger)
• Context2
• The child has already cut a dress.
• Oh, Oh, a present for me you are a very nice
  girl!
• But the ribbon is made from my suit! I already
  told
• you a hundred times not to touch my things.
• Regel IF (Feel Ag Anger) AND (Adoptive Ag I)
• THEN (Display Ag Anger)

Quelle De Carolis et al., Univ. of Bari, 2001.
37
D-Plan
n1 (Discuss (Preparation I present))
Contrast
n2 (Thank Ag I present)
n5 (Reproach Ag I (Cut I suit))
Justification
Elab
n6 (Explain Ag I Negative-Event (Cut I suit))
n3 (Inform Ag I (Seen Ag present))
n9 (Reproach Ag I Consequences(Cut I suit))
n4 (Inform Ag I (Appreciates Ag present))
Justification
n7 (Inform Ag I Use(Ag suit))
n8 (Inform Ag I Effect(Cut I suit))
38
E-Plan
n1 (Discuss (Preparation I present))
Contrast
n2 (Thank Ag I present) (Display Ag Joy)
n5 (Reproach Ag I (Cut I suit))
Justification
Elab
n6 (Explain Ag I Negative-Event (Cut I suit))
n3 (Inform Ag I (Seen Ag present)) (Display Ag
Joy)
n9 (Reproach Ag I Consequences(Cut I suit))
n4 (Inform Ag I (Appreciates Ag
present)) (Display Ag Joy)
Justification
n7 (Inform Ag I Use(Ag suit))
n8 (Inform Ag I Effect(Cut I suit))
39
Resulting APML-Structure

• Suppression of Anger
• lt?xml version"1.0"?gt
• ltAPMLgt
• ltaffective type"joy"gtOh, a ltdeictic
  obj"present" coord"0 30 0"gt
• present for me lt/deicticgtyou are a very nice
  girl! lt/affectivegt
• ltbelief-relation type"contrast"gt But
  lt/belief-relationgtthe ribbon is made
• ltdeictic obj"suit" coord"10 -25 0"gt
• from my suit lt/deicticgt that I need for my
• work. If you lttopic-comment type"comment"gtcut it
  lt/topic-commentgt, I can't wear it anymore.
• lt/APMLgt

40
Resulting APML-Structure

• Expression of Anger
• lt?xml version"1.0"?gt
• ltAPMLgt
• ltaffective type"joy"gtOh, a ltdeictic
  obj"present" coord"0 30 0"gt
• present lt/deicticgt for me you are a very nice
  girl! lt/affectivegt
• ltbelief-relation type"contrast"gtButlt/belief-relat
  iongt
• ltaffective type"anger"gt the ribbon is made from
  my
• ltdeictic obj"suit" coord"15 0 0"gt suit.
  lt/deicticgt
• I already told you a hundred times not to touch
  to my things!lt/affectivegt
• lt/APMLgt

41
Bayesian Networks

• Generation of Behaviors
• Set the values of the variables for personality
  and emotions on the state the agent should
  convey.
• The Bayesian network will then predict a
  probability distribution for possible behaviors.
  
• Recognition of Personality and Emotions (e.g.
  that of the user)
• Values for variables result from observations
• The Bayesian Network provides an estimation of
  the personality and emotions and of an agent.

42
Conditional Probabilities for SpeechVol and
WdsActive
43
Bidirectional Use
Quelle Breese Ball, Microsoft Research, 2000.
44
Bayessches Netzwerk zur Modellierung von
Persönlichkeit und Emotionen
Quelle Breese Ball, Microsoft Research, 2000.
45
Bayesian Network which associates different
expression styles with paraphrases of the concept
Greet

• Table represents the probability with which
  different paraphrases will be perceived as a
  positive greeting.
• Simulation Each style node (e.g.. wdsPositive)
  has a value distribution which depends on the
  agents personality and emotions. A negative
  emotion increases the probability that the agent
  says Oh, you again.
• Recognition For a friendly agent, the utterance
  Oh, you again is associated with a far more
  negative emotion than for an unfriendly agent.

Quelle Breese Ball, Microsoft Research, 2000.
46
Use of Bayesian Networks to Resolve Conflicts in
Greta

• A communicative act is usually conveyed via
  different channels of the face, such as eye
  brows, mouth shape, gaze, head direction and head
  movements
• For instance, sorry-for is conveyed via a
  special movement of the head as well as a special
  form of the eye brows, while certain is
  expressed by a single signal, namely a special
  form of the eye brows.

47
Conflict Resolution

• A special difficulty arises due to the fact that
  several tags may encompass one and the same piece
  of text.
• For instance, the sentence I'm sorry to tell
  you that you have been diagnosed as suffering
  from what we call angina pectoris is tagged for
  times.
• Each of these tags corresponds to a given facial
  expression.
• Some of these expressions may refer to the same
  channel.
• It may even happen that different values are
  assigned to the same channel.
• Sorry-for is expressed by oblique eye brows,
  while certain is conveyed by a small frown.

48
Use of a Bayesian Network for Conflict Resolution
49
Use of Bayesian Networks to Model Emotions

• Advantages
• Handling uncertainty
• Connections between nodes are intuitively
  comprehensible since they represent the
  relationship between cause and effect
• easy to extend
• suitable both for recognition as well as
  simulation
• Problems
• Determination of the initial probabilities
• only suitable for modeling independent matters of
  fact

50
Affective Jewelry
51
Expressive Animations What is already available
and what is missing?

• Expression of affect, but personality hardly
  addressed (especially for the face)
• Body mostly neglected and not synchronized with
  the face
• Commercial animation tools are available, but
• SDK/API often badly documented (e.g. Pulse 3D)
• Usually rely on pre-authored audio

52
Expressive Animations What is already available
and what is missing?

• A lot of manual work is still required from the
  human designer who have to control a large number
  of low-level parameters.
• Promising line of research
• EMOTE computational model of effort and shape
  (Badler et al. 2001)
• Control through the usage of high-level
  parameters that represent qualitative aspects of
  movements

53
Expressive Speech What is missing?

• No interaction between speech-related parameters,
  such as FO contour, and other linguistic
  information like sentence type
• Trade-off between flexibility of acoustic
  modeling and perceived naturalness
• In order to express a large number of emotional
  states with a natural-sounding voice, either
  rule-based techniques need to become more
  natural-sounding or selection-based techniques
  more flexible.
• Need of more appropriate evaluation techniques
• We need to move away from forced-choice tests
  that use abstract emotion words towards tests
  measuring naturalness of an utterance given an
  emotion-defining context.

M. Schröder, Emotional Speech Synthesis A
Review, Proc. Eurosppeech 2001.
54
Cognitive Models What is missing?

• No uniform model of personality, emotion and
  social context
• Models usually start from a given set of input
  parameters, but do not say where they come from
• Dynamic aspects of emotions are hardly addressed

55
Conclusion

• Much work is needed to bring the single pieces of
  the jigsaw together.
• To use affect effectively, it must be used both
  at an appropriate level for the application
  domain, and as an all-encompassing component of
  the system - from graphic design to system
  architecture to application.

56
Supporting Affective Interactions for Real-time
Applications (SAFIRA)

• Overall objective Bring to the software
  community an enabling technology to support
  affective interactions, in particular
• To create a framework to enrich interactions and
  applications with an affective dimension
• To implement a toolkit for affective computing
  combining a set of components addressing
  affective knowledge acquisition, representation,
  reasoning, planning, communication and
  expression
• To verify under which conditions the hypothesis
  that emotion, as well as other affective
  phenomena, contributes to improve rationality and
  general intelligent behavior of the synthetic
  characters.