Jonathan Gratch

1
The Architectural Role of Emotions in Cognitive
Systems

• Jonathan Gratch
• USC Institute for Creative Technologies
• Joint work with
• Stacy Marsella
• USC Information Sciences Institute

2
Outline
Outline

• Emotions are adaptive
• Can inform cognitive system design
• Ground in implemented cognitive system
• Mission Rehearsal Exercise system
• Cognitive Appraisal Theory
• Illustrate impact on architecture design
• General implications for cognitive systems

3
Outline
Adaptive Role of Emotions

• Revolutionary progress in emotion research
• Neurophysiology of emotion (Damasio, LeDoux)
• Appraisal theories (Frijda, Lazarus, Scherer)
• Emotions appear adaptive (in moderation?)
• Decision-making - Focus of attention
• Learning - Social relationships
• Belief formation - Communication
• Growing interest in emotional systems
• with focus on modeling human behavior
• HCI (non-verbal recognition and generation)
• User and human behavior modeling
• Believability/Entertainment

4
Outline
Architectural Perspective

• Can inform intelligent behavior in general
• Motivate behavior
• Balancing competing goals
• Balancing reaction and deliberation
• Disambiguating stimuli in light of existing
  beliefs and commitments
• Abstract and formalize as information processing
• Not new idea
• Simon(1967), OatleyJohson-Laird(1987), Sloman
• Revisit in light of new findings
• In intelligent systems
• In theories of emotion

5
Grounding Virtual Humans

• Face-to-face interaction
• Verbal non-verbal behavior
• Swartout, Gratch, Hill, Hovy, Johnson,
• Marsella, Narayanan, Rickel, Traum,

Marsella, Johnson Labore
6
Mission Rehearsal Exercise

• Social Training Simulation
• Explore high-stakes social interactions in safety
  of VR

7
Mission Rehearsal Exercise

• Team decision-making in crisis situations
• Non-scripted real-time interactions
• Planning, replanning, and plan execution
• Teamwork, distributed authority and
  responsibility
• Collaborative, mixed initiative dialogue
• Multi-party conversations
• Verbal and non-verbal communication
• Emotionally-biased behavior

8
Mission Rehearsal Exercise

• Assumptions/Limitations
• Tightly focused task-related dialogue
• Near-expert decision makers
• Stylized vocabulary (military speak)
• Stylized virtual environment

9
MRE Leadership Training
10
Voice Input
World Simulator
Speech Recognition (HTK)
Semantic Parser
Projection System
Communication Bus
Animation System
BDI
Vega
Haptek
Audio (Protools)
Motion/ Gesture Scheduler (Beat)
Speakers (10.2)
Text to Speech (Festival)
11
Cognitive Representation
Future
Present
Past
Eagle 1-6 Assist Desire LT Belief False
Child HealthyFalse
Accident Intend False Blame unresolved
Assist Eagle 1-6False
Get Medevac ResponsibilityLT Intend True
Medevac AvailableTrue
Child-Healthy Desire SGT Belief
False Probability 75
Soars Working Memory
Planning
Perception
Dialogue
Action
Soar operators
12
Cognitive Representation
Future Plans
Present
Past Events
Eagle 1-6 Assist Desire LT(50) Belief False
Child HealthyFalse
Accident Intend False Blame unresolved
Assist Eagle 1-6False
Get Medevac ResponsibilityLT Intend True
Medevac AvailableTrue
Child-Healthy Desire SGT(80) Belief
False Probability 75

• Causal Interpretation
• Combines decision-theoretic plans with models of
  belief and intention
• Uniform representation of past, present, future
• Agent centric subjective view

13
Architectural Role of Emotion

• Began with view emotion as veneer
• Ended up as central organizing construct
• Initial problem
• how to convey emotion in interactive setting?
• Built mechanism to infer plausible emotions
• In response to simulation events
• In response to user interventions
• But discovered resolved architectural issues
• Coherence is more than skin deep
• Build it and they will come

14
How to convey emotion

• Cognitive Appraisal Theory
• Influential and well-established theory
• Arnold, Frijda Lazarus Ortony, Clore
  Collins Scherer Smith
• Emphasizes tight coupling between
• Emotion
• Cognition
• Motivation

15
Cognitive Appraisal Theory
Goals, Beliefs
External Events
Appraisal
Emotion
Coping
Smith and Lazarus cognitive-motivational-emotive
system
16
Appraisal
Goals, Beliefs
External Events
Appraisal

• Appraisal Situation assessment
• Compare beliefs, desires and intentions
• with
• external circumstances

17
Appraisal

• Characterize via appraisal variables
• Desirability
• Likelihood
• Urgency
• Unexpectedness
• Causal attribution (causality, agency,
  blame/credit)
• Coping potential (controllability, adaptability)
• Superset of criteria considered by cog systems
• Decision theory desirability, likelihood
• Scheduling desirability, urgency

18
Coping Strategies

• Coping Response strategy
• Characterized by ontology of coping strategies

Emotion
Goals, Beliefs
External Events
Coping
Problem-focused
Emotion-focused
19
Coping Strategies

• Problem-focused (act on the world)
• Action execution
• Planning
• Seek instrumental social support
• Analogous to
• Deliberative or reactive problem solving
• Team negotiation

20
Coping Strategies

• Emotion-focused (act on belief)
• Denial
• Find silver lining
• Shift blame
• Distancing
• Not typically considered by cog systems systems
• More than a decision (e.g. abandon current plan)
• Provides self-justification for why
• Related to motivational / explanatory coherence
• Leads to persistent change in behavior

21
Modeling Appraisal and Coping
Future
Past
Soars Working Memory
Planning
Perception
Dialogue
Action
Soar Operators
22
Modeling Appraisal and Coping

• Appraisal as plan-evaluation
• Causal interpretation mediates agent-environment
  relationship
• Define appraisal variables in terms of features
  of interpretation
• Fast, reactive, parallel
• Coping as generalized plan critics
• Map to operators that change interpretation
• Problem-focused ? execute step, add plan step
• Emotion-focused
• Denial ? Change belief
• Find silver lining ? Change utilities
• Shift blame ? Change causal attribution
• ? Dialogue moves
• Distancing ? Drop goal / intention

23
Émile Architectural Manifestation
The Emotional Octopus
24
Eagle 1-6 Assist Desire LT Satisfied False
Child HealthyFalse
Assist Eagle 1-6False
Accident Blame unresolved
Child-Healthy Desire SGT Satisfied False
Perspective Self (Sgt) Desirability
-80 Likelihood 100 Blame/Credit unresolved
Distress 80
Sgts Appraisal of Accident from his perspective
25
Eagle 1-6 Assist Desire LT Satisfied False
Child HealthyFalse
Assist Eagle 1-6False
Accident Blame unresolved
Child-Healthy Desire SGT Satisfied False
Perspective Lieutenant Desirability
-80 Certainty 100 Blame/Credit unresolved
Distress 80
Distress 80
Sgts Appraisal of Accident from Lieutenants
Perspective
26
Eagle 1-6 Assist Desire LT Satisfied False
Child HealthyFalse
Assist Eagle 1-6False
Accident Blame unresolved
Child-Healthy Desire SGT Satisfied False
Distress 80
27
Eagle 1-6 Assist Desire LT Satisfied False
Child HealthyFalse
Assist Eagle 1-6False
Accident Blame unresolved
Child-Healthy Desire SGT Satisfied False
Sgts Own Perspective
Distress 80
28
Eagle 1-6 Assist Desire LT Satisfied False
Child HealthyFalse
Assist Eagle 1-6False
Accident Blame unresolved
Child-Healthy Desire SGT Satisfied
False Probability 75?
Get Medevac ResponsibilityLT
Make Amends
Distress 80
Distress 80
Problem-Focused Coping Form intention to help
Boy
29
Eagle 1-6 Assist Desire LT Satisfied False
Child HealthyFalse
Assist Eagle 1-6False
Accident Blame MOM
Child-Healthy Desire SGT Satisfied
False Probability 75?
Get Medevac ResponsibilityLT
Shift Blame
Make Amends
Distress 80
Distress 80
Emotion-Focused Coping Blame Mother
30
Eagle 1-6 Assist Desire LT Satisfied False
Child HealthyFalse
Assist Eagle 1-6False
Accident Blame unresolved
Child-Healthy Desire SGT Satisfied
False Probability 75?
Get Medevac ResponsibilityLT
Shift Blame
Make Amends
Distress 80
Distress 80
Personality
31
MRE Leadership Training
32
Architectural Implications

• Emotion as central control construct
• Planning (inform course-of-action selection)
• NLU (inform reference resolution)
• Dialogue (prompt dialogue initiative)
• NLG (biases sentence generation strategies)
• Non-verbal expression

33
General Implications

• Emotion and Reflection
• Appraisal is form of self-reflection / focus of
  attention
• Emotion as decision-making
• Generalization of decision-theory
• More to the world than probabilities and
  utilities
• Emotion and plausible reasoning
• Emotion-focused coping motivate
  preference/beliefs
• Attempt to construct coherent motivational
  explanation
• Non-rational but adaptive?

34
General Implications

• Emotion and Learning
• Focus learning on emotionally salient events
• Appraisal variables as features / case indexes

35
Conclusion

• Emotion is form of information processing
• Arguable adaptive
• Juggling competing goals and commitments
• Focusing cognitive resources
• Enforcing coherence
• Arguable unexplored by cognitive systems

36
(No Transcript)
37
Dialogue Example Sgts Behavior

• Focus1
• Lt U9 Secure a landing zone
• Committed(lt,7), 7 authorized, Obligation(sgt,U9)
• Sgt U10 First we should secure the assembly
  area
• Disparaged(sgt, 7), endorsed(sgt,2)
• Lt U11Secure the area
• Committed(lt,2), 2 authorized, Obligation(sgt,U11)
  
• Sgt U12 Yes sir
• Committed(sgt,2), Push(2)
• Goal7Announce(2,1sldr,2sldr,3sldr,4sldr)
• Goal8 Start-conversation(sgt, 1sldr,2sldr,,2)
• Goal8 ? Sgt U13 Squad leaders listen up!
• Goal7 ? Sgt U14 I want 360 degree security
• Push(3)
• Goal9authorize 3
• Goal9 ? Sgt u151st squad take 12-4
• Committed(sgt,3), 3 authorized
• Pop(3), Push(4)
• Goal10 authorize 4

1
Decomposition
Area Secure
Squads in area
ALt, RSgt
ALt, RSgt
2
7
Decomposition
3
4
ASgt, R1sldr
ASgt, R2sldr
5
6
ASgt, R3sldr
ASgt, R4sldr
38
More than a theory of emotion

• Appraisal as a mediating variable
• Direct mappings (e.g. Hayes-Roth personality
  model)
• Indirect mappings
• Direct More links, No insight on how to map
• Indirect more constrained. More modular

World state
Beliefs
Behavior
Desires
Personality
World state
Appraisal Variables
Beliefs
Behavior
Desires
Personality
39
Mediating Variable

• Appraisal Mediates Personality
• Personality Variable ? Appraisal Variables ?
  Behavior
• e.g Extroversion? ? Control? ? Hope?
• Penley Tomaka (2002)
• Appraisal Mediates Culture
• Culture Variable ? Appraisal Variables ? Behavior
• e.g. Uncertainty avoidance? ? Threat? ? Fear?
• Kupperbusch et al

40
Mediating variables

• Coping mediated by appraisal
• Undesirable Controllable
• ? Distress
• ? Problem directed coping
• Undesirable Uncontrollable
• ? Distress
• ? Emotion directed coping