Title: GENERATING AFFECTIVE CHARACTERS FOR ASSISTIVE APPLICATIONS
1GENERATING AFFECTIVE CHARACTERS FOR ASSISTIVE
APPLICATIONS
- Diana Arellano, Isaac Lera, Javier Varona and
Francisco J. Perales - Computer Graphics, Vision and Artificial
Intelligence Group - Universitat de les Illes Balears (UIB).
- Palma de Mallorca, Spain
- August 2009
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
2AGENDA
1.- Motivation What are we looking for? 2.-
Objective What did we do? 3.- Breaking into
pieces 4.- Results 5.- Applications 6.-
Conclusions
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
31.-What were we looking for?
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
41.- What were we looking for?
Individuals
Unique
Different
Distinguishable
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
51.- What were we looking for?
Realism
ORIGINAL PICTURE
RENDER
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
61.- What were we looking for?
More Humans
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
71.- What were we looking for?
Characters that
- Manifest emotional states
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
82.-What was the objective?
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
92.- Objective
World
represented by
Emotions
generates
Emotional States
Semantic Knowdlege
produce
Personality
influenced by
manifested by
Facial Expressions
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
102.- Objective
- We explore
- The role of new technologies and theories that
explore human affect. - How they can be used by persons in everyday life.
-
- The creation of virtual characters for specific
applications developed for physically, or
mentally, disable people.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
113.-Breaking into pieces
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
123.- Breaking into pieces
What surrounds and occurs to the character
EVENTS
World
Action
Time
People
Place
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
133.- Breaking into pieces
What is inside the character
World
Goals
Agent Admiration
Preferences
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
143.- Breaking into pieces
World
represented by
Semantic Knowdlege
ONTOLOGIES
?
?
?
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
153.- Breaking into pieces
ONTOLOGY
Formal representation of a set of concepts within
a domain and the relationships between those
concepts.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
163.- Breaking into pieces
World
represented by
Emotions
generates
Semantic Knowdlege
Personality
influenced by
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
173.- Breaking into pieces
Five Factor Model (OCEAN)
Personality
YEARS
- Opennes to Experience
Conscientiousness
Extraversion
Agreeableness
Neuroticism
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
183.- Breaking into pieces
Emotions
MINUTES
Ekman OCC Model
- Ekman Happiness, Sadness, Disgust, Anger,
Fear, Surprise.
- OCC Model (Ortony, Clore and Collins)
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
193.- Breaking into pieces
World
represented by
Emotions
generates
Emotional States
Semantic Knowdlege
produce
Personality
influenced by
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
203.- Breaking into pieces
PAD Space Pleasure-Arousal-Dominance
(PAD) Exuberant (-P-A-D) Bored
Proposed by Albert Mehrabian
D
(P-AD) Relaxed (-PA-D) Anxious
-A
-P
(PA-D) Dependent (-P-AD) Disdainful
P
A
(P-A-D) Docile (-PAD) Hostile
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
213.- Breaking into pieces
How did we do it?
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
223.- Breaking into pieces Affective Model
Implementing an Affective Model
D
ES Emotional State
ES
Intensity of Emotional State
-A
,if
-P
Slightly
Moderate
P
A
-D
Highly
PAD Space
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
233.- Breaking into pieces Affective Model
Implementing an Affective Model
D
DES Default Emotional State
DES
-A
-P
P
A
-D
PAD Space
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
243.- Breaking into pieces Affective Model
Implementing an Affective Model
D
DES Default Emotional State
DES
Ei Emotions
E1
E2
Emotion P A D Octant
Anger -0.51 0.59 0.25 Hostile
Disgust -0.4 -0.2 0.1 Disdainful
Disappoint. -0.3 -0.4 -0.4 Sadness
Sadness -0.4 -0.2 -0.5 Sadness
Fear -0.64 0.6 -0.43 Anxious
Relief 0.2 -0.3 0.4 Relaxed
-A
-P
E3
P
A
-D
PAD Space
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
253.- Breaking into pieces Affective Model
Implementing an Affective Model
D
DES Default Emotional State
DES
Ei Emotions
E1
EC Emotional Center
E2
EC
-A
-P
EC Center of Mass(Ei)
E3
P
A
-D
PAD Space
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
263.- Breaking into pieces Affective Model
Implementing an Affective Model
D
DES Default Emotional State
DES
Ei Emotions
E1
E1
EC Emotional Center
ES
E2
EC
-A
E2
-P
At time 0 in the process
E3
E3
ES Emotional State
ES displacement of DES due to EC
P
A
-D
PAD Space
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
273.- Breaking into pieces Affective Model
Implementing an Affective Model
D
DES Default Emotional State
Ei Emotions
E1
E1
ES(t)
EC Emotional Center
E2
ES(t1)
ES
EC
-A
E2
-P
At time t1
E3
ES(t) Actual Emotional State (en t)
ES(t1) New Emotional State (en t1)
P
A
-D
PAD Space
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
283.- Breaking into pieces Affective Model
Implementing an Affective Model
D
DES Default Emotional State
DES
Ei Emotions
E1
ESdec
ESdec Decayed Emotional State
-A
E2
-P
EE(t1)
Decay
E3
ESdec Center of Mass(DES, EE(t1))
P
A
-D
PAD Space
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
293.- Breaking into pieces
World
represented by
Emotions
generates
Emotional States
Semantic Knowdlege
produce
Personality
influenced by
manifested by
Facial Expressions
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
303.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Emotional State
Emotions
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
313.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Happiness, Sadness, Anger, Disgust, Fear, Surprise
Universal
Emotions
Hate, Love, Pity, Disappointment
Intermediate
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
323.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Expressions of Universal Emotions
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
333.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Expressions of Intermediate Emotions
Universal
Intermediate
Universal
Intermediate
Universal
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
343.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
GENERATION OF EXPRESSIONS
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
353.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
1. MPEG-4 Standard
FDPs Facial Definitions Parameters
4.1 Right corner of left eyebrow
4.2 Left corner of right eyebrow
4.3 Uppermost point of the left eyebrow
4.4 Uppermost point of the right eyebrow
4.5 Left corner of left eyebrow
Face
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
363.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
1. MPEG-4 Standard
FAPs Facial Animation Parameters
3 Open_jaw
4 lower t midlip
5 raise b midlip
6 stretch l cornerlip
7 stretch r cornerlip
FAP 5 500
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
373.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
2. Whissell Wheel
Activation ( )
Evaluation ( )
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
383.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Case 1 Two universal emotions involved
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
393.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Case 1 Two universal emotions involved
1.- The range of a FAP is
FAP 5 raise bottom midlip
FAP 5 500
FAP 5 0
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
403.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Case 1 Two universal emotions involved
2.- Activation values according to Whissell Wheel
3.- Angular distance between emotions
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
413.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Surprise
Joy
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
423.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Surprise
Happiness
Admiration
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
433.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Sadness
Fear
Pity
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
443.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Case 2 One universal emotion involved
a) The range of is subrange of the
universal emotion.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
453.- Breaking into pieces Generation of Facial
Expressions
Visualizing Emotional States
Happiness
Liking
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
464.-Which were the results?
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
474.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
Objective Demonstrate the coherence of the
affective results given a definition of the
character and their environment.
I. Story with 5 relevant EVENTS
1.- Breakfast oats in the kitchen. 2.- Getting
email with rejection for conference 3.- Arguing
with best friend 4.- Reconcilitation with best
friend 5.- Dinner with flatmates at home
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
484.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
II. 2 different PERSONALITIES were defined
Personality 1 (P1) Very NEUROTIC (N
0.99) Very UNFRIENDLY (A -
0.99). Personality 2 (P2) Very EXTROVERTED (E
0.99) Very FRIENDLY (A 0.99)
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
494.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
III. Event Dinner at 2100 with flatmates at
home
what
who
when
where
Configurations describe PREFERENCES, GOALS and
ADMIRATION for other AGENTS.
- Configuration 1 (C1)
- (a) Living room at home Good (0.7), Indifferent
(0.3) - (b) 2100 Good (0.9), Indifferent (0.1)
- (c) Flatmates Positive (0.8), Indifferent (0.2)
- (d) Have dinner with flatmates Goal Des(0.7)
- Satisfactory (0.7)
- Indifferent (0.2)
- Not Satisfactory (0.1)
- Configuration 2 (C2)
- (a) Living room at home Bad (0.7), Indifferent
(0.3) - (b) 2100 Bad (0.9), Indifferent (0.1)
- (c) Flatmates Negative (0.8), Indifferent (0.2)
- (d) Have dinner with flatmates Not
Satisfactory (0.7) - Indifferent (0.2)
- Satisfactory (0.1)
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
504.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
IV. Triggered Emotions using EMOTIONAL
CATEGORIZATION and FUZZY RULES.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
514.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
V. Triggered Emotional States
P1 NEUROTIC and UNFRIENDLY C1 Positive feelings
P2 EXTROVERTED and FRIENDLY C2 Negative feeling
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
524.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VI. Facial Expressions for Emotional States
Moderate Exuberant
Slightly Disdainful
Fully Exuberant
Slightly Exuberant.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
534.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VI. Another example Set P1-C1
A -0.99 (hard-headed, skeptical, competitive,
proud, good to be leader) N 0.99 (negative
reactions, prone to worry, anxiety)
Disdainful (Default Emotional State)
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
544.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VI. Another example Set P1-C1
Evento 1 Breakfast oats in the kitchen (She
hates oats)
Hostile 0.3
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
554.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VI. Another example Set P1-C1
Decay
Disdainful 0.42
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
564.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VI. Another example Set P1-C1
Event 2 She loves researching. Got a rejection
from a very important conference.
Bored 0.26
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
574.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VI. Another example Set P1-C1
Event 3 She argues with her best friend and she
is really concerned
Bored 0.77
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
584.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VI. Another example Set P1-C1
Decay
Bored 0.61
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
594.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VI. Another example Set P1-C1
Event 4 She makes up with her best friend.
Relaxed 0.08
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
604.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VI. Another example Set P1-C1
Event 5 Dinner with friends in her flat.
Exuberant 0.77
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
614.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VI. Another example Set P1-C1
All Events with Configuration 1, felt with
Personality 1
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
624.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VII. Experimentation
- Evaluation set 20 animations generated using the
MPEG-4 standard. -
- Each animation shows the transition between
previous emotional state to the actual emotional
state produced by its event categorized as P1-C1
or P2-C1, with P1-C2 or P2-C2.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
634.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VII. Experimentation
- Participants 21 persons (4 women and 17 men)
between 20 and 41 years old, different academic
backgrounds. - Procedure show 3 animations per event the
correct one and two incorrect ones, randomly
ordered. - Read the event, the personality, and the
emotional state of the character after the
occurrence of the event. - Observed the three animations, twice.
- Marked in the questionary the animation (A1, A2,
A3) they considered more appropriate to the
situation.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
644.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
Event Dinner with friends in her flat (she does
not like having guests).
- P1 Neurotic and Unfriendly.
Disdainful -gt Disgust 0.115.
Correct.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
654.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
VIII. Results
Percentage of persons that correctly associate
situation - emotional state - facial expression.
P1-C1 P2-C1 P1-C2 P2-C2
1.- Breakfast oats in the kitchen () 72 29 62 95
2.- Getting email with rejection for conference () 71 52 71 76
3.- Arguing with best friend () 81 81 48 67
4.- Reconcilitation with best friend () 76 85 76 85
5.- Dinner with flatmates at home () 85 29 95 62
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
664.- Evaluation and Results
Computational Model Evaluation Semantic and
Affective Model
IX. Conclusion
- More EXPRESSIVENESS on faces with strong
personalities. - Having 3 variables in
consideration (event, personality and emotional
state) results too confusing for people to
evaluate the correlation among them.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
674.- Evaluation and Results
Subjetive Evaluation Survey applied to 75
Computer Science students between 18 and 40 years
old.
1.- Which basic emotion do you recognize in the
expression?
- 86 of the expressions CORRECTLY recognized.
- X Fear and surprise were confused. Disgust was
not easily recognized.
Easily Recognized Hardly Recognized
Happiness (93 ) Reproach (54 )
Sadness (87 ) Fear (52 )
Admiration (86 ) Pity (52 )
Anger (84 ) Disappointment (38 )
Satisfaction (84 )
Neutral (80 )
Surprise (70 )
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
684.- Evaluation and Results
2.- Which emotional state do you recognize in the
expression? (IMPORTANT They were grouped by
Dominance Anxious-Hostile,
Bored-Disdainful, Exuberant-Dependent,
Relaxed-Docile)
- 73 of the expressions CORRECTLY associated.
- Surprise was associated with Anxious-Hostile
state. - X Positive emotions were easier to associate.
Negative were not.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
694.- Evaluation and Results
3.- Which emotional state do you recognize in the
video? (IMPORTANT They were grouped by
Dominance Anxious-Hostile,
Bored-Disdainful, Exuberant-Dependent,
Relaxed-Docile)
Emotion Recognized Emotional State
Sadness Bored-Disdainful 93
Happiness Exuberant-Dependent 88
Anger Anxious-Hostile 85
Disgust Bored-Disdainful 61
Fear Exuberant-Dependent 57
Surprise Exuberant-Dependent 53
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
704.- Evaluation and Results
Objective Evaluation Automatic recognizer of
facial expressions developed in collaboration
with GIGA team from the University of Zaragoza
(INEVAI 3d project).
- 82 of the expressions CORRECTLY recognized.
- Comparing with subjective evaluation results
were similar most of the times. - X Hope, pity and reproach Aversion.
- X Fear confused with surprise.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
715.-What is this useful for?
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
725.- Applications
1.- Tangible Interfaces for disable and elderly
users
We propose the development of a tangible avatar
in charge of the elderlys assistance, with
tele-assistance functionalities in chronic cases.
- Reaction capability, facing events and
environment changes. - Planning capability and decision making, to
carry out the tasks according to one or more
objectives. - Efficiency in decision making and in carrying
out tasks. - Interaction capability and communication with
other agents. - Capability to adapt to other environments.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
735.- Applications
1.- Tangible Interfaces for disable and elderly
users
Advantages
- Reduces new technologies rejection.
- Facilitates the use of the application by an
elderly person. Feeling that someone else is in
charge of the system. - Increments empathy with situations and feeling
of the user.
Disadvantages
X Too much realism Uncanny valley. X Exhaustive
evaluation of the visual appearance and behavior
of the avatar before implementing it in a real
application.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
745.- Applications
1.- Tangible Interfaces for disable and elderly
users
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
755.- Applications
2.- Virtual trainer for developing social
abilities
We propose the development of a virtual character
that facilitates the task of helping people to
express, or suppress, the expression of their
emotions.
Advantages
- Virtual model used as a base to show how the
face is moved and changed when expressing certain
emotions. - Framework with a set of daily events to train
the patient and help him to express the felt
emotion. - Improve communication abilities.
- Assistive tool in therapy to increase emotion
recognition with people with autism or Asperger
syndrome.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
765.- Applications
2.- Virtual trainer for developing social
abilities
Disadvantages
X People with severe communication and language
problems may not be able to recognize any emotion
at all. X Exhaustive research on the appearance
of the avatar to make it usable by the patient.
X Learning curve can be extremely slow.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
776.-Which are the conclusions?
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
786.- Conclusions
- Coherent elicitation and transition of emotional
states according to certain personality traits
and events. - Generation of recognizable intermediate emotions
using basic emotions. - Satisfactory recognition of emotional states in
facial expressions. Especially those in videos!!! - Affective avatars can be of great use as therapy
for people with communication problems. MUSIC can
help in the engagement of the user with the
application.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
796.- Conclusions
Lack of context, voice, and movement (static
images) make harder the recognition. Positive
emotions are hard to differentiate among them.
Problem Duchenne Smile. Expressions for
extroverted characters need to be more
exaggerated. A refinement of evaluation of the
computational model is needed. Less variables to
take into account, and let subject assess the
resulting emotional value.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
80Whats for the future?
81Current and future work
- Head movement, eye movement, blinking REALISM!
- Use of Geneva Wheel to validate emotion
recognition. - Implementation of our own developed ontology
that defines the character and their environment,
through an Affective Avatar/ Virtual Tutor. - Application of the computational model in a tool
that helps to the communication skills
improvement of people with autism or Asperger
syndrome.
EMOTIONS MACHINES Workshop Geneva,
Switzerland.
82GENERATING AFFECTIVE CHARACTERS FOR ASSISTIVE
APPLICATIONS
THANK YOU QUESTIONS??
- Diana Arellano, Isaac Lera, Javier Varona and
Francisco J. Perales - Unitat de Gràfics i Visió per Ordinador, i
Intelligència Artificial - Universitat de les Illes Balears (UIB).
- Palma de Mallorca, Spain
- August, 2009
EMOTIONS MACHINES Workshop Geneva,
Switzerland.