annotation of emotions in meetings in the AMI project

1
annotation of emotionsin meetingsin theAMI
project

• Roeland Ordelman Dirk Heylen
• Human Media Interaction
• University of Twente
• The Netherlands

2
overview

• on the AMI project 100 hours of audio/video
  recordings of project meetings
• on annotating the emotional state of the
  participants in the meetings
• on emotion annotation in AMI
• on implementing the annotation tool in the NXT
  framework

3
AMI in brief

• European Integrated Project of the IST 6th FWP,
  initiated in January 2004 and involving 15
  partners
• aims at advancing the state-of-the-art in
  important basic technologies such as human-human
  communication modeling, speech recognition,
  computer vision, multimedia indexing and
  retrieval
• produces tools for off-line and on-line browsing
  of multi-modal meeting data, including meeting
  structure analysis and summarizing functions
• makes recorded and annotated multimodal meeting
  data widely available for the European research
  community, thereby contributing to the research
  infrastructure in the field

4
on the AMI meetings

• scenario-based meetings
• participants are asked to carry out a certain
  task and are provided with role-restricted
  information
• design project, involve one particular task
  (remote control design)
• 35 hours (48 meetings)
• real design project meetings on
• real design projects
• such as engineering student projects, brochure or
  poster design, etc. 
• 35 hours
• other real or scenario-based meetings
• cover a wider variety of meeting types, topics,
  behaviours, etc.
• 30 hours

5
on the recording of the meetings

• smart meeting rooms at three different sites
• (TNO, Edinburgh, IDIAP)
• video (side camera, close up)
• audio (mic-array, lapel, headset, manikin)
• whiteboard strokes
• pen

6
ami annotations

• many features of meeting interactions are
  annotated
• speech
• gestures
• dialogue acts
• posture
• emotion
• information is used for (at least) three
  purposes
• primary meta-data source for multi-featured
  browsing of the recorded meetings.
• training recognition algorithms that eventually
  should be able to provide automatically generated
  meta-data.
• evidence on the basis of which theoretical models
  of human multi party interaction can be
  developed.

7
properties of the data

• 100 hours of meeting data
• emotion annotation of every single speaker in
  meeting 4 x 100 hours !
• only a very small proportion can be expected to
  show emotional states in any strong sense
• neutral state will cover much data

8
emotion annotation

• no general agreement on how to annotate or label
  emotional content in a natural database
• a number of emotion annotation or labeling
  schemes have been proposed in the literature
• given the gradations and subtlety of emotions
  occurring in natural data, the labeling of
  emotion using category labels is not
  straightforward

9
emotion annotation in AMI

• Given
• the amount of data to be annotated, and
• the expected gradations and subtlety of emotions
  occurring in meeting data
• a dimensional labeling approach complemented by a
  categorical labeling scheme seemed most
  appropriate in the context of AMI
• the FeelTrace software developed at Queens
  University Belfast (Cowie et al.) is reported to
  produce good quality annotations within a
  reasonable amount of time

10
FeelTrace

• judge the emotional experience of the
  participants of the meetings on two dimensions
  arousal and valence.

11
annotation with FeelTrace in AMI

• survey on meeting specific landmarks
• pilot annotations
• re-implementation of FeelTrace

12
landmarks survey (1)

• (main investigator Vincent Wan, University of
  Sheffield)
• Method
• list of 243 terms describing emotions
• participants (37) had to select twenty emotions
  that they most frequently perceived in their
  meetings
• participants from various companies and with
  various job descriptions, including lecturers,
  researchers, managers, secretaries and students.
• 243 terms were clustered by meaning into groups.
  The most frequently chosen one or two labels were
  shortlisted from each group. Taking some labels
  from each group ensures that there is sufficent
  coverage of the emotion space.

13
landmarks survey (2)

• list of 26 meeting domain specific emotional
  words
• at ease, bored, joking, annoyed, nervous,
  satisfied, frustrated, amused, relaxed,
  interested, cheerful, uninterested, disappointed,
  agreeable, contemplative, encouraging,
  sceptical, friendly, attentive, confused,
  confident, decisive, impatient, concerned,
  serious, curious

14
landmarks survey (3)

• second survey to determine where each of the
  shortlisted labels should appear in FeelTrace
  emotional space
• first presented participants with the five
  labels anger, irritation, sadness, happiness and
  contentment (presented so that participants
  unfamiliar with FeelTrace would get some minimal
  experience in its use).
• emotional words were presented twice the first
  to allow the participant to gain additional
  training and the second to collect data.

15
example results 1
16
example results 2
17
example results 3
18
resulting landmark distribution
19
annotation trials

• we have 15-20 annotators available
• coming weeks first annotation trials
• what do we want to learn
• inter-annotator agreement
• distribution of emotional states in meeting data
  (how much is neutral)
• annotator experience with tool
• effect of using/not using landmarks
• validation of manual, annotation area, etc

20
trial annotation set-up

• set of 4 meetings, each of about 20 minutes, 4
  speakers per meeting
• 10 minute segments (0-10,10-20)
• (opt.) dummy pass with Feeltrace
• offical pass with FeelTrace
• with and without landmarks
• no categorical labeling (yet)

21
implementation in NXT

• Belfast implementation of dimensional approach
  knows some limitations
• no cross-platform support
• cannot easily be tailored to specific needs from
  different stakeholders of annotations (e.g.,
  additional categorical of labeling of longer
  segments is hard in current setup)

22
stakeholders of annotation

• corpus developer
• defines the structure of corpus, maintains the
  data, takes care of a proper data/tool
  distribution (CVS, validation, time)
• corpus annotator
• needs to know as little as possible about
  configuration, installation and version control
  issues, only the annotation process itself is of
  concern for the annotator
• data consumer
• interested in the annotations for analysis and
  may want to configure in detail the annotation
  process (tool functionalities such as redo,
  fastforward, landmarks/no-landmarks, ect)
• tool developer
• creates the tool that serves the needs of the
  other users, keeping technical issues in mind

23
NXT

• Nite XML toolkit
• defines a data storage format that can easily be
  shared across a multitude of annotation and
  analysis tools for the many different aspects of
  a multi-modal corpus.
• the NXT libraries provide many ready-made
  components that facilitate easy development of
  new tools.
• expertise readily available at University of
  Twente

24
draft list of requirements

• easy distribution of data (and tools) to and from
  the annotators (e.g., annotate certain segments
  of a file, CVS functionality, batch
  functionality)
• easy management of multiple annotators, possibly
  working on the same data (e.g., CVS
  functionality)
• validation functionality or possibilities to plug
  this into the process
• as many input formats as possible on as many
  platforms as possible
• customizable landmarks, dimensions (1D, 2D, 3D),
  shortcuts
• categorical labeling options within the tool
• video control (fastforward/backward)
• progress bar
• replay annotation aligned with video
• selection of multiple video signals (close up,
  wide angle)
• color coding of the 2D space (provide a priori
  color feedback instead of feedback with a
  changing color of the cursor)
• easy configuration in general
• ? discuss requirements with AMI/HUMAINE
  researchers

25
roadmap

• starting first trial annotations
• investigate trial results
• discuss re-implementation of FeelTrace in NXT
  with interested parties and create an first
  implementation version
• follow-up trials, FeelTrace-NXT versions
• monitor process and discuss results with
  researchers in the field (AMI, Humaine)