Markup of Multimodal EmotionSensitive Corpora

1
Markup of Multimodal Emotion-Sensitive Corpora

• Berardina Nadja de Carolis, Univ. Bari
• Marc Schröder, DFKI

2
Our BackgroundGeneration of ECA Behavior

• Believable ECA multimodality, affect,
  personality, human-like, etc.
• Two approaches
• Bari APML (Affective Presentation Markup
  Language)
• NECA RRL (Rich Representation Language)

3
Overview

• Markup Schema Definition
• What to Annotate
• What is a Markup Language
• What is XML
• Dealing with complexity of human communication
• Labelling Tools

4
Markup Scheme Definition

• What to annotate? (verbal, non-verbal behavior,
  emotions, context, )
• Units of consideration (utterances/ turns/
  paragraphs/ time units/)
• Descriptive Scheme
• Categorical
• Dimensional
• Functional

5
The Question of Coding

• How to Share and Access Data Annotation?
• Standardization
• Format Readable for Computers (statistics,
  inference, )
• XML-based Markup Languages!

6
What is a Markup-Language

• Dictionary Definition
• a set of symbols and rules for their use when
  doing a markup of a document nomenclature,
  terminology, language - a system of words used in
  a particular discipline
• Computer Domain Examples
• HTML
• LATEX
• RTF

• But
• proprietary language
• not generic
• not standard
• fixed syntax and semantics

lt?XML!gt
7
What is XML

• XML (eXtensible Markup Language) is
• a language for creating markup languages that
  describe data thus, a meta language used to
  define new markup languages (domain- or
  industry-specific).
• Simple, open standard, extensible, ready to use,
  
• XML has a lot of friends that makes computation
  easy (XSL, DOM, SAX, ) and interoperable (RDF,
  RDF Schema, OWL, )

No No No! I said turn left!!!
8
Dealing with complexity of human communication

• Communication at different non-exclusive
  abstraction levels
• Mental state (Beliefs, Goals, Intentions,
  Affective State, )
• Discourse (Communicative Goals, Focus, Relations)
• Meanings (Performative, Affect, Turn-Taking,
  Emphasis)
• Signals (Physical Communicative Expressions)
• Defining Different Markup Layers
• Separate levels of meaning into different layers
• Define a Markup Language for each layer
• Define Interaction between layers
• Develop Easy-to-use tools

9
Labelling toolsText-centered annotation MMAX
http//mmax.eml-research.de
10
Labelling toolsVideo-centered annotation ANVIL
http//www.dfki.de/kipp/anvil
11
Feedback into Generation Systems

• Annotated Corpora can be used for improving ECA
  systems
• extracting knowledge
• markup language for specifying ECA behaviors

12
Overview of the session

• Introduction to XML structure
• Demos
• Use of APML in an ECA system
• XML markup in MARY Text-to-speech
• Hands-on The ANVIL tool and the resulting XML
  markup