Semantic annotation framework Part 2: Dialogue acts ISOTC37SC4 N442 rev00

1
Semantic annotation framework Part 2 Dialogue
actsISO/TC37/SC4 N442 rev00

• Harry Bunt
• Tilburg University
• ISO TC 37/SC 4 meeting Marrakech, May 25, 2008

2
Purpose and justification

• Dialogue acts are widely used in studies of
  dialogue phenomena, in dialogue annotation, and
  in the design of dialogue systems.
• Dialogue acts are particularly useful for
• describing functional and intentional aspects of
  the dialogue utterance meaning
• the design of dialogue management systems.

3
Dialogue acts

• Well-known examples of communicative functions
  (core dialogue acts)
• question
• WH-question
• YN-question
• check/verification
• statement/inform
• answer (WH-answer. YN-answer)
• confirmation, disconfirmation
• request
• instruct
• promise
• acknowledgement
• greeting

4
Purpose and justification (2)

• Alternative dialogue act schemas TRAINS, Map
  Task,
• Verbmobil, DAMSL, SWBD-DAMSL, COCONUT,...
• with different
• underlying approach to dialogue modelling
• definitions of basic concepts
• level of granularity
• and mutually inconsistent terminology
• Particularly unsatisfactory
• Lack of solid foundations of definitions and
  multidimensionality
• Lack of interoperability

5
ISO approach

• Preparatory studies in TDG 3 in a joint effort
  with eContent project LIRICS.
• Focus How to best support the annotation of
  dialogues with dialogue act information in an
  empirically and theoretically well-founded way.
• Outcome
• Design of a preliminary set of data categories
  for multidimensional dialogue act annotation,
  based on DAMSL, DIT, and other schemas, tested
  for coverage and usability by annotators and
  endorsed by ISO TC37/SC4/TDG3.
• Recommendation to set up an ISO project based on
  1 as part the Semantic Annotation Framework
  project.

6
Summary

• Main points of project outlined in ISO/TC37/SC4
  N442 (rev00)
• Aim to
• Provide more solid foundations for
  multidimensionality of DA tag sets
• Design consistent truly semantic definitions of
  core dialogue acts
• Develop agreed definitions in the form of ISO
  12620 data categories and enter in ISO registry
• Define annotation language with abstract syntax,
  concrete XML-based syntax, and semantics
  compliant with LAF

7
Theoretical foundations of DA annotation concepts

• Information-state change approach to dialogue
  semantics the meaning of an utterance in
  dialogue is the way in which the information
  state of a listener is changed by understanding
  the utterance (Bunt Romary, LREC 2002).
• A dialogue act has two components for describing
  utterance meanings
• the information which the speakers makes
  available to the addressee - the semantic
  content
• the communicative function, capturing the way
  the speaker intends an addressee to update his
  information state with the semantic content.

8
Multifunctionality

• U Can you tell me what time is the first
  train to the airport on Sunday?
• S On Sunday morning the first train to the
  airport is at 5.32
• U Thank you.
• - expression of thanks

9
Multifunctionality

• U Can you tell me what time is the first
  train to the airport on Sunday?
• S On Sunday morning the first train to the
  airport is at 5.32
• U Thank you.

10
Multifunctionality

• U Can you tell me what time is the first
  train to the airport on Sunday?
• S On Sunday morning the first train to the
  airport is at 5.32
• U Thank you.
• - expression of thanks

11
Multifunctionality

• U Can you tell me what time is the first
  train to the airport on Sunday?
• S On Sunday morning the first train to the
  airport is at 5.32
• U Thank you.
• - expression of thanks
• - positive feedback
• (about understanding and acceptance)

12
Multifunctionality

• U Can you tell me what time is the first
  train to the airport on Sunday?
• S On Sunday morning the first train to the
  airport is at 5.32
• U Thank you.
• - expression of thanks
• - positive feedback
• (about understanding and acceptance)
• - indication of dialogue closure

13
Multifunctionality

• U Can you tell me what time is the first
• train to the airport on Sunday?
• S The first train to the airport on Sunday
• is at ... let me see... 5.32
• - positive auto-feedback about
• perception and interpretation
• - WH-answer
• (to indirect WH-question)

14
Multidimensionality

• Utterances have multiple functions gt
• multiple annotation tags are required
• (or syntactically and semantically (!)
  complex tags --
  cf. studies by Popescu-Belis),
• i.e. annotation must be multidimensional.

15
Multidimensional annotation

• Usual informal notion of dimension
• Set of mutually exclusive tags
• Not satisfactory...
  See problems in multidimensional
  annotation according to DAMSL (Bunt, LREC
  2006)

16
Dimensions in Dialogue

• Basic intuition participants in a dialogue do
  multiple things simultaneously, such as
• making progress in performing the activity
  (task) which motivates the dialogue
• providing and eliciting communicative feedback
• take and assign turns
• monitor contact, attention, use of time,...
• greet, thank, apologize, say goodbye,...

17
Dimensions in Dialogue

• A dimension is an aspect of participating in a
  dialogue such that
• There is a class of dialogue acts for addressing
  this dimension (empirical foundation)
• It can be addressed independently of other
  dimensions
• Within a dimension, an utterance has at most one
  communicative function.

18
Observed dimensions (TDG3/LIRICS)

• Performing a certain task or activity through or
  with support from the communication
• Monitoring the interaction
• - providing and eliciting feedback
• - editing ones own or ones partners speech
• - managing the turn-taking
• - managing the use of time
• - managing contact and attention
• - managing the opening and closing of (sub-)
• dialogues and thematic progression
• Dealing with social obligations greeting,
• thanking, apologizing,

19
Dimensions for dialogue acts

• Examples
• dimension function
  example
• Auto-feedback OverallPositive Okay.
• Allo-feedback EvaluationElicitation
  Okay?
• Turn management TurnGiving
  Yes
• Time management Stalling
  Well, you know,..
• Contact mant ContactChecking
  Hello?
• Own comm. mant Self-correction
  I mean...
• Partner comm.man. Completion
  ... completion
• Topic management TopicShiftAnnounc.
  Something else.
• Dialogue structuring DA-announcement
  Question
• Social oblig. mant Valediction
  Bye
• Task/domain OpenMeeting I open
  this meeting

20
Dimensions of dialogue acts

• Example Inform
• The KL204 leaves at 12.30. Task/domain

21
Dimensions of dialogue acts

• Example Inform
• The KL204 leaves at 12.30. Task/domain
• I see what you mean. Auto-feedback

22
Dimensions of dialogue acts

• Example Inform
• The KL204 leaves at 12.30. Task/domain
• I see what you mean. Auto-feedback
• We should first discuss the agenda. Topic
  management

23
Dimensions of dialogue acts

• Example Inform
• The KL204 leaves at 12.30. Task/domain
• I see what you mean. Auto-feedback
• We should first discuss the agenda. Topic
  management
• Im very grateful for you help. Social obligation
  management

24
Dimensions of dialogue acts

• Example Inform
• The KL204 leaves at 12.30. Task/domain
• I see what you mean. Auto-feedback
• We should first discuss the agenda. Topic
  management
• Im very grateful for you help. Social obligation
  management
• gt Inform acts can be used in every dimension.

25
Dimensions for dialogue acts

• A number of the most commonly used types of
  dialogue act, such as questions, answers,
  statements, requests, instructions, or offers,..
  do not belong to any dimension they are general
  purpose functions they can be used in any
  dimension.
• (DAMSL dimensions like Info-request and Answer
  are clearly not proper dimensions.)

26
General-purpose functions

• Applicable in any dimension are
• Information-seeking functions
• WH-question, YN-question,
  Alternatives-question, Check,..
• Information-providing functions
• Inform, WH-Answer, YN-Answer, Confirmation,
  Disconfirmation, Agreement, Correction,..
• Commissive functions
• Offer, Promise, AcceptRequest,..
• Directive functions
• Instruct, Request, Suggest,..

27
Core dimensions and dialogue acts

• Data categories from LIRICS
• Set of 54 core dialogue act types
• 24 general-purpose functions
• 30 dimension-specific functions spread over 10
  dimensions
• described in the form of ISO (12620) data
  categories.
• Compare
• DAMSL 12 dimensions, 30 functions
• SWBD-DAMSL 60 functions
• DIT 11 dimensions, 95 functions

28

• Validation of LIRICS data categories
• Usability for human annotators
• Inter-annotator agreement measurements for
  English and Dutch
• 2 trained annotators working on raw text/audio
• Results almost perfect agreement (Rietveld
  van Hout, 1993 kappa 0.80)

29

• Inter-annotator agreement scores

30
Validation of LIRICS data categories

• Applicability also demonstrated for Italian
  (annotated test suite developed in Pisa).
• Application of to multi-party multimodal AMI
  dialogues (Petukhova Bunt, IWCS-7) results
  comparing favourably with use of AMI or DAMSL
  annotation schemes.
• Machine learnability investigations are promising
  (Geertzen et al., SIGDIAL 2007).

31
Towards a dialogue act annotation language

• DA tag components
• lt Dimension name, Function name gt
• Examples
• lt Activity, Confirm gt
• lt Feedback, CheckQuestion gt
• lt Turn Management,Turn Release gt
• lt Social Obligations Management, Apology gt
• Note for dimension-specific functions, the
  dimension name is in fact redundant.

32
Design of dialogue annotation language DiaML

• Distinction in Linguistic Annotation Framework
• annotations information structures independent
  of representation format (abstract syntax)
• representations annotations cast in a certain
  format (concrete syntax)

33
Design of dialogue annotation language DiaML

• Distinction in Linguistic Annotation Framework
• annotations information structures independent
  of representation format (abstract syntax)
• representations annnotations cast in a certain
  format (concrete syntax)
• In addition
• semantics, defined for abstract syntax

34
DiaML abstract syntax

• Abstract Syntax format-independent definition of
  information structures
• For dialogue acts pairs of stretches (possibly
  discontinuous) of dialogue behaviour and sets of
  dialogue act types (at most one function in each
  dimension)
• Information to be expressed in DiaML
• speaker and addressee(s)
• segments of dialogue behaviour
• dimensions
• communicative functions
• optionally functional dependencies
  (e.g. an utterance is
  an Answer to which Question, or provides Feedback
  on which previous dialogue act)

35
DiaML abstract syntax

• Information to be expressed in DiaML
• speaker and addressee(s)
• segment of dialogue behaviour
• dimension
• communicative function
• optionally functional dependency
• Conceptual elements
• finite set of dialogue participants
• finite, ordered set of segment begin/end
  indicators
• finite set of dimensions
• finite sets of domain-specific and
  general-purpose comm. functions

36
DiaML abstract syntax

• Information to be expressed in DiaML
• speaker and addressee(s)
• segment of dialogue behaviour
• dimension
• communicative function
• Structure definitions
• A DiaML segment (markable) is a finite sequence
  of pairs of segment begin/end indicators,
  defining a stretch of source text
• A DiaML tag is an n-tuple of pairs
  ltdimension, functiongt
• A complete DiaML structure is a 4-tuple
  ltspeaker, addressee, segment,
  DiaML-taggt

37
DiaML concrete syntax

• ltdiaML idd2 speakers addresseea
  markablem1 commfunctionscfs1gt
• ltsourceText idm1 sb1..se1blabla
  sb3..se3blablagt
• ltcfs idcfs1 taskFunf1 feedbackFunf2gt
• ltcomfun idf1 functionanwer
  respTod1gt
• ltcomfun idf2 functionpositiv
  respTod1gt
• lt/cfsgt
• lt/diaMLgt

38
Current status

• Result of NWIP ballot?
• If NWIP approved
• 1. Project (editorial) group
• David Traum
• Claudia Soria
• Jae-Woong Choe
• Andrei Popescu-Belis
• Jan Alexandersson
• Alex Chengyu Fang
• Koiti Hasida (tbc)
• .....
• 2. Time schedule and meetings
• Moscow, August 2008? (TC 37 annual meeting)
• Pisa, October 2008, workshop
• Tilburg, January 2009 (IWCS-9), workshop

39
(No Transcript)
40
(No Transcript)
41
Dimensions in DAMSL

• FLF Dimensions
• Statement
• Info-request
• Influencing-addressee-future-action
• Committing-speaker-future-action
• Conventional Opening or Closing
• Explicit-performative
• Exclamation
• Other

42
Dimensions in DAMSL (3)

• Example
• A I hope youll have a good time!
• B Yeah, thanks.
• And youll be visiting friends in
  Italy.
• 3. A Thats right.

43
Dimensions in DAMSL

• Definitions of some FLF Dimensions
• Statement Speaker makes a claim about the world
• Info-request Speaker requests Addressee to
  provide information
• Influencing-addressee-future-action
• Committing-speaker-future-action
• (.... 8)

44
Dimensions in DAMSL (2)

• BLF Dimensions
• Agreement
• Understanding
• Answer
• Information-relation

45
Dimensions in DAMSL (4)

• Conclusion
• Question and statement are mutually
  exclusive tags gt having them in different
  dimensions is wrong

46
Dimensions in DAMSL (2)

• Definitions of some BLF Dimensions
• Agreement
• Understanding Utterances concerning the
  understanding between Speaker and Addressee
• Answer Speaker provides information requested by
  the Addressee
• Information-relation

47
Dimensions in DAMSL (4)

• Can question and answer be alternatives in
  the same dimension?
• Consider
• 1. S Did you ask me something?
• 2. U Can I change the contrast?
• - question

48
Dimensions in DAMSL (4)

• Can question and answer be alternatives in
  the same dimension?
• Consider
• 1. S Did you ask me something?
• 2. U Can I change the contrast?
• - question
• - answer

49
Dimensions in DAMSL (4)

• Can question and answer be alternatives in
  the same dimension?
• Consider
• 1. S Did you ask me something?
• 2. U Can I change the contrast?
• - question
• - answer
• So question and answer can co-occur gt question
  and answer cannot be in the same dimension

50
Dimensions in DAMSL (3)

• Example
• A I hope youll have a good time!
• B Yeah, thanks.
• And youll be visiting friends in
  Italy.
• 3. A Thats right.

51
Dimensions in DAMSL (3)

• Example
• A I hope youll have a good time!
• B Yeah, thanks.
• And youll be visiting friends in
  Italy.
• Statement? Question?
• 3. A Thats right.

52
Dimensions in DAMSL (3)

• Example
• A I hope youll have a good time!
• B Yeah, thanks.
• And youll be visiting friends in
  Italy.
• Statement? Question?
• DAMSL - statement
• - info-request
• ... but a speaker cannot at the same time state
  something and question its truth!

53
Dimensions in DAMSL (4)

• Conclusion
• Question and statement are mutually
  exclusive tags gt having them in different
  dimensions is wrong
• Can they be alternatives in the same dimension?

54
Dimensions in DAMSL (4)

• Can question and answer be alternatives in
  the same dimension?
• Consider
• 1. S Did you ask me something?
• 2. U Can I change the contrast?

55
Dimensions in DAMSL (4)

• Can question and answer be alternatives in
  the same dimension?
• 1. S Did you ask me something?
• 2. U Can I change the contrast?
• - question about what the task doman
• - answer about what the speaker said
• gt The task domain and what the speaker said are
• different dimensions an utterance can be
  a question in
• one dimension and an answer in another.

56
Dimensions in DAMSL (5)

• Example
• A And what possibilities do you have on
• Tursday?
• B Did you say Thursday?
• - function in (DAMSLs) Understanding
• dimension, but which?
• - Signal-understanding?
• - Signal-non-understanding?
• - Check (function in DAMSLs Info-
• request dimension)

57
Multidimensional annotation scheme for dialogue
acts

• Two parts
• dimension-specific communicative functions for
  each dimension
• general-purpose functions (hierarchically
  organized, reflecting degrees of specificity)