Conversational role assignment problem in multi-party dialogues

1
Conversational role assignment problem in
multi-party dialogues

• Natasa Jovanovic Dennis Reidsma
• Rutger Rienks
• TKI group
• University of Twente

2
Outline

• Research tasks at TKI
• Interpretation of multimodal human-human
  communication in the meetings
• Conversational Role Assignment Problem (CRAP)
• Towards automatic addressee detection

3
A framework for multimodal interaction research
4
Multimodal annotation tool
5
Who is talking to whom?

• CRAP as one of the main issues in multi- parity
  conversation (Traum 2003.)
• Taxonomy of conversational roles (Herbert K.
  Clark)

speaker
addressee
side participant
bystander
all participants
all listener
eavesdropper
6

• Our goal
• Automatic addressee identification in small
  group discussions
• Addressees in meeting conversations single
  participant, group of people, whole audience
• Importance of the issue of addressing in
  multi-party dialogues

7
Addressing mechanisms

• What are relevant sources of information for
  addressee identification in the face-to-face
  meeting conversations?
• How does the speaker express who is the addressee
  of his utterance?
• How can we combine all this information in order
  to determine the addressee of the utterance?

8
Sources of information

• Speech
• Linguistic markers
• word classes personal pronouns, determiners in
  combination with personal pronouns, possessive
  pronouns and adjectives, indefinite pronouns,
  etc.
• Name detection ( vocatives)
• Dialogue acts
• Gaze direction
• Pointing gestures
• Context categories(features)

9
Dialogue Acts and Addressee detection (I)

• How many addresses may have an utterance?
• According to dialog act theory an utterance or an
  utterance segment may have more than one
  conversational function.
• Each DA has a addressee gt an utterance may have
  several addresses

10
Dialogue Acts and Addressee detection (II)

• MRDA (Meeting Recorder Dialogue Acts) tag set
  for labeling multiparty face to face meetings
  (ICSI)
• We use a huge subset of the MRDA set which is
  organized on two levels
• Forward looking functions (FLF )
• Backward looking functions (BLF)

11
Non-verbal features

• Gaze
• Contribution of the gaze to the addressee
  detection is dependent on participants location
  (visible area), utterance length, current meeting
  action
• Turn-taking behavior and addressing behavior
• Gesture ( pointing at a person)
• TALK_TO (X,Y) AND POINT_TO (X,Y)
• TALK_TO( X,Y) AND POINT_TO (X,Z) X talk to Y
  about Z

12
Context categories

• Bunt totality of conditions that may influence
  understanding and generation of communicative
  behavior
• Local context is an aspect of context that can be
  changed through communication
• Context categories
• Interaction history ( verbal and non-verbal)
• Meeting action history
• Spatial context (participants location,
  distance, visible area, etc. )
• User context (name, gender, roles, etc. )

13
Towards an automatic addressee detection

• Manual or automatic features annotation?
• An automatic target interpreter has to deal with
  uncertainty
• Methods
• Rule-based method
• Statistical method ( Bayesian networks)

14
Rule-based method

• Processing information obtained from the
  utterance ( linguistic markers,
  vocatives, DA). The result is a list of possible
  addressees with corresponding probabilities
• Eliminate cases where target is completely
  determined (for instance, name in vocative form)
• Set of rules for BLF
• Set of rules for FLF
• Processing gaze and gesture information adding
  the additional probability values to the
  candidates

15
Meeting actions and addressee detection

• Automatic addressee detection method can be
  applied to the whole meeting
• Knowledge about the current meeting action as
  well as about meeting actions history may help to
  better recognize the addressee of a dialogue act.

16
Future works

• Development of multimodal annotation tool
• Data annotation for
• training and evaluating statistical models
• obtaining inputs for rule-based methods
• New meeting scenarios for research in addressing