On the role of context and prosody in the interpretation of

1
On the role of context and prosody in the
interpretation of okay
ACL, June 2007, Prague

• Agustín Gravano, Stefan Benus, Julia Hirschberg
• Héctor Chávez, and Lauren Wilcox

Spoken Language Processing Group Columbia
University
2
Overview

• What information do subjects use to interpret
  the word okay in dialogue?
• Perception Study.
• Findings
• Contextual cues stronger predictors than acoustic
  / prosodic / phonetic cues.
• Final rising pitch Strongest prosodic cue.

3
Overview

• thats pretty much okay
• Speaker 1 between the yellow mermaid and
  the whaleSpeaker 2 okaySpeaker 1 and it is
• okay we gonna be placing the blue moon

4
Cue Words

• Linguistic expressions that can be used
• to convey information about the discourse
  structure, or
• to make a semantic contribution.
• Discourse markers, cue phrases, clue words,
• Examples
• now, well, so, alright, and, okay, first, on the
  other hand, by the way, for example,

5
Research Questions

• In spoken dialogue, how do hearers disambiguate
  cue words?
• How important is acoustic/prosodic information?
• What is the role of phonetic variation?
• What is the role of discourse context?

6
Why do we care?

• Spoken dialogue systems
• Need to convey potentially ambiguous terms with a
  particular intended meaning.
• Must interpret the users input correctly.

7
Previous Work

• Cues to cue phrase disambiguation
• Hirschberg Litman 87, 93 Hockey 93 Litman
  94
• Cues to Dialogue Act identification
• Jurafsky et al 98 Rosset Lamel 04
• Contextual cues to the production of backchannels
• Ward Tsukahara 00 Sanjanhar Ward 06

8
The Columbia Games Corpus

• 12 spontaneous task-oriented dyadic conversations
  in Standard American English.
• 2 subjects playing a computer game, no eye
  contact.

Follower
Describer
9
The Columbia Games CorpusAnnotation of
Affirmative Cue Words

• Cue Words
• alright
• gotcha
• huh
• mm-hm
• okay
• right
• uh-huh
• yeah
• yep
• yes
• yup

• Functions
• Acknowledgment / Agreement
• Backchannel
• Cue beginning discourse segment
• Cue ending discourse segment
• Check with the interlocutor
• Stall / Filler
• Back from a task
• Literal modifier
• Pivot beginning
• Pivot ending

• count
• the 4565
• of 1534
• okay 1151
• and 886
• like 753

10
Perception StudyExperiment Design
Speaker 1 yeah um there's like there's some
space there's Speaker 2 okay I think I got it
Speaker 1 but it's gonna be below the
onion Speaker 2 okay
Speaker 1 okay alright I'll try it
okay Speaker 2 okay the owl is blinking
11
Perception StudyExperiment Design

• 54 instances of okay (18 for each function).
• 2 tokens for each okay
• Isolated condition Only the word okay.
• Contextualized condition 2 full speaker turns
• The turn containing the target okay and
• The previous turn by the other speaker.

12
Perception StudyExperiment Design

• Two parts
• Part 1 54 isolated tokens
• Part 2 54 contextualized tokens
• Subjects asked to classify each token of okay
  as
• Acknowledgment / Agreement, or
• Backchannel, or
• Cue beginning discourse segment.

13
Perception StudyExperiment Implementation

• Subjects
• 20 paid subjects (10 female, 10 male).
• Ages between 20 and 60.
• Native speakers of English.
• No hearing problems.
• GUI on a laboratory workstation with headphones.

14
ResultsInter-Subject Agreement

• Kappa measure of agreement with respect to chance
  (Fleiss 71)

Isolated Condition Contextualized Condition
Overall .120 .294
Ack / Agree vs. Other .089 .227
Backchannel vs. Other .118 .164
Cue beginning vs. Other .157 .497
15
ResultsCues to Interpretation

• Phonetic transcription of okay
• Isolated Condition
• Strong correlation for realization of
• ? Backchannel
• ? Ack/Agree, Cue Beginning
• Contextualized Condition
• No strong correlations found for phonetic
  variants.

16
ResultsCues to Interpretation
Isolated Condition Contextualized Condition
Ack / Agree Shorter /k/ Shorter latency between turns Shorter pause before okay
Backchannel Lower intensity Higher final pitch slope Longer 2nd syllable Higher final pitch slope More words by S2 before okay Fewer words by S1 after okay
Cue beginning Lower final pitch slope Lower overall pitch slope Longer latency between turns More words by S1 after okay Lower final pitch slope
S1 Utterer of the target okay. S2 The
other speaker.
17
ResultsCues to Interpretation

• Final intonation using the ToBI conventions.
• (Both isolated and contextualized conditions.)
• H-H ? Backchannel
• H-L
• L-H ? Ack/Agree, Backchannel
• L-L ? Ack/Agree, Cue beginning

18
Conclusions

• Agreement
• Availability of context improves inter-subject
  agreement.
• Cue beginnings easier to disambiguate than the
  other two functions.
• Cues to interpretation
• Contextual features trump features of word okay.
• Exception Final pitch slope of okay in both
  conditions.

19
Further Work

• Benus et al, 2007
• The prosody of backchannels in American
  English, ICPhS 2007, Saarbrücken, Germany,
  August 2007.
• Gravano et al, 2007
• Classification of discourse functions of
  affirmative words in spoken dialogue,
  Interspeech 2007, Antwerp, Belgium, August 2007.

20
On the role of context and prosody in the
interpretation of okay
ACL, June 2007, Prague

• Agustín Gravano, Stefan Benus, Julia Hirschberg
• Héctor Chávez, and Lauren Wilcox

Spoken Language Processing Group Columbia
University
21
The Columbia Games CorpusAnnotation

• Orthographic transcription and alignment.
• Laughs, coughs, breaths, smacks,
  throat-clearings.
• Self repairs.
• Intonation, using the ToBI convention.
• Function of affirmative cue words (alright,
  mm-hm, okay, right, uh-huh, yeah, yes, ).
• Question form and function.

22
The Columbia Games CorpusAnnotation of
Intonation ToBI

• Tones
• Pitch accents L, H, LH, H!H,
• Phrase accents L-, H-, !H-
• Boundary tones L, H
• Break Indices
• Degrees of junction0 no word boundary ... 4
  full intonational phrase boundary
• Miscellanea
• Disfluencies, non-speech sounds, etc.

23
The Columbia Games CorpusAnnotation of
Intonation ToBI
24
Perception StudyDefinitions Given to the Subjects

• Acknowledge/Agreement
• The function of okay that indicates I believe
  what you said and/or I agree with what you
  say.
• Backchannel
• The function of okay in response to another
  speaker's utterance that indicates only Im
  still here or I hear you and please continue.
• Cue beginning discourse segment
• The function of okay that marks a new segment
  of a discourse or a new topic. This use of okay
  could be replaced by now.