Uncertainty Corpus: Resource to Study User Affect in Complex Spoken Dialogue Systems

1
Uncertainty Corpus Resource to Study User Affect
in Complex Spoken Dialogue Systems
Kate Forbes-Riley, Diane Litman, Scott Silliman,
Amruta PurandareUniversity of
PittsburghPittsburgh, PA, USA
2
Outline

• Introduction
• WOZ-TUT System
• Experimental Design
• Uncertainty Corpus Description
• Uses of the Uncertainty Corpus

3
Overview Towards Affect-Adaptive Spoken Dialogue
Systems

• Automatic Detection promising across affective
  states and applications, e.g. (Craig et al.,
  2006 Litman Forbes-Riley, 2006 Lee
  Narayanan, 2005 Vidrascu Devillers, 2005
  Batliner et al., 2003)
• Larger goal is automatic adaptation, but results
  are sparser
• More public affect-annotated corpora of
  human-computer dialogues could help, e.g. HUMAINE
  project
• SYMPAFLY, AIBO (Batliner et al., 2004) (German)
• Communicator (Walker et al., 2001 Ang et al.,
  2002) (English)
• Uncertainty Corpus
• new complex domain spoken dialogue tutoring
• new affect annotation student uncertainty

4
Uncertainty Corpus Collection WOZ-TUT System

• WOZ-TUT Adaptive Wizard of OZ Tutoring System
• modified version of ITSPOKE (Litman Silliman,
  2004)
• system responses based on combined uncertainty
  and correctness
• human recognizes speech, annotates uncertainty
  and correctness
• Why uncertainty?
• Most frequent in ITSPOKE corpora (Forbes-Riley
  Litman, 2007)
• Most systems respond only to correctness, but
  literature suggests uncertain and incorrect
  answers signal learning impasses
• What uncertainty adaptation?
• Treating uncertaincorrect answers as incorrect
  should provide additional knowledge to bridge
  impasse

5
WOZ-TUT Screenshot
6
Experimental Design

• 3 Conditions used parameterized WOZ-TUT dialogue
  manager
• Experimental treat all uncertaincorrect turns
  as incorrect
• First Control ignore uncertainty (logged)
• Second Control ignore uncertainty (logged), but
  treated a percentage of random correct answers as
  incorrect

7
Corpus Excerpts

• TUTOR What will the velocity of the object be a
  second after that (where the initial velocity is
  9.8m/s and the acceleration is 9.8m/s2)?
• STUDENT Nineteen point six meters per second??
  uncertaincorrect
• TUTOR in First Control Condition moves on Good.
  So at every point in time during the fall of the
  man and his keys, how do their velocities compare
  with each other?
• TUTOR in Experimental Condition remediates Okay.
  As we have seen, if a falling object has an
  acceleration of 9.8m/s2, its velocity changes by
  9.8m/s every second. So if a second after it
  began falling its velocity is 9.8m/s, a second
  later its velocity will be 9.8m/s 9.8m/s
  19.6m/s. So what will its velocity be a second
  after that?

8
Experimental Procedure

• 60 subjects randomly assigned to 3 conditions
  (gender-balanced)
• Native English speakers with no college physics
• Procedure 1) read background material, 2) took
  pretest, 3) worked training problem with WOZ-TUT,
  4) took posttest, 5) worked isomorphic test
  problem with non-adaptive WOZ-TUT

9
Corpus Description
Student Tutor
Total Turns 2171 2531
Total Uncertain Turns 796 -
Total Words 13533 111829
Average Words per Turn 6.23 44.20

• 120 dialogues from 60 students (.ogg format)
• 20 total hours of dialogue
• Student turns manually transcribed, including
  disfluency and non-syntactic question annotation
• Tutor turns and Wizard annotations in log files

10
Student Answer Attributes
Training Problem EXP CTRL1 CTRL2
Ave Turns 20.65 18.60 19.75
Ave Correct Turns 13.80 12.55 14.20
Ave Uncertain Turns 9.95 8.60 11.15
Ave UncertainCorrect Turns 4.75 3.75 6.10
Ave Adapted-To Turns 4.75 0 3.65
Ave UncertainCorrect and Adapted-To Turns 100 0 36

• One-way ANOVAs showed no significant differences
• number of correct, uncertain, or
  uncertaincorrect turns
• number adapted-to turns (EXP vs CTRL2)

11
Uses of the Uncertainty Corpus I

• Compare student performance across conditions to
  isolate impact of uncertainty adaptation
• No significant differences in learning
• We are comparing dialogue-based metrics in the
  isomorphic test problem (Forbes-Riley, Litman and
  Rotaru, 2008)
• - Feedback confound identified and rectified in
  larger follow-on study

Isomorphic Test Problem EXP CTRL1 CTRL2
Ave Turns 16.50 16.80 16.25
Ave Correct Turns 14.60 14.35 14.10
Ave Uncertain Turns 3.30 3.15 3.65
12
Uses of the Uncertainty Corpus II

• Resource for analyzing linguistic features of
  naturally-occurring user affect in human-computer
  dialogue
• Models built from elicited emotions generally
  transfer poorly to naturally-occurring dialogue
  (Cowie and Cornelius, 2003 Batliner et al.,
  2003)
• Uncertainty Corpus provides a new resource for
  modeling naturally-occurring dialogue
• Large number of features in speech, transcript,
  log files

13
Summary and Current Directions

• The Uncertainty Corpus is a collection of
  tutorial dialogues between students and an
  adaptive Wizard-of-Oz spoken dialogue system
• Corpus (speech, transcripts, uncertainty and
  correctness annotations) publicly available by
  request through the Pittsburgh Science of
  Learning Center https//learnlab.web.cmu.edu/da
  tashop/index.jsp
• Follow-on experiments and corpora
• Larger WOZ study just completed, with learning
  results!
• Fully automated study to begin Fall 2008

14
Thank You!

• Questions?
• Further Information?
• web search ITSPOKE or PSLC