Impact of Initiative on Measures of Dialog Quality

1
Impact of Initiative on Measures of Dialog Quality

• Amanda Stent

2
Outline

• Evaluating Dialog Systems
• The Rate-a-Course Dialog System
• Initiative in the Rate-a-Course Dialog System
• Impact of Initiative on Measures of Dialog
  Success

3
Measures of Dialog Quality

• Popularized by Jokinen, di Eugenio, Walker et al.
• Used post-dialog
• dialog evaluation
• or in-dialog
• e.g. (PO)MDP approaches
• Incorporate intrinsic factors
• e.g. dialog length, SR error rate
• and extrinsic factors
• e.g. task completion, knowledge gained, user
  satisfaction

4
The Rate-a-Course Dialog System

• A spoken dialog system for collecting surveys
  (particularly, course surveys)

5
The Rate-a-Course Dialog System

• First the set-up
• What department? What course number? What
  semester and year?
• Then the survey
• Five topics instructor, TA, class size,
  homework assignments, exams

6
Initiative in the Rate-a-Course Dialog System

• Two kinds of initiative (Guinn)
• Dialog -- who is in control of the dialog
• Task -- who is in control of the task
• In Rate-a-Course, three levels of task
  initiative
• System
• Mixed
• User

7
System Initiative

• S We will now ask your opinion of the following
  aspects of your course the instructor, the
  assignments, the class size, the teaching
  assistant and the exams. Is the instructor very
  good, good, okay, bad, or very bad?
• U okay
• S Why do you think the instructor is okay?
• U .

8
Mixed Initiative

• S We will now ask your opinion of the following
  aspects of your course the instructor, the
  assignments, the class size, the teaching
  assistant and the exams. Which topic is the most
  important to you?
• U the instructor
• S Is the instructor very good, good, okay,
  bad, or very bad?
• U okay
• S Why do you think the instructor is okay?
• U .
• S Which topic is the next most important to
  you? Your choices are .

9
User Initiative

• S We will now ask your opinion of the following
  aspects of your course the instructor, the
  assignments, the class size, the teaching
  assistant and the exams. Which topic is the most
  important to you?
• U the instructor
• S Is the instructor very good, good, okay,
  bad, or very bad?
• U okay
• S Why do you think the instructor is okay?
• U .
• S Would you like to discuss another topic?

10
What Makes a Good Survey Response?

• Controlled
• All participants addressed the same topics
• All participants had the same type of interaction
• Informative
• Participants were motivated to be truthful and
  informative
• In practice, controlled wins (but see work by
  Schober, Conrad)

11
Impact of Initiative on Measures of Dialog Success

• Measures of informativeness
• Dialog length ( of topics addressed)
• Number of speech recognition errors
• Proportion of responses that were negative,
  neutral, positive
• Measures of engagement for different types of
  response length, speaking rate, F0 (avg,
  range), volume

12
Dialog Length, Error Rate
13
Response Type
14
Response Length
15
Speaking Rate
16
Mean F0
17
Max F0
18
Intensity
19
Summary

• In User condition, participants exhibit more
  engagement -- speaking more loudly and with wider
  range of F0
• In User condition, participants are more
  informative -- longer free responses
• In User condition, participants focus more on
  things they feel strongly about
• Dont rate the others, in general

20
Examples

• Good instructor
• Misinterpretation
• Harsh
• What do SBU students want?
• Funny, available instructors
• Predictable exams
• Available teaching assistants

21
Conclusions

• Measures of dialog quality should include
  informativeness, engagement (see also Litman)
• The overall measure of dialog quality varies by
  dialog goal
• Sometimes, giving up a little control gives rich
  rewards

22
Past and Future Work

• Past use of meta-linguistic information to
  acquire user and course models (Stent et al.,
  2006)
• Past Course review presentation/summary
• Future analysis by utterance subjectivity
• Future fold user/course models into running
  system