Error Awareness and Recovery in TaskOriented Spoken Dialogue Systems

1
Error Awareness and Recovery in Task-Oriented
Spoken Dialogue Systems

• Thesis Proposal
• Dan Bohus
• Carnegie Mellon University, January 2004
• Thesis Committee
• Alex Rudnicky (Chair)
• Roni Rosenfeld
• Jeff Schneider
• Eric Horvitz (Microsoft Research)

2
Problem
Lack of robustness when faced with understanding
errors

• Spans most domains and interaction types
• Has a significant impact on performance

3
An example

• S Are you a registered user ?
• U No I'm not. No NO I'M NOT NO
• S What is your full name ?
• U Adam Schumacher ATHENS IN AKRON
• S There is an Athens in Georgia and in Greece.
  Which destination did you want ?
• U Neither NEITHER
• S I'm sorry, I didn't catch that. There is an
  Athens in Georgia and in Greece. Which
  destination did you want ?
• U Georgia GEORGIA
• S A flight from Athens... Where do you want to
  go ?
• U Start over START OVER
• S Do you really want to start over ?
• U Yes YES
• S What city are you leaving from ?
• U Hamilton, Ontario HILTON ONTARIO
• S Sorry, I'm not sure I understood what you
  said. Where are you leaving from ?
• U Hamilton HILTON
• S I'm still having trouble understanding you. To
  go on, I need you to answer the following
  question. What city are you leaving from ?
• U Toronto TORONTO

4
Some Statistics

• Semantic error rates

• Corrections Krahmer, Swerts, Litman, Levow
• 30 of utterances correct system mistakes
• 2-3 times more likely to be misrecognized

5
Significant Impact on Interaction

• CMU Communicator

40
Failed
26
33
sessions
Contain understanding errors

• Multi-site Communicator Corpus Shin et al

Failed
37
63
sessions
6
Outline

• Problem
• Approach
• Infrastructure
• Research Program
• Timeline Summary

problem approach infrastructure indicators
strategies decision process summary
7
Increasing Robustness

• Increase the accuracy of speech recognition
• Assume recognition is unreliable, and create the
  mechanisms for acting robustly at the dialogue
  management level

• ASR performance increases / demands increase
• More general

problem approach infrastructure indicators
strategies decision process summary
8
Snapshot of Existing Work Slide 1

• Theoretical models of grounding
• Contribution Model Clark, Grounding Acts
  Traum
• Practice heuristic rules
• Misunderstandings
• Threshold(s) on confidence scores
• Non-understandings

Analytical/Descriptive, not decision oriented
Ad-hoc, lack generality, not easy to extend
problem approach infrastructure indicators
strategies decision process summary
9
Snapshot of Existing Work Slide 2

• Conversation as Action under Uncertainty Paek
  and Horvitz
• Belief networks to model uncertainties
• Decisions based on expected utility, VOI-analysis
• Reinforcement learning for dialogue control
  policies Singh, Kearns, Litman, Walker, Levin,
  Pieraccini, Young, Scheffler, etc
• Formulate dialogue control as an MDP
• Learn optimal control policy from data

Do not scale up to complex, real-world domains
problem approach infrastructure indicators
strategies decision process summary
10
Research Program Goals Approach
A task-independent, adaptive and scalable
framework for error recovery in task-oriented
spoken dialogue systems
A task-independent, adaptive and scalable
framework for error recovery in task-oriented
spoken dialogue systems
Approach

• Decision making under uncertainty

problem approach infrastructure indicators
strategies decision process summary
11
Three Components
0. Infrastructure

• 1. Error awareness
• 2. Error recovery strategies
• 3. Error handling decision process

Develop indicators that ? Assess reliability
of information ? Assess how well the dialogue is
advancing
? Develop and investigate an extended set of
conversational error handling strategies
? Develop a scalable reinforcement-learning
based approach for error recovery in spoken
dialogue systems
problem approach infrastructure indicators
strategies decision process summary
problem approach infrastructure indicators
strategies decision process summary
12
Infrastructure

• RavenClaw
• Modern dialog management framework for complex,
  task-oriented domains
• RavenClaw spoken dialogue systems
• Test-bed for evaluation

Completed
Completed
problem approach infrastructure indicators
strategies decision process summary
13
RavenClaw
Dialogue Task (Specification)
Domain-Independent Dialogue Engine
problem approach infrastructure indicators
strategies decision process summary
14
RavenClaw-based Systems

• RoomLine
• CMU Lets Go!! Bus Information System
• LARRI Symphony
• TeamTalk 11-741
• Eureka 11-743

problem approach infrastructure indicators
strategies decision process summary
15
Three Components

• 0. Infrastructure
• 1. Error awareness
• 2. Error recovery strategies
• 3. Error handling decision process

Develop indicators that ? Assess reliability
of information ? Assess how well the dialogue is
advancing
? Develop and investigate an extended set of
conversational error handling strategies
? Develop a scalable reinforcement-learning
based approach for error recovery in spoken
dialogue systems
problem approach infrastructure indicators
strategies decision process summary
16
Existing Work

• Confidence Annotation
• Traditionally focused on speech
  recognizerBansal, Chase, Cox, and others
• Recently, multiple sources of knowledgeSan-Segun
  do, Walker, Bosch, Bohus, and others
• Recognition, parsing, dialogue management
• Detect misunderstandings 80-90 accuracy
• Correction and Aware Site DetectionSwerts,
  Litman, Levow and others
• Multiple sources of knowledge
• Detect corrections 80-90 accuracy

problem approach infrastructure indicators
strategies decision process summary
17
Proposed Belief Updating

• Continuously assess beliefs in light of initial
  confidence and subsequent events

• An example

S Where are you flying from? U
CityNameAspen/0.6 Austin/0.2 S Did you
say you wanted to fly out of Aspen? U No
CityNameBoston/0.8
initial belief
system action
user response
updated belief
CityNameAspen/? Austin/?
Boston/?
problem approach infrastructure indicators
strategies decision process summary
18
Belief Updating Approach

• Model the update in a dynamic belief network

t
t 1
C
C
C
C
initial belief
updated belief
system action
system action
User response features
CurrentTop
Current2nd
Current3rd
Confidence
Yes
No
Positive Markers
Negative Markers
Utterance Length
problem approach infrastructure indicators
strategies decision process summary
19
Three Components

• 0. Infrastructure
• 1. Error awareness
• 2. Error recovery strategies
• 3. Error handling decision process

Develop indicators that ? Assess reliability
of information ? Assess how well the dialogue is
advancing
? Develop and investigate an extended set of
conversational error handling strategies
? Develop a scalable reinforcement-learning
based approach for error recovery in spoken
dialogue systems
problem approach infrastructure indicators
strategies decision process summary
20
Is the Dialogue Advancing Normally?

• Locally, turn-level
• Non-understanding indicators
• Non-understanding flag directly available
• Develop additional indicators
• Recognition, Understanding, Interpretation
• Globally, discourse-level
• Dialogue-on-track indicators
• Summary statistics of non-understanding
  indicators
• Rate of dialogue advance

problem approach infrastructure indicators
strategies decision process summary
21
Three Components

• 0. Infrastructure
• 1. Error awareness
• 2. Error recovery strategies
• 3. Error handling decision process

Develop indicators that ? Assess reliability
of information ? Assess how well the dialogue is
advancing
? Develop and investigate an extended set of
conversational error handling strategies
? Develop a scalable reinforcement-learning
based approach for error recovery in spoken
dialogue systems
problem approach infrastructure indicators
strategies decision process summary
22
Error Recovery Strategies

• Identify
• Identify and define an extended set of error
  handling strategies
• Implement
• Construct task-decoupled implementations of a
  large number of strategies
• Evaluate
• Evaluate performance and bring further refinements

23
List of Error Recovery Strategies
problem approach infrastructure indicators
strategies decision process summary
24
List of Error Recovery Strategies
problem approach infrastructure indicators
strategies decision process summary
25
Error Recovery Strategies Evaluation

• Reusability
• Deploy in different spoken dialogue systems
• Efficiency of non-understanding strategies
• Simple metric Is the next utterance understood?
• Efficiency depends on decision process
• Construct upper and lower bounds for efficiency
• Lower bound decision process which chooses
  uniformly
• Upper bound human performs decision process (WOZ)

problem approach infrastructure indicators
strategies decision process summary
26
Three Components

• 0. Infrastructure
• 1. Error awareness
• 2. Error recovery strategies
• 3. Error handling decision process

Develop indicators that ? Assess reliability
of information ? Assess how well the dialogue is
advancing
? Develop and investigate an extended set of
conversational error handling strategies
? Develop a scalable reinforcement-learning
based approach for error recovery in spoken
dialogue systems
problem approach infrastructure indicators
strategies decision process summary
27
Previous Reinforcement Learning Work

• Dialogue control Markov Decision Process
• States
• Actions
• Rewards
• Previous work successes in small domains
• NJFun Singh, Kearns, Litman, Walker et al
• Problems
• Lack of scalability
• Once learned, policies are not reusable

S2
A
S3
S1
problem approach infrastructure indicators
strategies decision process summary
28
Proposed Approach

• Overcome previous shortcomings
• Focus learning only on error handling
• Reduces the size of the learning problem
• Favors reusability of learned policies
• Lessens the system development effort
• Use a divide-and-conquer approach
• Leverage independences in dialogue

problem approach infrastructure indicators
strategies decision process summary
29
Gated Markov Decision Processes
RoomLine
Login
Welcome
GreetUser
Gating Mechanism
AskRegistered
AskName

• Small-size models
• Parameters can be tied across models
• Easy to design initial policies

• Decoupling favors reusability of policies
• Accommodate dynamic task generation
• Independence assumption

problem approach infrastructure indicators
strategies decision process summary
30
Reward structure learning

• Rewards based on any dialogue performance metric
• Atypical, multi-agent reinforcement learning
  setting

• Multiple, standard RL problems
• Model-based approaches

problem approach infrastructure indicators
strategies decision process summary
31
Evaluation

• Performance
• Compare learned policies with initial heuristic
  policies
• Metrics
• Task completion
• Efficiency
• Number and lengths of error segments
• User satisfaction
• Scalability
• Deploy in a system operating with a sizable task
• Theoretical analysis

problem approach infrastructure indicators
strategies decision process summary
32
Outline

• Problem
• Approach
• Infrastructure
• Research Program
• Summary Timeline

problem approach infrastructure indicators
strategies decision process summary
33
Summary of Contributions

• Overall Goal develop a task-independent,
  adaptive and scalable framework for error
  recovery in task-oriented spoken dialogue systems
• Modern dialogue management framework
• Belief updating framework
• Investigation of an extended set of error
  handling strategies
• Scalable data-driven approach for learning error
  handling policies

problem approach infrastructure indicators
strategies decision process summary
34
Timeline
data
indicators
strategies
decisions
proposal
now
Misunderstanding andnon-understandingstrategies
Investigatetheoreticalaspects
ofproposedreinforcementlearningmodel
end ofyear 4
Evaluatenon-understandingstrategies
developremaining strategies
Data collection forbelief updating andWOZ study
milestone 1
Develop andevaluate thebelief updatingmodels
Implementdialogue-on-trackindicators
milestone 2
Error handling decision process reinforcement lea
rning experiments
Data collection forRL training
Data collection forRL evaluation
end ofyear 5
milestone 3
Contingency data collection efforts
Additional experiments extensions
or contingency work
defense
5.5 years
problem approach infrastructure indicators
strategies decision process summary
35
Thank You!

• Questions Comments
• committee members,
• then floor

36
Indicators Goals

• Goal Increase awareness and capacity to detect
  problems
• Develop indicators which can inform the error
  handling process about potential problems

37

problem approach support work indicators
strategies decision process summary
38
Three Desired Properties

• Task-Independence
• Reuse the proposed architecture across different
  spoken dialogue systems with a minimal amount of
  authoring effort
• Adaptability
• Learn from experience how to adapt to the
  characteristics of various domains
• Scalability
• Applicable in spoken dialogue systems operating
  with large, practical tasks

39
ExplConf
ExplConf
ExplConf
ImplConf
ImplConf
ImplConf
HC
LC
MC
NoAct
NoAct
NoAct
NoAct
0
40
Belief Updating Approach

• Model the update in a dynamic belief network

• Top-N values
• Fixed structure
• Learn parameters
• Data collection
• Evaluation
• Accuracy
• Soft-error

t
t 1
C
C
C
C
System Action
System Action
CurrentTop
Current2nd
Current3rd
CurrentTop
Current2nd
Current3rd
Confidence
Confidence
No
No
Yes
Yes
Negative Markers
Utterance Length
Negative Markers
Utterance Length
Positive Markers
Positive Markers
User response features
problem approach infrastructure indicators
strategies decision process summary
41
Gated Markov Decision Processes
RoomLine
Login
Welcome
GreetUser
Gating Mechanism
AskRegistered
AskName

• Issues
• Structure of individual MDPs
• Gating mechanism
• Reward structure and learning

problem approach infrastructure indicators
strategies decision process summary
42
Structure for individual MDPs

• State-space
• informative subset of corresponding indicators
• Concept-MDPs confidence / beliefs
• Topic-MDPs non-understanding, dialogue-on-track
  indicators
• Action-space
• corresponding system-initiated error handling
  strategies

problem approach infrastructure indicators
strategies decision process summary
43
Gating Mechanism

• Heuristic derived from domain-independent
  dialogue principles
• Give priority to topics over concept
• Give priority to entities closer to the
  conversational focus

problem approach infrastructure indicators
strategies decision process summary