MITRE%20Dialog%20Management%20Workshop%20

1
MITRE Dialog Management Workshop a review

• Dan Bohus
• Dialogs on Dialogs reading group
• CMU, November 2003

2
The Workshop

• MITRE Dialog Workshop
• _at_ MITRE, Bedford/Boston
• October 27-28, 2003
• Idea
• Bring together researchers working on dialog
  management
• Give them a homework
• Adapt you dialog manager to a medical diagnosis
  domain (details in a sec)
• Discuss, compare, learn

workshop godis ravenclaw collagen themes
3
The Homework

• Implement a dialog system for the medical
  diagnosis domain
• Task left open-ended (diagnosis, tutoring, etc)
• No speech, just text in and out
• Backend provided backend.doc
• Java version and web-based interface version
• 3 diseases malaria, coccidioidomycosis, another
  one
• List of symptoms headache, nausea, muscle pain,
  etc.
• Decision tree involving symptoms and tests
  (fever, blood tests, travel patterns, etc)
• Small enough to presumably not be lots of work,
  but large enough to allow illustration of
  functionalities, and provide some skeleton to the
  discussions

workshop godis ravenclaw collagen themes
4
Participants

• MITRE (Carl Burke et al) MiDiKi
• Gothenburg (Staffan Larsson) GoDiS (TRINDIKit)
• USC ICT (David Traum) ICT Dialogue Manager
• NTT/CMU (Matthias Denecke) Ariadne
• CMU (Dan, Alex) RavenClaw
• Ames (Beth-Ann Hockey) NASA Dialogue Manager
• DFKI (Norbert Reithinger) DFKI Dialogue Manager
• MERL (Candy Sidner, Charles Rich) COLLAGEN
• and others invited but not present

workshop godis ravenclaw collagen themes
5
GoDiS
GoDiS
workshop godis ravenclaw collagen themes
6
GoDiS

• TRINDIKit information state update dialogue
  management toolkit
• Information state
• Private dialog plan, beliefs, agenda (short term
  goals)
• Shared established facts, QUD, last utterance
  information
• Dialog moves
• Update rules
• GoDiS dialog management system implemented in
  TRINDIKit, handing
• information oriented dialogue
• action oriented dialogue

workshop godis ravenclaw collagen themes
7
TRINDIKit / GoDiS architecture
Dialog plansOntology
Connection to Java Backend
workshop godis ravenclaw collagen themes
8
GoDiS Task Representation

• Plans propositional logic
• Dialogue plans for dealing with diagnosis (issues
  opened at dialogue start)
• ?x.disease(x) which disease is diagnosed?
• ?confirmed_by_interview Is the diagnosis
  confirmed by additional information?
• ?confirmed_by_tests Is the diagnosis confirmed
  by medical tests?
• Additional plans
• ?x.info(x) What information is there about a
  given disease?
• ?x.treatment(x) What treatment is there for a
  given disease?

workshop godis ravenclaw collagen themes
9
GoDiS Alternate Tasks

• User-driven dialogue (implemented)
• Not load issues when resetting user has to raise
  all issues
• User can ask system to
• Provide a diagnosis
• Confirm whether user has given disease
• Decision trees as dialogue plans
• Move backend knowledge into dialogue plans
• Information conversion could be done
  automatically
• Separate genre expert system dialogue
• Add special purpose update rules
• Dynamic dialogue planning by expert

workshop godis ravenclaw collagen themes
10
GoDiS Highlights / Lowlights

• Highlights
• Reuse, you get for free
• Grounding
• Accomodation / plan recognition
• Multiple simultaneous issues info sharing
• High-level abstraction for dialog plans
• Rapid prototyping
• Lowlights
• Not used in this type of domain so far, so not
  entirely straight-forward (update rule changes)
• Dynamic dialog plans (backend decides)

workshop godis ravenclaw collagen themes
11
GoDiS
RavenClaw
workshop godis ravenclaw collagen themes
12
RavenClaw

• Captures all domain-specific dialog (task) logic
  with a hierarchical description
• The authoring effort is focused entirely here

Dialog Task (Specification)
Domain-independent Dialog Engine

• Manages dialog by executing the dialog task
  specification
• Provides domain-independent conversational
  strategies

workshop godis ravenclaw collagen themes
13
RavenClaw Architecture
Madeleine
ELoadSymptoms
GeneralFeel
Diagnose
IWelcome
RHowAreYou?
IGlad
ISorry
Fever
Travel
RAskFever
EMeasureTemp
IInformFever
Dialog Stack
Expectation Agenda
workshop godis ravenclaw collagen themes
14
RavenClaw Architecture
Madeleine
ELoadSymptoms
GeneralFeel
Diagnose
IWelcome
RHowAreYou?
IGlad
ISorry
Fever
Travel
RAskFever
EMeasureTemp
IInformFever
Dialog Stack
Expectation Agenda
Madeleine
workshop godis ravenclaw collagen themes
15
RavenClaw Architecture
Madeleine
ELoadSymptoms
GeneralFeel
Diagnose
IWelcome
RHowAreYou?
IGlad
ISorry
Fever
Travel
RAskFever
EMeasureTemp
IInformFever
Dialog Stack
Expectation Agenda
Welcome
Madeleine
workshop godis ravenclaw collagen themes
16
RavenClaw Architecture
Madeleine
ELoadSymptoms
GeneralFeel
Diagnose
IWelcome
RHowAreYou?
IGlad
ISorry
Fever
Travel
RAskFever
EMeasureTemp
IInformFever
Dialog Stack
Expectation Agenda
Hi, this is Madeleine, the automated
Madeleine
workshop godis ravenclaw collagen themes
17
RavenClaw Architecture
Madeleine
ELoadSymptoms
GeneralFeel
Diagnose
IWelcome
RHowAreYou?
IGlad
ISorry
Fever
Travel
RHeadache
R
R
R
RAskFever
EMeasureTemp
IInformFever
Dialog Stack
Expectation Agenda
Hi, this is Madeleine, the automated
LoadSymptoms
Madeleine
workshop godis ravenclaw collagen themes
18
RavenClaw Architecture
Madeleine
ELoadSymptoms
GeneralFeel
Diagnose
IWelcome
RHowAreYou?
IGlad
ISorry
Fever
Travel
RHeadache
R
R
R
RAskFever
EMeasureTemp
IInformFever
Dialog Stack
Expectation Agenda
Hi, this is Madeleine, the automated
Madeleine
workshop godis ravenclaw collagen themes
19
RavenClaw Architecture
Madeleine
ELoadSymptoms
GeneralFeel
Diagnose
IWelcome
RHowAreYou?
IGlad
ISorry
Fever
Travel
RHeadache
R
R
R
RAskFever
EMeasureTemp
IInformFever
Dialog Stack
Expectation Agenda
Hi, this is Madeleine, the automated
GeneralFeel
Madeleine
workshop godis ravenclaw collagen themes
20
RavenClaw Architecture
Madeleine
ELoadSymptoms
GeneralFeel
Diagnose
IWelcome
GeneralFeel
RHowAreYou?
IGlad
ISorry
Fever
Travel
RHeadache
R
R
R
IGlad
ISorry
RAskFever
EMeasureTemp
IInformFever
Dialog Stack
Expectation Agenda
Hi, this is Madeleine, the automated
general_feeling good, bad, soso
How are you feeling today?
general_feeling good, bad, soso
Not so good, I think I have a fever
general_feeling good, bad,
sosohave_fever fever. !yes,
!noheadache headache, !yes, !nocough
cough, !yes, !no
soso(not so good)fever(I think I have a
fever)
HowAreYou
GeneralFeel
GeneralFeel
Madeleine
workshop godis ravenclaw collagen themes
21
Illustrated Features

• Dynamic generation of dialog task structure
• Symptoms loaded from backend, appropriate
  structures to talk about them created
  on-the-fly
• New symptoms no DM changes
• Dynamic dialog control policy
• The order in which symptoms are addressed is
  controlled by the backend
• Conversational skills

workshop godis ravenclaw collagen themes
22
Illustrated Features

• Dynamic generation of dialog task structure
• Symptoms loaded from backend, appropriate
  structures to talk about them created
  on-the-fly
• New symptoms no DM changes
• Dynamic dialog control policy
• The order in which symptoms are addressed is
  controlled by the backend
• Conversational skills

workshop godis ravenclaw collagen themes
23
Dynamic Dialog Control
Madeleine
ELoadSymptoms
GeneralFeel
Diagnose
IWelcome
RHowAreYou?
IGlad
ISorry
Fever
Travel
RHeadache
R
R
R
RAskFever
EMeasureTemp
IInformFever
Dialog Stack
Expectation Agenda
Hi, this is Madeleine, the automated How are you
today? Not so good, I think I have a
headacheSorry to hear youre not feeling so
good,Tell me more about your symptoms Do you
have abdominal pain?
Diagnose
Madeleine
workshop godis ravenclaw collagen themes
24
Illustrated Features

• Dynamic generation of dialog task structure
• Symptoms loaded from backend, appropriate
  structures to talk about them created
  on-the-fly
• New symptoms no DM changes
• Dynamic dialog control policy
• The order in which symptoms are addressed is
  controlled by the backend
• Conversational skills

workshop godis ravenclaw collagen themes
25
Conversational Skills

• Corresponding agencies added automatically to the
  dialog task tree
• Help
• What Can I Say?
• Repeat
• Suspend / Resume
• Start Over
• Timeout handling (not illustrated)
• Still need all the language generation prompts
  and grammar, but some of those are develop-once,
  too

workshop godis ravenclaw collagen themes
26
RavenClaw Conclusion

• Highlights
• Set task posed no challenges to the framework
• Easy to implement
• Dynamic dialog structure and control
• Automatic use of domain-independent
  conversational skills
• Lowlights?
• Toolkit perspective how easy would it be for
  someone else to build it?
• Asynchronous behaviors? (timing)
• Couple of bugs / fixes (or is that a highlight?)

workshop godis ravenclaw collagen themes
27
GoDiS
Collagen
workshop godis ravenclaw collagen themes
28
COLLAGEN
workshop godis ravenclaw collagen themes
29
COLLAGEN Systems

• air travel planning
• email reading and responding (w. IBM/Lotus)
• GUI design tool operation
• car navigation system operation
• airport landing path planning (w. MITRE)
• gas turbine operator training (w. USC/ISI)
• personal video recorder operation
• programmable thermostat operation (with Delft
  U.)
• multi-modal web-based form-filling

workshop godis ravenclaw collagen themes
30
Collagen Theory and Implementation
workshop godis ravenclaw collagen themes
31
Collagen Discourse Segments and Purposes
workshop godis ravenclaw collagen themes
32
Discourse state representation
workshop godis ravenclaw collagen themes
33
Discourse interpretation algorithm
The current (communication or manipulation) act
either

• starts a new segment/focus space (push)
• ends the current segment/focus space (pop)
• continues (contributes to) the current
  segment/... (add)

(Lochbaum, 1998)
workshop godis ravenclaw collagen themes
34
COLLAGEN my take

• Separation of task from dialog/discourse engine
• Recipes / Domain plans / Task tree
• Full-blown HTN
• Hierarchical
• Preconditions (constraints)
• Effects
• Completion / failure
• Live nodes
• Stack to keep track of focus and discourse
  structure
• Tree explicitly contains agent and user nodes
• Formalized / descriptive recipe specs (actually
  Java underneath), with procedure overwrites

workshop godis ravenclaw collagen themes
35
GoDiS
Themes
workshop godis ravenclaw collagen themes
36
Themes Task Representation

• Task representation
• Separation of task representation from dialog
  engine
• High-level representations of task
• Descriptive rather than procedural
• Procedural will be unavoidable for complex tasks
• Expressive power
• GoDiS, RavenClaw, Collagen plan based
  representations of task

workshop godis ravenclaw collagen themes
37
Themes Task/Domain/Gendre

• The notion of dialog gendre
• Tutoring
• Diagnosis
• Information Access
• Where to fold it in a dialog manager?
• GoDiS update/select rules
• Ariadne plugins
• RavenClaw collapsed with task
• How clear is that separation task vs. gendre?

workshop godis ravenclaw collagen themes
38
Themes Development time

• Systems took on the order of 3-5 days to develop
• Significant effort in the backend connection
• Some sites shortcut it
• Significant effort in grammar/language generation
  development
• Some sites shortcut it
• Everyone that had an implementation fixed a
  couple of bugs, but no major changes required

workshop godis ravenclaw collagen themes
39
Themes Development tools

• Regression testing (GoDiS)
• Systems are complex. Change something in a dialog
  management framework, can you prove that it did
  not screw up things that used to work?
• System-wise, very intractable
• Component-wise, maybe i.e. DM with DM
  inputs/outputs
• System diagnosis / log visualization tools
  (Collagen)

workshop godis ravenclaw collagen themes
40
Themes Timing

• (Micro)timing
• unaddressed
• Turn-taking models
• in general, very rudimentary
• Asynchronous behaviors
• Could be accomplished, but no-one seemed to have
  it
• Multi-party conversation
• unaddressed

workshop godis ravenclaw collagen themes
41
Themes the important problems

• Different people have different views of what
  those are
• Plan / Intention recognition
• Reference resolution
• Backup in complex systems
• Tense problems
• Negations
• Grounding error prevention / recovery

workshop godis ravenclaw collagen themes
42
Themes Reasoning

• Dialog Managers vs Backends
• Where to draw the line?
• Who does the reasoning?
• Can we avoid duplicating it?
• How rich is the interaction between them?
• Dialog systems - use language to act in a domain,
  so they are generally strongly tied
• Basic set of conversational skills can be
  identified
• Drawing that line is still an art, no general
  agreement or solutions exist

workshop godis ravenclaw collagen themes
43
Themes Science of Dialog?

• How much science do we have?
• Theory vs. experiment
• Interesting Collagen / RavenClaw similarities
• Representation or not?
• GUI analogy
• Do we have the checkboxes and radio-buttons?

workshop godis ravenclaw collagen themes