Flexible Dialog Management for In-vehicle Dialog Systems

1
Flexible Dialog Management for In-vehicle Dialog
Systems

• Jeongwoo Ko
• jko_at_cs.cmu.edu

2
Outline

• Introduction
• Approach in CAMMIA
• Dialog Management Framework
• System Architecture
• ScenarioTemplate/ScenarioXML
• ScenarioXML Development Kit
• Pilot Systems Experiments
• Current Research Focus
• Future Work

3
Introduction of CAMMIA Project
(Conversational Agent for Multilingual Mobile
Information Access)
4
Sample Dialog

• S1 How may I help you?
• U1 I want to go to Carnegie Mellon University.
• S2 Do you want to go to Carnegie Mellon
  University?
• U2 Yes.
• S3 The distance to the destination is 100 miles.
  
• It takes about 2 hours.
• U3 I would like to know weather.
• S4 Please tell me the area and the date.
• U4 Pittsburgh
• (Navigation System sends the next direction to
  the dialog manager)
• S5 To go to Carnegie Mellon University, please
  make a left turn here.
• Please tell me the date for Pittsburgh.
• U5 Tomorrow
• S6 The weather for Pittsburgh tomorrow will be
  fine.

5
Some Related Research (Speech Interface)

• Pellom et al, HLT 2001 (CU-Move)
• Route planning and navigation
• Noise suppression front-end and back-end
  navigation information retrieval
• Download driving instructions from the Internet
  after route planning
• User can ask route information during travel
• (ex. Whats my next turn?)
• Coletti et al, IEEE 2003
• Hotel retrieval/reservation, POI retrieval,
  Simple Route Query
• Car Wide Web module XML-based DB interface
• Local database for tour and geographic data

6
Some Related Research (Multimodal Interface)

• Minker et al, ICSLP 2002 (SmartKom Mobile)
• Provide framework for modality control
• Prototype on Compac iPAG H3630 handheld computer
• Apply to pedestrian and driver environment
• Display maps for route information
• Slide shows for sight information

7
Approach in CAMMIA

• Asynchronous communication with navigation system
• When getting the next direction, interrupt the
  current dialog
• Notify the next turn direction
• Resume the pending dialog
• Maintain dialog history for smooth dialog
  switching
• Flexible robust Dialog Manager based on
  VoiceXML (Voice eXtensible Markup Language)
• Error handling
• Correction, Explicit/Implicit confirmation
• Support multi-lingual (Japanese, English)

8
Next

• Introduction
• Approach in CAMMIA
• Dialog Management Framework
• System Architecture
• ScenarioTemplate/ScenarioXML
• ScenarioXML Development Kit
• Pilot Systems Experiments
• Current Research Focus
• Future Work

9
Dialog Management Framework
Dialog Manager (DM)
Speech Interface
Navigation System (NS)
Dialog Controller
VXI
Navigation System Simulator
Dialog history Dialog scenario Data source
Julius
Map Display
ADM
VXI VoiceXML Interpreter ADM Asynchronous
Dialog Manager
10
Dialog Manager

• Support multi-user/multi-dialogs
• Keep track of user session and dialog flows for
  smooth task switching
• Convert user utterances to database query
• Mixed-initiative interaction
• Create VoiceXML with dynamic contents
• gt Hard to write VoiceXML by hands
• gt Provide abstract level of dialog
  description

11
Dialog Description

• ScenarioTemplate (ST)
• Designed to facilitate new dialog creation
• Consist of prompts and variables
• gt Dialog designer does not have to know
  grammar
• Support explicit/implicit confirmation
• Compiled into SXML
• ScenarioXML (SXML)
• Consist of dialog states and transitions
• Dialog developers need to know grammar, but dont
  have to add other dialogs for dialog switching
• Compiled into JSP to create VXML with dynamic
  contents

12
Example of ST and SXML
lt?xml version"1.0" encoding"UTF-8"?gt lt!DOCTYPE
function SYSTEM "dtd/function.dtd"gt ltfunction
name"Weather"gt ltstate name"ask_weather"
position"start"gt ltgrammar src"grammars/ask_w
eather_jp.gad"/gt ltpromptgt lttextgtPlease
tell me the area and datelt/textgt lt/promptgt
ltjumpgt ltnextstate next"ask_weather_area"gt
ltfieldgtweather_datelt/fieldgt
lt/nextstategt ltnextstate next"ask_weather_da
te"gt ltfieldgtweather_arealt/fieldgt
lt/nextstategt ltnextstate next"confirm_weathe
r"gt ltfieldgtweather_arealt/fieldgt
ltfieldgtweather_datelt/fieldgt lt/nextstategt
ltdefaultgtCan you tell me again?lt/defaultgt
lt/jumpgt lt/stategt ltstate name"ask_weather_area
" position"transition"gt lt/stategt lt/function
gt

• ltvariablesgt
• weather_area String
• weather_date String
• weather_result String
• ltstatesgt
• prompt Please tell me the area and date.
• variable weather_area weather_date
• prompt Please tell me the area.
• variable weather_area
• prompt please tell me the date.
• variable weather_date
• confirmation Would you like to know the weather
  for weather_area weather_date?
• response The weather for weather_date
  weather_area is weather_result.

13
Example of SXML and VoiceXML
lt?xml version"1.0" encoding"UTF-8"?gt lt!DOCTYPE
function SYSTEM "dtd/function.dtd"gt ltfunction
name"Weather"gt ltstate name"ask_weather"
position"start"gt ltgrammar src"grammars/ask_w
eather_jp.gad"/gt ltpromptgt lttextgtPlease
tell me the area and datelt/textgt lt/promptgt
ltjumpgt ltnextstate next"ask_weather_area"gt
ltfieldgtweather_datelt/fieldgt
lt/nextstategt ltnextstate next"ask_weather_da
te"gt ltfieldgtweather_arealt/fieldgt
lt/nextstategt ltnextstate next"confirm_weathe
r"gt ltfieldgtweather_arealt/fieldgt
ltfieldgtweather_datelt/fieldgt lt/nextstategt
ltdefaultgtCan you tell me again?lt/defaultgt
lt/jumpgt lt/stategt ltstate name"ask_weather_area
" position"transition"gt lt/stategt lt/function
gt
lt?xml version"1.0" encoding"UTF-8" ?gt ltvxml
version"2.0" xmlns"http//www.w3.org/2001/vxml"gt
ltformgt ltproperty name"message"
value"'SessionID99 IP127.0.0.1 PORT1001
COMGRAMMAR PARAMC\CAMMIA\grammars/enter_direct
ion_jp.gadenter_parking_jp.gadenter_restaurant_j
p.gadcancel_jp.gadask_weather_jp.gadCorrection.
gad'" /gt ltblockgt ltpromptgt Please tell me the
area and datelt/promptgt lt/blockgt ltfield
name"weather_intention" /gt ltfield
name"weather_area" /gt ltfield name"weather_date"
/gt ltfilled namelist"weather_intention"gt
ltif cond"(weather_date!undefined)"gt  
ltsubmit namelist"recog_result weather_date"
next"ask_weather_area.jsp" /gt  
lt/ifgt ltif cond"(weather_area!undefined)"gt  
ltsubmit namelist"recog_result weather_area"
next"ask_weather_date.jsp" /gt  
lt/ifgt lt/filledgt lt/formgt lt/vxmlgt
ltfield nameparking_intention" /gt ltfield
namerestaurant_intention" /gt ltfield
namedirection_intention" /gt
ltfilled namelistdirection_intention"gt ltgoto
next"ask_direction.jsp"/gt lt/filledgt
14
ScenarioTemplate Format

• lttemplategt ltvariablesgtltstatesgt
• ltvariablesgt ltvariablegt
• ltvariablegt ltstatenamegt_ltidgtlttypegt
• ltstatenamegt ltlettergt
• ltidgt ltlettergt
• lttypegt StringArrayList
• ltlettergt abcdefghijklmnopqr
  stuvwxyz
• ABCDEFGHIJKLMNOP
  QRSTUVWXYZ
• ltstatesgt ltstategt
• ltstategt ltquestiongtltconfirmationgtltresponsegt
  
• ltquestiongt ltvariablegtltpromptgtltbackpromptgt
• ltpromptgt ltlettergt ltvariablegt " "
• ltbackpromptgt ltpromptgt
• ltconfirmationgt ltpromptgt
• ltresponsegt ltpromptgt

15
ScenarioXML DTD

• lt?xml version"1.0" encoding"UTF-8"?gt
• lt!ELEMENT function (state)gt
• lt!ATTLIST function name CDATA REQUIREDgt
• lt!ELEMENT state (grammar?, result, prompt?,
  backprompt?, filled?, jump?, nofound?)gt
• lt!ATTLIST state name CDATA REQUIRED
• position CDATA IMPLIEDgt
• lt!ELEMENT grammar EMPTYgt
• lt!ATTLIST grammar src CDATA REQUIREDgt
• lt!ELEMENT result (field, list, param)gt
• lt!ATTLIST result class CDATA REQUIRED
• method CDATA REQUIREDgt
• lt!ELEMENT field (PCDATA)gt
• lt!ELEMENT list (PCDATA)gt
• lt!ATTLIST field condition CDATA IMPLIEDgt
• lt!ELEMENT param (PCDATA)gt
• lt!ELEMENT prompt ANYgt
• lt!ELEMENT text (PCDATA)gt
• lt!ELEMENT backprompt ANYgt
• lt!ELEMENT filled (field)gt

16
SXML Development Kit (SXMLDK)

• Two-step compilation by SXMLDK

ScenarioXML Development Kit
Scenario Template
Voice XML
JSP files
SXML Compiler
17
SXMLDK User Interface
18
Next

• Introduction
• Approach in CAMMIA
• Dialog Management Framework
• System Architecture
• ScenarioTemplate/ScenarioXML
• ScenarioXML Development Kit
• Pilot Systems Experiments
• Current Research Focus
• Future Work

19
Pilot Systems

• Prototype I (2002)
• Focus on building general architecture to support
  SXML
• HTTP is session-less and VoiceXML needs to keep
  users utterances in each session
• SXML provides session variable to keep users
  utterances
• ltsession-variablesgt
• ltpersistent names"DMEnv_persistent_topic
  DMEnv_persistent_weather_date DMEnv_persistent_wea
  ther_area"/gt
• lttransient names"DMEnv_answer_weather_date
  DMEnv_answer_weather_area DMEnv_answer"/gt
• lt/session-variablesgt

20
Pilot Systems

• Prototype II (2003)
• Improve Prototype I
• Develop ScenarioTemplate SXMLDK
• Filled values and dialog states are stored in
  back-end dialog controller
• Only the current values are stored in VoiceXML

21
Pilot Experiments

• Compare Prototype I II
• Test environments
• VXI Intel Pentium III 800MHz, 512M RAM
• DM Intel Pentium II 400MHz, 224M RAM
• Sample Dialog
• U1 Weather information, please
• S1 Please tell me the area and the date
• U2 Tomorrow
• S2 Please tell me the area
• U3 Pittsburgh
• S3 Are you asking the weather for Pittsburgh
  tomorrow?
• U4 Yes.
• S4 The weather for Pittsburgh tomorrow will
  be sunny.

date
area
confirmation
response
22
Comparison of Bandwidth Utilization

• To compare bandwidth utilization, VXML file sizes
  were measured

Sub Task Prototype I Prototype II
Ask date 6266 931
Ask area 1949 618
Confirmation 1709 607
Response 3110 283
Avg VXML Size 3258 610
Unit Byte
23
Comparison of User Wait Time

• To compare end-user wait time, loading time in
  VXI were measured

Sub Task Prototype I Prototype II
Ask date 5.25 0.88
Ask area 3.92 1.28
Confirmation 4.47 1.50
Response 3.44 1.14
Avg Loading Time 4.27 1.20
Unit second
24
Summary of Prototype II

• Improvement
• Efficient VXML management VXML includes only the
  current state information
• gt faster less use of network bandwidth
• Easy to add new dialogs using ScenarioTemplate
• Easy to support multiple languages by changing
  ScenarioTemplate prompts and grammars
• Issues
• Predefined dialog states and prompts
• Hard to support more complex dialogs

25
Example of Complex Dialog

• U1 Tour guide, please.
• S1 Tour guidance is started. Where is a
  destination?
• U2 Carnegie Mellon University
• S2 The distance to the destination is 100 miles.
  It takes about 2 hours.
• U3 Do you know a good restaurant near here?
• S3 What kind of food do you like?
• U4 I want to eat Italian food.
• S4 There are Olive garden, Bravo and Laromana.
• U5 Well, I'd like to go to Laromana.
• S5 It does not have a parking lot. Is it OK?
• U6 Tell me the one which has parking lots.
• S6 Olive garden and Bravo have parking lots.
• U7 Which one is closer?
• S7 Bravo
• U8 I would like to go there.
• S8 Do you want to add it as a way point?
• U9 Yes
• S9 I set Bravo as the waypoint.

26
Next

• Introduction
• Approach in CAMMIA
• Dialog Management Framework
• System Architecture
• ScenarioTemplate/ScenarioXML
• ScenarioXML Development Kit
• Pilot Systems Experiments
• Current Research Focus
• Future Work

27
Current Research Focus

• Flexible dialog management
• Push model Offer important information even
  though user does not request
• Ex) do not take credit card, do not have
  parking lot
• Search data from the list which was already
  retrieved
• Support comparison (the cheapest, closer)
• Anaphora resolution (there, it)
• Add way point to Navigation Map

28
Current Research Focus

• Robust dialog management for signal loss
• Task Manager
• - Retrieval task management for signal loss
• Dialog Manager
• - Located in vehicle
• Info Manager
• - Retrieve data from the remote database servers
• - Maintain local cache with timestamps

29
Extended Architecture
Dialog Manager
User Interface
Info Manager
Data Server
Dialog Controller
Voice
Julius
Local Cache
Task Manager
ADM
DB

DB
Navigation System Simulator
Map Display
Navigation System
Remote Server
Vehicle
30
Dialog Manager

• To support information seeking dialogs, it has
  three dialog states
• Search
• - Ask the user to fill the minimum search
  constraints
• - Display the search results
• Ex) There are three Italian restaurants near
  hear, A, B and C
• Search refinement
• - User can narrow down the searched items with
  different search options such as price and
  distance
• Ex) Which one is closer?
• Selection
• - Automatically check important information (Push
  model)
• - Add way point to the Navigation map
• Ex) I set it as the waypoint

31
Dynamic State Transition
need constraintsyes
need constraintsyes
Check search constraints
Ask constraints
need featuresno
New constraints
need featuresno
Display the results
Search
Search options
Selection
need selectionno
Search options
Find new lists from the previous results Display
it
Search Refinement
Selection
Search options
Notify it to the user
Check special info
Not satisfied
satisfied
Display detailed info Add way point if user wants
Selection
End dialog
32
Future work

• More work on Task Manager
• Anaphora and ellipsis resolution
• Multi-modal interface
• Integration of screen with Navigation map
• Complementary prompts (voice and screen)
• Dynamic grammar generation
• More intelligent push model based on user
  preference
• Missed turn