CSLP 2004 2004

1
Representing ActTopicbased Dialogue Phenomena
Hans Dybkjær SpeechLogic, Prolog Development
Center A/SLaila DybkjærNISLab, University of
Southern Denmark
2
System-user dialogue
(Opening)

• S 0.1 PAUSE1000 Welcome to FerieKontos
  automatic guide. You may let the system
  suggest topics or you can try your own words.
  You select topics by speaking to the system.
  Choose between PAUSE200 'employee'.
  PAUSE200 On leave'. PAUSE200 'live
  abroad'. PAUSE200 Or 'stopped working' due to
  'age' or 'health'. PAUSE500 Say a topic, say
  repeat' or ask for 'more options'?
• U 0.1 Im a student
• S 1.1 Did you ask for Main menu
• U 1.1 student
• S 2.1 If you are a student with SU, you may get
  your holiday allowance without taking leave from
  the studies. But if you work besides the
  studies, you must take vacation at your work
  before you can get allowance.
• S 2.2 If you have work, it is your employer who
  must sign your FerieKonto certificate, otherwise
  it is the social authorities in your city.
• S 2.3 PAUSE500

(Offer)
(Request)
Start
(Check)
(Repair)
(Information)
(Information)
Success
A typical, system driven dialogue with
meta-communication
3
Dialogue, continued

• S 2.4 Do you want to ask for more?
• U 2.1 must I write the saturdays
• S 3.1 From the holiday year 2002 that starts 1
  May, the vacation week is changed to 5 vacation
  days, and saturdays are thus no longer counted as
  part of the vacation.
• S 3.2 Would you like to know more about the
  'holiday year'?
• U 3.1 no thanks goodbye
• S 4.1 Goodbye and thank you for calling us.
• Event 1 nothing recorded
• Event 2 disconnect

(Offer)
Start
(Request)
(Information)
Success
(Offer)
(Reject request)
Start
(Feedback)
(Other)
(Other)
Success
FAQ conversation has many small transactions
4
Automating annotation

• Key concerns for commercial spoken dialogue
  systems (SDSs)
• High transaction success rate
• Smooth dialogue
• Very time consuming and costly to measure
  manually
• Manual annotation more or less only possibility
  today
• Twostep approach towards automatic annotation
• Annotate utterances with a basic acttopic
  structure
• Can be automated using a parser
• Transform acttopic patterns into transaction
  segments
• Using rule engine

Automation only way to serious statistics
5
First step annotation

• .u .inform T.student
• I am a student"
• .s .inform T.student, T.payment, T.money
  
• If you are a student living from a grant
  you may get your holiday allowance while still
  studying. If you also work you need to take
  holiday to get your money.
• .s .inform T.more
• Is there anything else you want to ask
  about?"
• .u .accept
• Yes

Move Act Topic
Turn Move (of same speaker)
Context free annotation, simplistic,
6
Second step Transformation

• Apply acttopic rules to step one annotated
  dialogues
• Transform basic acts into composite acts

.s .inform T.pay "Payment in general .u .accept "Yes"
rule select1 _y .select Ts_a lt
_x .inform Ts_a _y .accept where
_x ! _y end rule
.u .select T.pay lt select1 .s .inform T.pay .u .accept
Formalism designed for the problem
7
Rule examples
fromplace lt place rule selectSub1 _y .select V_a lt _x .inform N_a _y .inform V_b where _a lt _b _x ! _y end rule rule success1 _y .success N_b lt _x .select T_b _y .inform Vs_a where _b in Vs_a _x ! _y end rule
.s inform N.fromplace Where does the travel start .u inform V.place Copenhagen .s inform V.fromplace.s inform N.toplace From Copenhagen, where do you want to go
Exchanges and transactions belong to different
levels
8
Dialogue phenomena investigated

• Different acts, different topics, and different
  speakers
• Rejecting topics
• Differentiating topic names and topic values
• Sets of topics
• Patterns across turns (move sets) not using all
  moves in that turn
• Topic relation (sub-topic) ISA (topic)
• Metacommunication (repair, clarification, ...)
• Multilevel rules Some rules only apply after
  match by other rules
• Summarising feedback

Only structural phenomena propositional
contents not considered
9
Status and next steps

• Restriction Task oriented humancomputer spoken
  dialogues
• Declarative rewrite rules with constraints
• Customised language close to dialogue analysis
  domain
• Smoothness criteria not clearly defined
• How does smoothness affect transaction counts?
• To obtain a fully automated annotation process
• Parse dialogues to produce basic acttopic
  annotation
• Combine into automatic batch system
• Need for evaluation of method
• Test on larger number of dialogues of different
  kinds
• Establish human coder baseline and compare

Both theoretical work and practical tools needed
10
Supplementary slides Warning you are over
time!
11
Background

• Overthephone FAQ SDS on holiday allowance
• General (nonperson related) questions, e.g.
• is Saturday considered a holiday?
• 2700 lines of grammar, 800 (full) words in
  vocabulary
• 85 semantic concepts in input, 100 stories in
  output
• Contractual minimum transaction success rate, but
• transaction not clearly defined
• no baseline from humanhuman dialogues
• Approach to measurement
• Transactions defined in terms of patterns of
  acttopics
• Manually 225 test and 217 production system
  dialogues
• Created webbased manual annotation tool

Complex domain but simple tasks
12
Problem in only identifying topics

• Basically only distinguish between two composite
  acts
• select continue with same topic
• request change to new topic
• So cannot distinguish success and failure

No success .u .inform T.phone Your phone number? .s .inform T.phone Phone number" Success .u .inform T.phone Your phone number? .s .inform T.phone Phone [phone]"
More distinction needed
13
Name and value

• Solution distinguish
• topic name N the mentioning of a topic
• topic value V details about a topic

.u .inform N.phone Your phone number? .s .inform N.phone Phone number" .u .inform N.phone Your phone number? .s .inform V.phone Phone [phone]"
Simple yet powerful distinction may still be
parseable
14
More rule examples
rule answer _y .request N_b _x .inform Vs_a lt _y .inform N_b _x .inform V_b where _x ! _y end rule rule success3 _y .success N_b lt _x .request N_b _y .inform Vs_a where _b in Vs_a _x ! _y end rule
Many variations possible
15
Transaction success

• Dialogue level task completion?
• Works if task is welldefined and goal state
  clear
• But many independent tasks
• So no single clear goal state
• Need to define transaction at subtask level
• What constitutes a transaction?
• Initiation and conclusion?
• Is miscommunication part of a transaction?
• What about subtasks?
• Subtask level transactions may also inform on
  which parts of the system may be problematic
• Start and end does not tell about dialogue
  smoothness

Provide users with required information
16
Smooth dialogues

• More precise overview of problems and their
  causes and seriousness
• Same topic may have fail and success in same call
• Few or many repairs
• distinction between unwanted and erroneous
  information
• erroneous information is unacceptable (tomorrow
  is Friday, phone [phone])
• other information than asked for may be more or
  less serious (fax instead of phone, fax instead
  of email)
• misunderstanding a yes for a no is usually not so
  serious (repairable) but can be a nuisance
• Misrecognitions
• Information blocks may contain more than asked for

You are way beyond your time frame!