Stochastic Language Generation for Spoken Dialog Systems

1
Stochastic Language Generation for Spoken Dialog
Systems

• Alice Oh
• aliceo_at_cs.cmu.edu
• 30 October 2009

2
Problem Statement

• Traditional NLG produces high-quality output
• but requires substantial knowledge engineering
• Template-based NLG often does not deliver quality
  
• but is simple to build
• Can we combine the advantages of the two
  approaches?

3
Stochastic NLG overview

• Language Model an n-gram language model built
  from a corpus of travel reservation dialogs
• Generation given an utterance class, randomly
  generates a set of candidate utterances based on
  the LM distributions
• Scoring based on a set of rules, scores the
  candidates and picks the best one
• Slot filling substitute slots in the utterance
  with the appropriate values in the input frame

4
Stochastic NLG Details

• Language Model travel agent/client dialogs
• Generation Scoring 75 msec
• Slot-Filling
• What time do you need to arrive in arrive_city?
• What time do you need to arrive in New York?

5
Preliminary Evaluation

• Batch-mode generation using two systems,
  comparative evaluation of output by human
  subjects
• User Preferences (49 utterances total)
• Weak preference for Stochastic NLG (p 0.18)

subject
stochastic
templates
difference
1
41
8
33
2
34
15
19
3
17
32
-15
4
32
17
15
5
30
17
13
6
27
19
8
7
8
41
-33
average
27
21.29
5.71
6
Stochastic NLG Advantages

• corpus-driven
• easy to build (minimal knowledge engineering)
• fast prototyping
• minimal input (speech act, slot values)
• natural output
• leverages data-collecting/tagging effort

7
Stochastic NLG Shortcomings

• Output Quality What might sound natural
  (imperfect grammar, intentional omission of
  words, etc.) for a human speaker may sound
  awkward (or wrong) for the system.
• Corpus Markup It is difficult to define
  utterance boundaries and utterance classes. Some
  utterances in the corpus may be a conjunction of
  more than one utterance class.
• Context Other Factors
• Factors other than the utterance class may affect
  the words (e.g., discourse history).
• Some sophistication built into traditional NLG
  engines is not available (e.g., aggregation,
  anaphorization).

8
Future Work

• How big of a corpus do we need?
• How can tagging be automated?
• How does the n in n-gram affect the output
  quality?
• What happens to output when two different human
  speakers are combined in one model?
• Can we replace generate-and-test with an
  efficient search algorithm?

9
Evaluation

• Must be able to evaluate generation independent
  of the rest of the dialog system
• Comparative evaluation using dialog transcripts
• need 15-20 subjects
• 8-10 dialogs system output generated batch-mode
  by two different engines