Learning to Generate Utterances for Spoken Dialogue Systems by Mining User Reviews

1
Learning to Generate Utterances for Spoken
Dialogue Systems by Mining User Reviews

• Prof. Marilyn Walker
• (Higashinaka, Prasad and Walker, COLING/ACL 2006)

2
Background

• Statistical methods are predominant in Spoken
  Dialogue Systems (SDS) and are quite mature for
  speech recognition, language understanding, and
  dialogue act detection tasks (Young 2004)
• Statistical methods also commonly used in NLP
  tasks such as information retrieval, automatic
  summarization, information extraction and machine
  translation
• These methods provide scalability and portability
  across domains

3
Spoken Dialogue System Components
4
Spoken Language Generation

• DM and SLG typically handcrafted
• Template Based Generation (usual)
• Used in most dialogue systems, generate responses
  simply matching a template with the current
  dialogue context
• Pros efficient, highly customized
• Cons not portable cross domain, hard to encode
  linguistic constraints (subject/verb agreement),
  non scalable when more than few hundred
  templates
• Natural Language Generation (recently)
• Clear separation between 1) text (or content)
  planning, 2) sentence planning, 3) surface
  realization. Uses general rules for each
  generation module
• Pros some aspects are portable cross domain and
  dialogue context
• Cons Specific rules are usually needed to tune
  the quality of the general rules, could be slow
  for real-time systems

5
NLG System Components
DM Communicative Goals
How to say it
NLG
Text Planner What to say
How to say it
Sentence Planner
Surface Realizer
Prosody Assigner
TTS
6
Spoken Dialogue Generation

• Dialogue Manager Select communicative goals
• Text Planning Refine communicative goal into
  structured atomic communicative goals, select
  content to be expressed
• Sentence planning choose linguistic resources
  (lexicon, syntax) to achieve atomic communicative
  goals
• Realization use grammar (syntax, morphology) to
  generate surface sentence(s)
• (Rambow Korelsky, 1992 Reiter, 1994)
  Consensus Architecture

7
Statistical Methods in SLG/NLG

• Statistical Surface Realizers overgenerate and
  rank using parameters trained from corpora
  (Halogen, Langkilde 2002 Fergus, Bangalore and
  Rambow 2000 Chen etal 2002 )
• Trainable Sentence Planning learn which
  combination operations for aggregation and
  content ordering produce highest quality output
  (SPoT, Rambow etal 2002, Walker etal 2004
  SPaRKy, Stent etal. 2004)
• Prosody Assignment learn from labelled data to
  assign appropriate prosody (Hirschberg90, Pan
  etal. 2003)

8
Problem

• Even the trainable sentence planner requires a
  domain-specific handcrafted generation dictionary
  to specify the mapping between syntactic
  realizations and the text plan propositions
  (e.g., X has good food, X has good service)
• Mappings are created by hand ? It is costly,
  needed for each domain
• Variation limited by original mappings and
  combination operations ? Utterances can be
  unnatural

9
Semantic Representations
Example SLG
Joint relation (RST)
Assert-food_quality(Babbo,
superb)
Assert-décor(Babbo, superb)
Dictionary
have I proper_noun X II
common_noun food ATTR adjective
superb
Assert-food_quality(X, superb) ?
have I proper_noun X II
common_noun decor ATTR adjective
superb
Assert-décor(X, superb) ?
Utterance
have I proper_noun X II
common_noun food COORD and
II common_noun décor
ATTR adjective superb
Babbo has superb food and
decor.
10
Solution?

• In many domains, there are web pages that
  describe and evaluate domain entities
• These web pages may include
• Textual reviews of domain entities
• Scalar ratings of specific attributes of domain
  entities on a per review or per entity basis
• Tabular data with values for particular
  attributes
• Domain or product specific ontologies
• Is it possible to mine these corpora to bootstrap
  a spoken language generator?

11
Restaurant Domain
12
Sample Restaurant Review
13
Hotel Domain

• a little gem!!
• Submitted by kathy b. of cincinnati,  oh usa
   May 03, 2006
• Date of visit 12/05
• Traveler's Rating 5
• the history, ambience and old world charm of the
  algonquin are a unique combination that appeals
  very much. staff is very friendly and helpful
  rooms small but restored to period charm. great
  lobby. say hello to matilda, the resident cat,
  another great algonquin tradition.
• Best Feature staff ambience
• Needs Improvement not a thing
• Amenities rated on a 1-5 scale
• (1Lowest 5Highest N/ANot Rated)
• Rooms 4
• Dining 5
• Public Facilities 5 Sports/Activities N/A
• Entertainment 4
• Service 5

14
What can be used

• Textual reviews of domain entities
• Scalar ratings of specific attributes of domain
  entities on a per review or per entity basis
• Tabular data with values for categorial
  attributes
• Specified attributes gt partial ontology

15
Bootstrapping SLG

• Automatically acquire dictionary entries from
  user reviews on the web
• Dictionary entry is a triple (U,R,S)
• U (Utterance), R (Semantic representation), and S
  (Syntactic structure)
• Use user ratings and categorial attribute values
  to pinpoint semantic representation
• Use Minipar parser and DSyntS converter to
  produce dependency syntactic representation
  (Lavoie and Rambow 98, Melcuk, 1988)

16
Related Work

• Create dictionary from parallel corpus (Barzilay
  et al., 2002)
• ? Requires a corpus of parallel semantic
  representation/syntactic realizations
• Information Extraction Find phrases/patterns
  expressing particular relations
• ? Dont know entities that realize the
  relations, want most frequent/general pattern
• Find opinion expressions in reviews
• Adjectives for products (Hu and Liu, 2005)
• Product features and adjectives with polarity
  (Popescu and Etzioni, 2005)
• ? Do not focus on creating dictionary

17
Method

• Create a population of utterances U from user
  reviews
• For each U Derive semantic representation
  R Derive syntactic structure S
• Filter inappropriate mappings
• Add remaining mappings to dictionary

18
Experiment

• Obtaining dictionary for restaurant domain
• Data collected from we8there.com
• 3,004 user reviews on 1,810 restaurants
• 18,466 review sentences
• 451 mappings after filtering
• Objective evaluation
• Subjective evaluation

19
Collect user reviews

• Select review websites with individual ratings
  for review entities
• Collect review comments and ratings
• Collect tabular data

Ratings Food, Service, Value, Atmosphere, Overall
Tabular Data Name, Food Type, Location
20
Derive Domain Ontology

• Assume meronymy relation between
• Any attribute that the user rates
• Any attribute for which categorical values are
  specified on the web page
• Relations

RESTAURANT has foodquality RESTAURANT has
servicequality RESTAURANT has valuequality RESTAUR
ANT has atmospherequality RESTAURANT has
overallquality RESTAURANT has FOODTYPE RESTAURANT
has LOCATION
21
Hypothesis/Assumptions

• Closed Domain At least some of the utterances in
  reviews realize the relations in the domain
  ontology (identify these utterances)
• Hypoth If an utterance U realizes named entities
  corresponding to the domain entity and the
  distinguished attributes, then R for that
  utterance includes the relation concerning that
  attribute in the domain ontology.

22
Specify Lexicalizations of attributes
Attribute Lexicalizations
food food, meal
service service, staff, waitstaff, wait staff, server, waiter, waitress
atmosphere atmosphere, décor, ambience, decoration
value value, price, overprice, pricey, expensive, inexpensive, cheap, affordable, afford
overall recommend, place, experience, establishment
23
Create and Label Named Entities

• Scrape web pages structured data for named
  entities for categorial attributes
• Foodtype gt Spanish, Italian, French,
• Location gt New York, San Francisco, London
• Run Gate on U to label
• Named entities
• Lexicalizations of attributes

24
Derive Syntactic Representation

• Run Minipar parser
• Convert Minipar output to DSyntS for RealPro
  (Lavoie and Rambow 98)

25
Ratings Food5, Service5, Value5, Atmosphere5,
Overall5
Review Comment The best Spanish food in New York.
I am from Spain and I had my 28th birthday
Review Sentence (U)
DSyntS converter
The best Spanish food in New York.
DSyntS (S)
NE-tagged Review Sentence
The best NEfoodtype, stringSpanish NEfood,
stringfood, rating5 in NElocation,
stringNew York.
Semantic Representation (R)
RESTAURANT has FOODTYPE RESTAURANT has
foodquality5 RESTAURANT has LOCATION
26
Filter dictionary entries

• No Relations Filter, Other Relations Filter
• ? Check whether a mapping has exactly the
  relations expressed in the ontology
• Contextual Filter
• ? Check whether U can be uttered independently
  of the context (looks for context words)
• Parsing Filter
• ? Check whether S(U) regenerates U
• Unknown Words Filter (typos, common nouns, etc.)

27
Filtering Statistics
filtered retained
No Relations Filter 7,947 10,519
Other Relations Filter 5,351 5,168
Contextual Filter 2,973 2,195
Unknown Words Filter 1,467 728
Parsing Filter 216 512
Duplicates Filter 61 451
28
Filtering Examples
U
The river was beautiful and the food okay.
We had a wonderful time.
What an awful place.
R
RESTAURANT has foodquality3
RESTAURANT has overallquality1
S
Filtered by Unknown Words Filter
Filtered by No Relations Filter
(river is a common noun)
Filtered by Parsing Filter
This DSyntS generates What awful place.
29
Objective Evaluation

• Domain coverage
• How many relations are covered by the dictionary?
• Linguistic variation
• What do we gain over handcrafted dictionary?
• Generativity
• Can the dictionary entries can be used in
  conventional sentence planner ?

30
Domain Coverage
Distribution of single scalar-valued relation
mappings
1 2 3 4 5 Total
food 5 8 6 18 57 94
service 15 3 6 17 56 97
atmosphere 0 3 3 8 31 45
value 0 0 1 8 12 21
overall 3 2 5 15 45 70
Total 23 15 21 64 201 327
31
Multi-Relation Entries (122 in all)

• Food-service 39
• Food-value 21
• Atmosphere-food 14
• Atmosphere-service 10
• Atmosphere-food-service 7
• Food-Foodtype 4
• Atmosphere-food-value 4
• etc

32
Linguistic Variation

• 137 syntactic patterns
• 275 distinct lexemes, 2-15 lexemes per DSyntS
  (mean 4.63)
• 55 syntactic patterns ATTR is AP
• The atmosphere is wonderful.
• The food is excellent and the atmosphere is great
• 45 are not.
• An absolutely outstanding value with fantastic
  foodtype food.

33
Examples Food Adjectival phrases Attribute
specificity

• RATING ADJECTIVE
• 1awful, bad, cold, burnt, very ordinary
• 2 acceptable, bad, flavored, not enough very
  bland, very good
• 3 adequate, bland and mediocre, flavorful but
  cold, pretty good, rather bland, very good
• 4 absolutely wonderful, awesome, decent,
  excellent, good and generatou, very fresh and
  tasty
• 5 absolutely delicious, ample, well seasoned and
  hot, delicious but simple, delectable and
  plentiful, fancy but tasty, so very tasty

34
Example Service Adjectival phrases

• RATING ADJECTIVE
• 1 awful, bad, forgetful and slow, marginal,
  young, silly and inattentive
• 2 overly slow, very slow and inattentive
• 3 bland and mediocre, friendly and
  knowledgeable, pleasant, prompt
• 4 all very warm and welcoming, attentive,
  extremely friendly and good, great and courteous,
  swift and friendly, very friendly and
  accommodating
• 5 polite, great, all courteous, excellent and
  friendly, fabulous, impeccable, intrusive,
  legendary, very friendly, very friendly and
  totally personal, very helpful, very timely

35
Example Atmosphere Adjectival phrases

• RATING ADJECTIVE
• 2 eclectic, unique and pleasant
• 3 busy, pleasant but extremely hot
• 4 fantastic, great, quite nice and simple,
  typical, very casual, very trendy
• 5 beautiful, comfortable, lovely, mellow, nice
  and comfortable, very cozy, very intimate, very
  relaxing, warm and contemporary

36
Generativity

• Incorporate the learned mappings into SPaRKy
  generator (Stent et al., 2004)
• Combination operations need extension because of
  assumption that restaurant name is the subject of
  the utterance
• ORIGINAL Because it has excellent food, superb
  service and excellent décor, Babbo has the best
  overall quality among the selected restaurants.
• MODIFIED Because the food is excellent, the wait
  staff is professional and the decor is beautiful
  and very comfortable, Babbo has the best overall
  quality among the selected restaurants

37
Subjective Evaluation

• 10 native English speakers
• Compare baseline and learned mappings
• 27 hand-crafted mappings from SPaRKy
• 451 learned mappings
• Evaluation criteria
• Consistency between semantic representations and
  realizations
• Naturalness/colloquialness of realizations
• 1-5 Likert scale

38
Results
Consistency
Naturalness
4.71
4.61
4.46
4.23
baseline
learned
baseline
learned

• Consistency is significantly lower, but still
  high
• Naturalness is significantly higher

39
Conclusion

• A new method for automatically acquiring a
  generation dictionary in spoken dialogue systems
• Applied to hotel domain and had results in one
  day
• Reduce the cost involved with hand-crafting a
  spoken language generation module
• Achieve more natural system utterances using
  attested language examples
• Results suggest that this approach is promising

40
Future Work

• Issues of meaning vs. polarity, not
  substitutable
• Handcrafted lexicalizations need to
  automatically generate lexicalizations for domain
  concepts (increase recall?)
• Method for extending domain ontology (food is
  plentiful, delicious, beautifully prepared)
• More complex domains