Title: SIMS 256: Applied Natural Language Processing
1SIMS 256 Applied Natural Language Processing
Marti Hearst November 27, 2006
2Outline
- Discourse Processing
- Going beyond the sentence
- Characteristics
- Issues
- Segmentation
- Linear
- Hierarchical
- Co-reference / anaphora resolution
- Dialogue Processing
3What makes a text/dialogue coherent?
- Consider, for example, the difference between
passages (18.71) and (18.72). Almost certainly
not. The reason is that these utterances, when
juxtaposed, will not exhibit coherence. Do you
have a discourse? Assume that you have collected
an arbitrary set of well-formed and independently
interpretable utterances, for instance, by
randomly selecting one sentence from each of the
previous chapters of this book. - vs.
4What makes a text/dialogue coherent?
- Assume that you have collected an arbitrary
set of well-formed and independently
interpretable utterances, for instance, by
randomly selecting one sentence from each of the
previous chapters of this book. Do you have a
discourse? Almost certainly not. The reason is
that these utterances, when juxtaposed, will not
exhibit coherence. Consider, for example, the
difference between passages (18.71) and (18.72).
(JM695)
5What makes a text coherent?
- Discourse/topic structure
- Appropriate sequencing of subparts of the
discourse - Rhetorical structure
- Appropriate use of coherence relations between
subparts of the discourse - Referring expressions
- Words or phrases, the semantic interpretation of
which is a discourse entity
6Information Status
- Contrast
- John wanted a poodle but Becky preferred a corgi.
- Topic/comment
- The corgi they bought turned out to have fleas.
- Theme/rheme
- The corgi they bought turned out to have fleas.
- Focus/presupposition
- It was Becky who took him to the vet.
- Given/new
- Some wildcats bite, but this wildcat turned out
to be a sweetheart. - Contrast Speaker (S) and Hearer (H)
7Determining Given vs. New
- Entities when first introduced are new
- Brand-new (H must create a new entity)
- I saw a dinosaur today.
- Unused (H already knows of this entity)
- I saw your mother today.
- Evoked entities are old -- already in the
discourse - Textually evoked
- The dinosaur was scaley and gray.
- Situationally evoked
- The light was red when you went through it.
- Inferrables
- Containing
- I bought a carton of eggs. One of them was
broken. - Non-containing
- A bus pulled up beside me. The driver was a
monkey.
8Given/New and Definiteness/Indefiniteness
- Subject NPs tend to be syntactically definite and
old - Object NPs tend to be indefinite and new
- I saw a black cat yesterday. The cat looked
hungry. - Definite articles, demonstratives, possessives,
personal pronouns, proper nouns, quantifiers like
all, every - Indefinite articles, quantifiers like some, any,
one signal indefinitenessbut. - This guy came into the room
9Discourse/Topic Structure
- Text Segmentation
- Linear
- TextTiling
- Look for changes in content words
- Hierarchical
- Grosz Sidners Centering theory
- Morris Hirsts algorithm
- Lexical chaining through Rogets thesaurus
- Hierarchical Relations
- Mann et al.s Rhetorical Structure Theory
- Marcus algorithm
10TextTiling (Hearst 94)
- Goal find multi-paragraph topics
- Example 21 paragraph article called Stargazers
11TextTiling (Hearst 94)
- Goal find multi-paragraph topics
- But its difficult to define topic (Brown
Yule) - Focus instead on topic shift or change
- Change in content, by contrast with setting,
scene, characters - Mechanism
- compare adjacent blocks of text
- look for shifts in vocabulary
12Intuition behind TextTiling
13TextTiling Algorithm
- Tokenization
- Lexical Score Determination
- Blocks
- Vocabulary Introductions
- Chains
- Boundary Identification
14Tokenization
- Convert text stream into terms (words)
- Remove stop words
- Reduce to root (inflectional morphology)
- Subdivide into token-sequences
- (substitute for sentences)
- Find potential boundary points
- (paragraphs breaks)
15Determining Scores
- Compute a score at each token-sequence gap
- Score based on lexical occurrences
- Block algorithm
16(No Transcript)
17Boundary Identification
- Smooth the plot (average smoothing)
- Assign depth score at each token-sequence gap
- Deeper valleys score higher
- Order boundaries by depth score
- Choose boundary cut off (avg-sd/2)
18Evaluation
- Data
- Twelve news articles from Dialog
- Seven human judges per article
- major boundaries chosen by gt 3 judges
- Avg number of paragraphs 26.75
- Avg number of boundaries 10 (39)
- Results
- Between upper and lower bounds
- Upper bound judges averages
- Lower bound reasonable simple algorithm
19Assessing Agreement Among Judges
- KAPPA Coefficient
- Measures pairwise agreement
- Takes expected chance agreement into account
- P(A) proportion of times judges agree
- P(E) proportion expected chance agreement
- .43 to .68 (Isard Carletta 95, boundaries)
- .65 to .90 (Rose 95, sentence segmentation)
- Here, k .647
20TextTiling Conclusions
- First computational investigation into
multi-paragraph discourse units - Simple Discourse Cue position-sensitive term
repetition - Acceptable performance for some tasks
- Has been reproduced/used by many researchers
- Multi-lingual
- (applied by others to French, German, Arabic)
21What Can Hierarchical Structure Tell Us?
- Welcome to word processing.
- Thats using a computer to type letters and
reports. Make a typo? - No problem.
- Just back up, type over the mistake, and its
gone. - ?And, it eliminates retyping.
- ?And, it eliminates retyping.
22Centering Theory of Discourse Structure (Grosz
Sidner 86)
- A prominent theory of discourse structure
- Provides for multiple levels of analysis Ss
purpose as well as content of utterances and S
and Hs attentional state - Identifies only a few, general relations that
hold among intentions - Often leads to a hierarchical structure
- Three components
- Linguistic structure
- Intentional structure
- Attentional structure
23Example of Hierarchical Analysis(Morris and
Hirst 91)
24(No Transcript)
25Rhetorical Structure Theory (Mann, Matthiessen,
and Thompson 89)
- One theory of discourse structure, based on
identifying relations between parts of the text - Identify meaningful units and the relations
between them - Clauses and clause-like units that are
unequivocally the nucleus or satellite of a
rhetorical relation. - Only the midday sun at tropical latitudes is warm
enough to thaw ice on occasions, but any
liquid water formed in this way would evaporate
almost instantly because of the low atmospheric
pressure. - Nucleus/satellite notion encodes asymmetry
26Rhetorical Structure Theory
- Some rhetorical relations
- Elaboration (set/member,class/instance/whole/part
) - Contrast multinuclear
- Condition Sat presents precondition for N
- Purpose Sat presents goal of the activity in N
- Sequence multinuclear
- Result N results from something presented in Sat
- Evidence Sat provides evidence for something
claimed in N
27Determining high-level relations
Smart cards are not a new phenomenon.1 They
have been in development since the late 1970s and
have found major applications in Europe, with
more than a quarter of a billion cards made so
far.2 The vast majority of chips have gone into
prepaid, disposable telephone cards, but even so
the experience gained has reduced manufacturing
costs, improved reliability and proved the
viability of smart cards.3 International and
national standards for smart cards are well under
development to ensure that cards, readers and the
software for the many different applications that
may reside on them can work together seamlessly
and securely.4 Standards set by the
International Organization for Standardization
(ISO), for example, govern the placement of
contacts on the face of a smart card so that any
card and reader will be able to connect.5
28Representing implicit relations
Smart cards are becoming more attractive2 as
the price of microcomputing power and storage
continues to drop.3 They have two main
advantages over magnetic-stripe cards.4 First,
they can carry 10 or even 100 times as much
information5 - and hold it much
more robustly.6 Second, they can execute
complex tasks in conjunction with a terminal.7
29Whats the Rhetorical Structure?
- System Hello. How may I help you?
- User I would like to find out why I was charged
for a call? - System What call would you like to inquire
about? - User My bill says I made a call to Syncamaloo,
Texas, but Ive never even heard of this town. - System May I have the date of the call that
appears on your bill?
30Issues for RST
- Many variations in expression
- I have not read this book. It was written by
Bertrand Russell. - I have not read this book, which was written
by Bertrand Russell. - I have not read this book written by Bertrand
Russell. - I have not read this Bertrand Russell book.
- Rhetorical relations are ambiguous
- He caught a bad fever while he was in Africa.
- Circumstance gt Temporal-Same-Time
- With its distant orbit, Mars experiences frigid
weather conditions. Surface temperatures
typically average about 60 degrees Celsius at
the equator and can dip to 123 degrees C near
the poles. - Evidence gt Elaboration
31Identifying RS Automatically (Marcu 99)
- Train a parser on a discourse treebank
- 90 RS trees, hand-annotated for rhetorical
relations - Elementary discourse units (edus) linked by RR
- Parser learns to identify N and S and their RR
- Features Wordnet-based similarity, lexical,
structural - Uses discourse segmenter to identify discourse
units - Trained to segment on hand-labeled corpus (C4.5)
- Features 5-word POS window, presence of
discourse markers, punctuation, seen a verb?, - Eval 96-8 accuracy
32Identifying RS Automatically (Marcu 99)
- Evaluation of parser
- Id edus Recall 75, Precision 97
- Id hierarchical structure (2 edus related)
Recall 71, Precision 84 - Id nucleus/satellite labels Recall 58,
Precision 69 - Id RR Recall 38, Precision 45
- Later errors due mostly to edu mis-identification
- Id of hierarchical structure and n/s status
comparable to human when hand-labeled edus used - Hierarchical structure is easier to id than RR
33Some Problems with RST (cf. Moore Pollack 92)
- How many Rhetorical Relations are there?
- How can we use RST in dialogue as well as
monologue? - RST does not allow for multiple relations holding
between parts of a discourse - RST does not model overall structure of the
discourse
34Referring Expressions
- Referring expressions are words or phrases, the
semantic interpretation of which is a discourse
entity - (also called referent)
- Discourse entities are semantic objects .
- Can have multiple syntactic realizations within
a text - Discourse entities exist in the domain D, in
which a text is interpreted
35Referring Expressions Example
- A pretty woman entered the restaurant. She sat at
the table next to mine and only then I recognized
her. This was Amy Garcia, my next door neighbor
from 10 years ago. The woman has totally changed!
Amy was at the time shy
36Pronouns vs. Full NP
- A pretty woman entered the restaurant. She sat at
the table next to mine and only then I recognized
her. This was Amy Garcia, my next door neighbor
from 10 years ago. The woman has totally changed!
Amy was at the time shy
37Definite vs. Indefinite NPs
- A pretty woman entered the restaurant. She sat at
the table next to mine and only then I recognized
her. This was Amy Garcia, my next door neighbor
from 10 years ago. The woman has totally changed!
Amy was at the time shy
38Common Noun vs. Proper Noun
- A pretty woman entered the restaurant. She sat at
the table next to mine and only then I recognized
her. This was Amy Garcia, my next door neighbor
from 10 years ago. The woman has totally changed!
Amy was at the time shy
39Modified vs. Bare head NP
- A pretty woman entered the restaurant. She sat at
the table next to mine and only then I recognized
her. This was Amy Garcia, my next door neighbor
from 10 years ago. The woman has totally changed!
Amy was at the time shy
40Premodified vs. Postmodified
- A pretty woman entered the restaurant. She sat at
the table next to mine and only then I recognized
her. This was Amy Garcia, my next door neighbor
from 10 years ago. The woman has totally changed!
Amy was at the time shy
41Anaphora resolution
- Finding in a text all the referring expressions
that have one and the same denotation - Pronominal anaphora resolution
- Anaphora resolution between named entities
- Full noun phrase anaphora resolution
42Anaphora Resolution
- A pretty woman entered the restaurant. She sat at
the table next to mine and only then I recognized
her. This was Amy Garcia, my next door neighbor
from 10 years ago. The woman has totally changed!
Amy was at the time shy
43Pronominal anaphora resolution
- Rule-based vs statistical
- (Ken 1996), (Lap 1994) vs (Ge 1998)
- Performed on full syntactic parse vs on shallow
syntactic parse - (Lap 1994), (Ge 1998) vs (Ken 1996)
- Type of text used for the evaluation
- (Lap 1994) computer manual texts (86 accuracy)
- (Ge 1998) WSJ articles (83 accuracy)
- (Ken 1996) different genres (75 accuracy)
44Pronominal anaphora resolution
- Generic vs specific reference
- 1. The Vice-President of the United States is
also President of the Senate. - 2. Historically, he is the Presidents key person
in negotiations with Congress - 3a. He is required to be 35 years old.
- 3b. As Ambassador to China, he handled many
tricky negotiations, so he is well prepared for
the job
45Talking to a Machine.and (often) Getting an
Answer
- Todays spoken dialogue systems make it possible
to accomplish real tasks without talking to a
person - Key advances
- Stick to goal-directed interactions in a limited
domain - Prime users to adopt the vocabulary you can
recognize - Partition the interaction into manageable stages
- Judicious use of system vs. mixed initiative
46Acoustic and Prosodic Cues to Discourse Structure
- Intuition
- Speakers vary acoustic and prosodic cues to
convey variation in discourse structure - Systematic? In read or spontaneous speech?
- Evidence
- Observations from recorded corpora
- Laboratory experiments
- Machine learning of discourse structure from
acoustic/prosodic features
47Boston Directions Corpus (Hirschberg Nakatani
96)
- Experimental Design
- 12 speakers 4 used
- Spontaneous and read versions of 9
direction-giving tasks - Corpus 50m read 67m spon
- Labeling
- Prosodic ToBI intonational labeling
- Discourse Grosz Sidner
48Boston Directions Corpus Describe how to get to
MIT from Harvard
- ds1 step 1, enter and get token
- first
- enter the Harvard Square T stop
- and buy a token
- ds2 inbound on red line
- then
- proceed to get on the
- inbound
- um
- Red Line
- uh subway
49- ds3 take subway from hs, to cs to ks
- and
- take the subway
- from Harvard Square
- to Central Square
- and then to Kendall Square
- ds4 describe ks station
- youll see a music sculpture there
- which will tell you its Kendall Square
- its very nice
- ds5 get off T.
- then get off the T
50Dialogue vs. Monologue
- Monologue and dialogue both involve interpreting
- Information status
- Coherence issues
- Reference resolution
- Speech acts, implicature, intentionality
- Dialogue involves managing
- Turn-taking
- Grounding and repairing misunderstandings
- Initiative and confirmation strategies
51Segmenting Speech into Utterances
- What is an utterance?
- Why is EOU detection harder than EOS?
- How does speech differ from text?
- Single syntactic sentence may span several turns
- A We've got you on USAir flight 99
- B Yep
- A leaving on December 1.
- Multiple syntactic sentences may occur in single
turn - A We've got you on USAir flight 99 leaving on
December. Do you need a rental car? - Intonational definitions intonational phrase,
breath group, intonation unit
52Turns and Utterances
- Dialogue is characterized by turn-taking who
should talk next, and when they should talk - How do we identify turns in recorded speech?
- Little speaker overlap (around 5 in English
--although depends on domain) - But little silence between turns either
- How do we know when a speaker is giving up or
taking a turn? Holding the floor? How do we
know when a speaker is interruptable?
53Simplified Turn-Taking Rule (Sacks et al)
- At each transition-relevance place (TRP) of each
turn - If current speaker has selected A as next
speaker, then A must speak next - If current speaker does not select next speaker,
any other speaker may take next turn - If no one else takes next turn, the current
speaker may take next turn - TRPs are where the structure of the language
allows speaker shifts to occur
54- Adjacency pairs set up next speaker expectations
- GREETING/GREETING
- QUESTION/ANSWER
- COMPLIMENT/DOWNPLAYER
- REQUEST/GRANT
- Significant silence is dispreferred
- A Is there something bothering you or not?
(1.0s) - A Yes or no? (1.5s)
- A Eh?
- B No.
55Turntaking and Initiative Strategies
- System Initiative
- S Please give me your arrival city name.
- U Baltimore.
- S Please give me your departure city name.
- User Initiative
- S How may I help you?
- U I want to go from Boston to Baltimore on
November 8. - Mixed initiative
- S How may I help you?
- U I want to go to Boston.
- S What day do you want to go to Boston?
56Grounding (Clark Shaefer 89)
- Conversational participants dont just take turns
speaking.they try to establish common ground (or
mutual belief) - H must ground a S's utterances by making it clear
whether or not understanding has occurred - How do hearers do this?
- Several different mechanisms
57Grounding Mechanisms(Clark Shaefer 89)
- S I can upgrade you to an SUV at that rate.
- Continued attention
- (U gazes appreciatively at S)
- Relevant next contribution
- U Do you have a RAV4 available?
- Acknowledgement/backchannel
- U Ok/Mhmmm/Great!
- Demonstration/paraphrase
- U An SUV.
- Display/repetition
- U You can upgrade me to an SUV at the same rate?
- Request for repair
- U I beg your pardon?
58How do we evaluate Dialogue Systems?
- PARADISE framework (Walker et al 00)
- Performance of a dialogue system is affected
both by what gets accomplished by the user and
the dialogue agent and how it gets accomplished - Efficiency of the InteractionUser Turns, System
Turns, Elapsed Time - Quality of the Interaction ASR rejections, Time
Out Prompts, Help Requests, Barge-Ins, Mean
Recognition Score (concept accuracy),
Cancellation Requests - User Satisfaction
- Task Success perceived completion, information
extracted
59Identifying Misrecognitions and User Corrections
Automatically (Hirschberg, Litman Swerts)
- Collect corpus from interactive voice response
system - Identify speaker turns
- incorrectly recognized
- where speakers first aware of error
- that correct misrecognitions
- Identify prosodic features of turns in each
category and compare to other turns - Use Machine Learning techniques to train a
classifier to make these distinctions
automatically
60Turn Types
TOOT Hi. This is ATT Amtrak Schedule System.
This is TOOT. How may I help you? User Hello.
I would like trains from Philadelphia to New York
leaving on Sunday at ten thirty in the evening.
TOOT Which city do you want to go to? User
New York.
misrecognition
correction
aware site
61Results
- Reduced error in predicting misrecognized turns
to 8.64 - Error in predicting awares (12)
- Error in predicting corrections (18-21)
62Dialogue Conclusions
- Spoken dialogue systems presents new problems --
but also new possibilities - Recognizing speech introduces a new source of
errors - Additional information provided in the speech
stream offers new information about users
intended meanings, emotional state (grounding of
information, speech acts, reaction to system
errors) - Why spoken dialogue systems rather than web-based
interfaces?