Classification of Discourse Coherence Relations: An Exploratory Study using Multiple Knowledge Sources

1
Classification of Discourse Coherence Relations
An Exploratory Study using Multiple Knowledge
Sources

• Ben Wellner, James Pustejovsky, Catherine
  Havasi, Anna Rumshisky and Roser Saurí

Brandeis University The MITRE Corporation
2
Outline of Talk

• Overview and Motivation for Modeling Discourse
• Background
• Objectives
• The Discourse GraphBank
• Overview
• Coherence Relations
• Issues with the GraphBank
• Modeling Discourse
• Machine learning approach
• Knowledge Sources and Features
• Experiments and Analysis
• Conclusions and Future Work

3
Modeling Discourse Motivation

• Why model discourse?
• Dialogue
• General text understanding applications
• Text summarization and generation
• Information extraction
• MUC Scenario Template Task
• Discourse is vital for understanding how events
  are related
• Modeling discourse generally may aid specific
  extraction tasks

4
Background

• Different approaches to discourse
• Semantics/formalisms Hobbs 1985, Mann and
  Thomson1987, Grosz and Sidner1986, Asher
  1993, others
• Different objectives
• Informational vs. intentional, dialog vs. general
  text
• Different inventories of discourse relations
• Coarse vs. fine-grained
• Different representations
• Tree representation vs. Graph
• Same steps involved
• 1. Identifying discourse segments
• 2. Grouping discourse segments into sequences
• 3. Identifying the presence of a relation
• 4. Identifying the type of the relation

5
Discourse Steps 1
Mary is in a bad mood because Fred played tuba
while she was taking a nap.
1. Segment
A
B
C
r1
2. Group
r2
A
3. Connect segments
B
C
r1 cause-effect r2 elaboration
4. Relation Type
Example from Danlos 2004
6
Discourse Steps 2
Fred played the tuba. Next he prepared a pizza
to please Mary.
1. Segment
A
B
C
r1
r2
2. Group
3. Connect segments
A
B
C
r1 temporal precedence r2 cause-effect
4. Relation Type
Example from Danlos 2004
7
Objectives

• Our Main Focus Step 4 - classifying discourse
  relations
• Important for all approaches to discourse
• Can be approached independently of representation
• But relation types and structure are probably
  quite dependent
• Task will vary with inventory of relation types
• What types of knowledge/features are important
  for this task
• Can we apply the same approach to Step 3
• Identifying whether two segment groups are linked

8
Discourse GraphBank Overview
Wolf and Gibson, 2005

• Graph-based representation of discourse
• Tree-representation inadequate multiple parents,
  crossing dependencies
• Discourse composed of clausal segments
• Segments can be grouped into sequences
• Relations need not exist between segments within
  a group
• Coherence relations between segment groups
• Roughly those of Hobbs 1985
• Why GraphBank?
• Similar inventory of relations as SDRT
• Linked to lexical representations
• Semantics well-developed
• Includes non-local discourse links
• Existing annotated corpus, unexplored outside of
  Wolf and Gibson, 2005

9
Resemblance Relations
The first flight to Frankfurt this morning was
delayed. The second flight arrived late as well.
Similarity (parallel)
The first flight to Frankfurt this morning was
delayed. The second flight arrived on time.
Contrast
There have been many previous missions to Mars. A
famous example is the Pathfinder mission.
Example
Generalization
Two missions to Mars in 1999 failed. There are
many missions to Mars that have failed.
A probe to Mars was launched from the Ukraine
this week. The European-built Mars Express is
scheduled to reach Mars by Dec.
Elaboration
The elaboration relation is given one or more
sub-types organization, person, location, time,
number, detail
10
Causal, Temporal and Attribution Relations
Cause-effect
There was bad weather at the airport and so our
flight got delayed
Causal
If the new software works, everyone should be
happy.
Conditional
The new software worked great, but nobody was
happy.
Violated Expectation
First, John went grocery shopping. Then, he
disappeared into a liquor store.
Temporal
Precedence
John said that the weather would be nice tomorrow.
Attribution
Attribution
The economy, according to analysts, is expected
to improve by early next year.
Same
11
Some Issues with GraphBank

• Coherence relations
• Conflation of actual causation and
  intention/purpose
• Granularity
• Desirable for relations hold between
  eventualities or entities, not necessarily entire
  clausal segments

The university spent 30,000
to upgrade lab equipment in 1987
cause
?? John pushed the door
to open it.
cause
elaboration
the new policy came about after President
Reagans historic decision in mid-December to
reverse the policy of refusing to deal with
members of the organization,
long shunned as a band of terrorists. Reagan
said PLO chairman Yasser Arafat had met US
demands.
12
A Classifier-based Approach

• For each pair of discourse segments, classify
  relation type between them
• For segment pairs on which we know a relation
  exists
• Advantages
• Include arbitrary knowledge sources as features
• Easier than implementing inference on top of
  semantic interpretations
• Robust performance
• Gain insight into how different knowledge sources
  contribute
• Disadvantages
• Difficult to determine why mistakes happen
• Maximum Entropy
• Commonly used discriminative classifier
• Allows for a high-number of non-independent
  features

13
Knowledge Sources

• Knowledge Sources
• Proximity
• Cue Words
• Lexical Similarity
• Events
• Modality and Subordinating Relations
• Grammatical Relations
• Temporal relations
• Associate with each knowledge source
• One or more Feature Classes

14
Example
SEG2 The university spent 30000 SEG1 to
upgrade lab equipment in 1987
15
Proximity

• Motivation
• Some relations tend to be local i.e. Their
  arguments appear nearby in the text
• Attribution, cause-effect, temporal precedence,
  violated expectation
• Other relations can span larger portions of text
• Elaboration
• Similar, contrast

Feature Class
Proximity - Whether segments are adjacent or
not - Directionality (which argument appears
earlier in the text) - Number of intervening
segments
16
Example
SEG2 The university spent 30000 SEG1 to
upgrade lab equipment in 1987
Fea. Class Example Feature
Proximity adjacent distlt3 distlt5 direction-reverse same-sentence
17
Cue Words

• Motivation
• Many coherence relations are frequently
  identified by a discourse cue word or phrase
  therefore, but, in contrast
• Cues are generally captured by the first word in
  a segment
• Obviates enumerating all potential cue words
• Non-traditional discourse markers (e.g.
  adverbials or even determiners) may indicate a
  preference for certain relation types

Feature Class
Cue Words - First word in each segment
18
Example
SEG2 The university spent 30000 SEG1 to
upgrade lab equipment in 1987
Fea. Class Example Feature
Proximity adjacent distlt3 distlt5 direction-reverse same-sentence
Cue Words First1to First2The
19
Lexical Coherence

• Motivation
• Identify lexical associations, lexical/semantic
  similarities
• E.g. push/fall, crash/injure, lab/university
• Brandeis Semantic Ontology (BSO)
• Taxonomy of types (i.e. senses)
• Includes qualia information for words
• Telic (purpose), agentive (creation),
  constitutive (parts)
• Word Sketch Engine (WSE)
• Similarity of words as measured by their contexts
  in a corpus (BNC)

Feature Class
BSO - Paths between words up to length
10 WSE - Number of word pairs with similarity gt
0.05, gt 0.01 - Segment similarities (sum of
word-pair similarities / words)
20
Example
SEG2 The university spent 30000 SEG1 to
upgrade lab equipment in 1987
Fea. Class Example Feature
Proximity adjacent distlt3 distlt5 direction-reverse same-sentence
Cue Words First1to First2The
BSO Research LabgtEducational ActivitygtUniversity
WSE WSEgt0.05 WSE-sentence-similarity0.005417
21
Events

• Motivation
• Certain events and event-pairs are indicative of
  certain relation types (e.g. push-fall
  cause)
• Allow learner to associate events and event-pairs
  with particular relation types
• Evita EVents In Text Analyzer
• Performs domain independent identification of
  events
• Identifies all event-referring expressions (that
  can be temporally ordered)

Feature Class
Events - Event mentions in each segment -
Event mention pairs drawn from both segments
22
Example
SEG2 The university spent 30000 SEG1 to
upgrade lab equipment in 1987
Fea. Class Example Feature
Proximity adjacent distlt3 distlt5 direction-reverse same-sentence
Cue Words First1to First2The
BSO Research LabgtEducational ActivitygtUniversity
WSE WSEgt0.05 WSE-sentence-similarity0.005417
Events Event1upgrade event2spent event-pairupgrade-spent
23
Modality and Subordinating Relations

• Motivation
• Event modality and subordinating relations are
  indicative of certain relations
• SlinkET Saurí et al. 2006
• Identifies subordinating contexts and classifying
  as
• Factive, counter-factive, evidential, negative
  evidential, or modal
• E.g. evidential gt attribute relation
• Event class, polarity, tense, etc.

Feature Class
SlinkET - Event class, polarity, tense and
modality of events in each segment -
Subordinating relations between event pairs
24
Example
SEG2 The university spent 30000 SEG1 to
upgrade lab equipment in 1987
Fea. Class Example Feature
Proximity adjacent distlt3 distlt5 direction-reverse same-sentence
Cue Words First1to First2The
BSO Research LabgtEducational ActivitygtUniversity
WSE WSEgt0.05 WSE-sentence-similarity0.005417
Events Event1upgrade event2spent event-pairupgrade-spent
SlinkET Class1occurrence Class2occurrence Tense1infinitive Tense2past modal-relation
25
Cue Words and Events

• Motivation
• Certain events (event types) are likely to appear
  in particular discourse contexts keyed by certain
  connectives.
• Pairing connectives with events captures this
  more precisely than connectives or events on
  their own

Feature Class
CueWords Events - First word of SEG1 and
each event mention in SEG2 - First word of
SEG2 and each event mention in SEG1
26
Example
SEG2 The university spent 30000 SEG1 to
upgrade lab equipment in 1987
Fea. Class Example Feature
Proximity adjacent distlt3 distlt5 direction-reverse same-sentence
Cue Words First1to First2The
BSO Research LabgtEducational ActivitygtUniversity
WSE WSEgt0.05 WSE-sentence-similarity0.005417
Events Event1upgrade event2spent event-pairupgrade-spent
SlinkET Class1occurrence Class2occurrence Tense1infinitive Tense2past modal-relation
CueWord Events First1to-Event2spent First2The-Event1upgrade
27
Grammatical Relations

• Motivation
• Certain intra-sentential relations captured or
  ruled out by particular dependency relations
  between clausal headwords
• Identification of headwords also important
• Main events identified
• RASP parser

Syntax - Grammatical relations between two
segments - GR SEG1 head word - GR
SEG2 head word - GR Both head words
Feature Class
28
Example
SEG2 The university spent 30000 SEG1 to
upgrade lab equipment in 1987
Fea. Class Example Feature
Proximity adjacent distlt3 distlt5 direction-reverse same-sentence
Cue Words First1to First2The
BSO Research LabgtEducational ActivitygtUniversity
WSE WSEgt0.05 WSE-sentence-similarity0.005417
Events Event1upgrade event2spent event-pairupgrade-spent
SlinkET Class1occurrence Class2occurrence Tense1infinitive Tense2past modal-relation
CueWord Events First1to-Event2spent First2The-Event1upgrade
Syntax Grncmod Grncmod-Head1equipment Grncmod-Head2spent
29
Temporal Relations

• Motivation
• Temporal ordering between events constrains
  possible coherence relations
• E.g. E1 BEFORE E2 gt NOT(E2 CAUSE E1)
• Temporal Relation Classifier
• Trained on TimeBank 1.2 using MaxEnt
• See Mani et al. Machine Learning of Temporal
  Relations ACL 2006

Feature Class
TLink - Temporal Relations holding between
segments
30
Example
SEG2 The university spent 30000 SEG1 to
upgrade lab equipment in 1987
Fea. Class Example Feature
Proximity adjacent distlt3 distlt5 direction-reverse same-sentence
Cue Words First1to First2The
BSO Research LabgtEducational ActivitygtUniversity
WSE WSEgt0.05 WSE-sentence-similarity0.005417
Events Event1upgrade event2spent event-pairupgrade-spent
SlinkET Class1occurrence Class2occurrence Tense1infinitive Tense2past modal-relation
CueWord Events First1to-Event2spent First2The-Event1upgrade
Syntax Grncmod Grncmod-Head1equipment Grncmod-Head2spent
Tlink Seg2-before-Seg1
31
Relation Classification

• Identify
• Specific coherence relation
• Ignoring elaboration subtypes (too sparse)
• Coarse-grained relation (resemblance,
  cause-effect, temporal, attributive)
• Evaluation Methodology
• Used Maximum Entropy classifier ( Gaussian prior
  variance 2.0 )
• 8-fold cross validation
• Specific relation accuracy 81.06
• Inter-annotator agreement 94.6
• Majority Class Baseline 45.7
• Classifying all relations as elaboration
• Coarse-grain relation accuracy 87.51

32
F-Measure Results
Relation Precision Recall F-measure True positives
elaboration 88.72 95.31 91.90 512
attribution 91.14 95.10 93.09 184
similar (parallel) 71.89 83.33 77.19 132
same 87.09 75.00 80.60 72
cause-effect 78.78 41.26 54.16 63
contrast 65.51 66.67 66.08 57
example 78.94 48.39 60.00 31
temporal precedence 50.00 20.83 29.41 24
violated expectation 33.33 16.67 22.22 12
conditional 45.45 62.50 52.63 8
generalization 0 0 0 0
33
Results Confusion Matrix
Hypothesis
elab par attr ce temp contr same exmp expv cond gen
elab 488 3 7 3 1 0 2 4 0 3 1
par 6 110 2 2 0 8 2 0 0 2 0
attr 4 0 175 0 0 1 2 0 1 1 0
ce 18 9 3 26 3 2 2 0 0 0 0
temp 6 8 2 0 5 3 0 0 0 0 0
contr 4 12 0 0 0 38 0 0 3 0 0
same 3 9 2 2 0 2 54 0 0 0 0
exmp 15 1 0 0 0 0 0 15 0 0 0
expv 3 1 1 0 1 4 0 0 2 0 0
cond 3 0 0 0 0 0 0 0 0 5 0
gen 0 0 0 0 0 0 0 0 0 0 0
Reference
34
Feature Class Analysis

• What is the utility of each feature class?
• Features overlap significantly highly
  correlated
• How can we estimate utility?
• Independently
• Start with Proximity feature class (baseline)
• Add each feature class separately
• Determine improvement over baseline
• In combination with other features
• Start with all features
• Remove each feature class individually
• Determine reduction from removal of feature class

35
Feature Class Analysis Results
Feature Class Accuracy Coarse-grain Acc.
All Features 81.06 87.51
- Proximity 71.52 84.88
- Cuewords 75.71 84.69
BSO 80.65 87.04
WSE 80.26 87.14
Events 80.90 86.92
- SlinkET 79.68 86.89
- CueWord / Event 80.41 87.14
- Syntax 80.20 86.89
- TLink 80.30 87.36
Feature Class Accuracy Coarse-grain Acc.
Proximity 60.08 69.43
Cuewords 76.77 83.50
BSO 62.92 74.40
WSE 62.20 70.10
Events 63.84 78.16
SlinkET 69.00 75.91
CueWord / Event 67.18 78.63
Syntax 70.30 80.84
TLink 64.19 72.30
Feature Class Contributions in Conjunction
Feature Class Contributions in Isolation
36
Relation Identification

• Given
• Discourse segments (and segment sequences)
• Identify
• For each pair of segments, whether a relation
  (any relation) exists on those segments
• Two issues
• Highly skewed classification
• Many negatives, few positives
• Many of the relations are transitive
• These arent annotated and will be false negative
  instances

37
Relation Identification Results

• For all pairs of segment sequence in a document
• Used same features as for classification
• Achieved accuracy only slightly above majority
  class baseline
• For segment pairs in same sentence
• Accuracy 70.04 (baseline 58)
• Identification and classification in same
  sentence
• Accuracy 64.53 (baseline 58)

38
Inter-relation Dependencies

• Each relation shouldnt be identified in
  isolation
• When identifying a relation between si and sj,
  consider other relations involving si and sj
• Include as features the other (gold standard
  true) relation types both segments are involved
  in
• Adding this feature class improves performance to
  82.3
• 6.3 error reduction
• Indicates room for improvement with
• Collective classification (where outputs
  influence each other)
• Incorporating explicit modeling constraints
• Tree-based parsing model
• Constrained DAGs Danlos 2004
• Including, deducing transitive links may help
  further

39
Conclusions

• Classification approach with many features
  achieves good performance at classifying
  coherence relation types
• All feature classes helpful, but
• Discriminative power of most individual feature
  classes captured by union of remaining feature
  classes
• Proximity CueWords acheives 76.77
• Remaining features reduce error by 23.7
• Classification approach performs less well on
  task of identifying the presence of a relation
• Using same features as for classifying coherence
  relation types
• Parsing may prove better for local relationships

40
Future Work

• Additional linguistic analysis
• Co-reference both entities and events
• Word-sense
• lexical similarity confounded with multiple types
  for a lexeme
• Pipelined or stacked architecture
• Classify coarse-grained category first, then
  specific coherence relation
• Justification different categories require
  different types of knowledge
• Relational classification
• Model decisions collectively
• Include constraints on structure
• Investigate transitivity of resemblance relations
• Consider other approaches for identification of
  relations

41
Questions?
42
Backup Slides
43
GraphBank Annotation Statistics

• Corpus and Annotator Statistics
• 135 doubly annotated newswire articles
• Identifying discourse segments had high agreement
  (gt 90 from pilot study of 10 documents)
• Corpus segments ultimately annotated once (by
  both annotators together)
• Segment grouping - Kappa 0.8424
• Relation identification and typing - Kappa 0.8355

44
Factors Involved in Identifying Coherence
Relations

• Proximity
• E.g. Attribution local, elaboration non-local
• Lexical and phrasal cues
• Constrain possible relation types
• But gt contrast, expected violation
• And gt elaboration, similar, contrast
• Co-reference
• Coherence established with references to
  mentioned entities/events
• Argument structure
• E.g. similar gt similar/same event and/or
  participants
• Lexical Knowledge
• Type inclusion, word sense
• Qualia (purpose of an object, resulting state of
  an action), event structure
• Paraphrases delay gt arrive late
• World Knowledge
• E.g. Ukraine is part of Europe

45
Architecture
Training
Knowledge Source 1
Pre-processing
Knowledge Source 2
Feature Constructor
Model
Classifications
Knowledge Source n
Prediction
46
Scenario Extraction MUC

• Pull together relevant facts related to a
  complex event
• Management Succession
• Mergers and Acquisitions
• Natural Disasters
• Satellite launches
• Requires identifying relations between events
• Parallel, cause-effect, elaboration
• Also identity, part-of
• Hypothesis
• Task independent identification of discourse
  relations will allow rapid development of
  Scenario Extraction systems

47
Information Extraction Current
Scenario Extraction
Fact Extraction
Task 1.1
Domain 1
Task 1.N
Pre-process
Task 2.1
Domain 2
Task 2.N
Domain N
48
Information Extraction Future
Pre-process
Fact Extraction
Discourse