Annotating Opinions in the World Press

1
Annotating Opinions in the World Press

• Theresa Wilson and Janyce Wiebe
• University of Pittsburgh
• Intelligent Systems Program
• and
• Department of Computer Science

2
Introduction

• Private State mental or emotional state, which
  cannot be directly observed or verified (Quirk et
  al., 1985)

Speculation
Emotions
Beliefs Opinions Evaluations
3
Private States in Text

• Subjectivity aspects of language used to express
  
• private states, including opinions, emotions,
  evaluations, speculations, sentiments (Banfield
  1982)

Complained You Idiot! Terrible product
Speculated Maybe I wonder
Enthused Wonderful! Great idea
(See Fodor 1979, Cohn 1978, Chafe 1986, Fludernik
1993, Wiebe 1994, Stein Wright 1995 for
additional references.)
4
Why do we care?

• News articles mix of factual/subjective language
• Help existing NLP applications IE, IR,
  summarization systems, etc.
• Support new tasks multi-perspective question
  answering

5
Two Main Ways of Expressing Private States

• Direct mentions of private states
• The United States fears a spill-over from the
  anti-terrorist campaign.
• Expressive subjective elements
• The part of the US human rights report about
  China is full of absurdities and fabrications.

6
Private States and Speech Events

• Speech Event ? speaking or writing
• We foresaw electoral fraud but not daylight
  robbery, Tsvangirai said.
• Medical Department head Dr. Hameed Saeed said
  the patients blood has been sent to the
  Institute for Virology in Johannesburg for
  analysis.

7
Nested Sources
8
Annotation Scheme

• Implemented in GATE (http//gate.ac.uk)

Shorthand for direct private state or speech
event expression
9
Attributes for Annotation Types

• On (Direct Private State
• or Speech Event)
• nested-source
• onlyfactive
• overall-strength
• on-strength
• attitude-type
• attitude-toward
• is-implicit
• minor

• Expressive Subjective
• Element
• nested-source
• strength
• attitude-type

10
Attributes for Annotation Types

• On (Direct Private State
• or Speech Event)
• nested-source
• onlyfactive
• overall-strength
• on-strength
• attitude-type
• attitude-toward
• is-implicit
• minor

• Expressive Subjective
• Element
• nested-source
• strength
• attitude-type

11
Attributes for Annotation Types

• On (Direct Private State
• or Speech Event)
• nested-source
• onlyfactive
• overall-strength
• on-strength
• attitude-type
• attitude-toward
• is-implicit
• minor

• Expressive Subjective
• Element
• nested-source
• strength
• attitude-type

12
Attributes for Annotation Types

• On (Direct Private State
• or Speech Event)
• nested-source
• onlyfactive
• overall-strength
• on-strength
• attitude-type
• attitude-toward
• is-implicit
• minor

• Expressive Subjective
• Element
• nested-source
• strength
• attitude-type

13
onlyfactive Attribute
14
Important! onlyfactiveno may still contain facts
(writer, John) onlyfactiveno

• John criticized Mary for smoking.

15
Strength of Private State
on-strengthhigh
16
Overall Strength of Private State
17
Data

• English-language versions of articles from world
  press (FBIS few other sources)
• 8500 sentences have been annotated
• Agreement Study
• 3 annotators (A, M, S)
• 210 sentences (13 documents)
• 1/3 news, 1/3 news on hot-topic, 1/3 editorials

18
Types of Agreement

• Agreement for spans
• Ons
• Expressive Subjective Elements
• Agreement for ON ? onlyfactive attribute
• Agreement for sentences
• Derive sentence-level annotations from
  lower-level annotations

19
Agreement Issues Spans

• Overlapping spans ok ? its a match

20
Agreement Issues Spans

• Different set of annotations for each annotator ?
  Kappa inappropriate

21
Agreement Evaluation spans

• Agreement measured by percentage of As
  constituents matching Bs, and vice versa

agr (ab)
22
Span Agreement Ons
a b agr(ab) agr(ba) average
A M 0.75 0.91
A S 0.80 0.85
M S 0.86 0.75
0.82
23
Measuring Agreement ON ? onlyfactive
24
Measuring Agreement ON ? onlyfactive
25
Measuring Agreement ON ? onlyfactive
onlyfactiveyes
onlyfactiveno
onlyfactiveyes
onlyfactiveno
onlyfactiveno
onlyfactiveyes
26
Agreement results onlyfactive
Ave Kappa

• Borderline onlyfactive
• At least one annotator marked expression
  onlyfactiveno
• Neither annotator marked expression
  overall-strength gt low

27
Deriving Sentence-level Annotations
28
Agreement results sentence-level subjective
annotations
Ave Kappa

• Borderline subjective
• sentence
• At least one annotator marked at least one
  expression onlyfactiveno
• Neither annotator marked an overallstrength gt low

29
Conclusions

• Detailed scheme for annotation of opinions and
  other private states
• Good agreement between annotators
• Strong agreement about clear cases of
  subjectivity
• Lower-level annotations ? higher-level subjective
  sentence annotations
• Extendable

30
Using GATE to perform private state annotations

• GATE
• http//gate.ac.uk
• General annotation instructions and GATE
  instructions
• http//www.cs.pitt.edu/wiebe/pubs/ardasummer02

31
Thank you

• This work was performed in support of the
  Northeast Regional Research Center (NRRC) which
  is sponsored by the Advanced Research and
  Development Activity in Information Technology
  (ARDA), a U.S. Government entity which sponsors
  and promotes research of import to the
  Intelligence Community which includes but is not
  limited to the CIA, DIA, NSA, NIMA and NRO.

32
Contingency Table onlyfactive judgements
Tagger M Tagger M
Yes No
Tagger A Yes nyy181 nyn25
Tagger A No nny12 nnn252
Tagger M Tagger M
Yes No
Tagger A Yes nyy181 nyn8
Tagger A No nny11 nnn224
Agreement for onlyfactive judgments, borderline
cases removed
Agreement for onlyfactive judgments
33
Onlyfactive Agreement
All Expressions All Expressions Borderline Removed Borderline Removed Borderline Removed
Kappa agree Kappa agree removed
A M 0.84 0.91 0.94 0.96 10
A S 0.84 0.92 0.90 0.95 8
M S 0.74 0.87 0.84 0.92 12
34
Subjective Sentence Agreement
All Expressions All Expressions Borderline Removed Borderline Removed Borderline Removed
Kappa agree Kappa agree removed
A M 0.75 0.89 0.87 0.95 11
A S 0.84 0.94 0.92 0.97 8
M S 0.72 0.88 0.83 0.93 13
35
Why we exclude minor on annotations
This is the first time that the Cuban
government has expressed its position over the
U.S. holding of the Taliban and al-Qaeda
prisoners in its territory.
(writer, Cuban gov) onlyfactiveno
Objective Sentence!