Annotating Topics of Opinions

1
Annotating Topics of Opinions

• Veselin Stoyanov
• Claire Cardie

2
Talk Overview

• Fine-grained sentiment analysis
• Definitions
• Examples
• Opinion topic annotation
• Definitions
• Issues
• Approach and Corpus
• IA agreement

3
Background

• Sentiment Analysis
• Extraction and representation of attitudes,
  evaluations, opinions, and sentiment in text.
• Fine-grained Sentiment Analysis
• At the level of individual expressions of
  opinions.

4
Fine-grained vs. Coarse-grained Sentiment Analysis

• Coarse-grained
• Sentiment classification
• Useful in the product review domain

• Fine-grained
• Individual expressions of opinions
• Multiple opinions per document (even sentence)

5
Fine-grained opinions Example
The Australian press has launched a bitter
attack on Italy.
6
Fine-grained opinions Example
The Australian press has launched a bitter
attack on Italy.

• Definitions differ, but five main components

• Opinion trigger (opinion words)
• Source (opinion holder)
• Polarity positive/negative
• Strength
• Topic (target)

7
Fine-grained opinions Example
The Australian press has launched a bitter
attack on Italy.

• Definitions differ, but five main components

launched a bitter attack

• Opinion trigger (opinion words)
• Source (opinion holder)
• Polarity positive/negative
• Strength
• Topic (target)

8
Fine-grained opinions Example
SThe Australian press has launched a bitter
attack on Italy.

• Definitions differ, but five main components

launched a bitter attack

• Opinion trigger (opinion words)
• Source (opinion holder)
• Polarity positive/negative
• Strength
• Topic (target)

The Australian press
9
Fine-grained opinions Example
SThe Australian press has launched a bitter
attack on Italy.

• Definitions differ, but five main components

launched a bitter attack

• Opinion trigger (opinion words)
• Source (opinion holder)
• Polarity positive/negative
• Strength
• Topic (target)

The Australian press
negative
10
Fine-grained opinions Example
SThe Australian press has launched a bitter
attack on Italy.

• Definitions differ, but five main components

launched a bitter attack

• Opinion trigger (opinion words)
• Source (opinion holder)
• Polarity positive/negative
• Strength
• Topic (target)

The Australian press
negative
high
11
Fine-grained opinions Example
SThe Australian press has launched a bitter
attack on TItaly

• Definitions differ, but five main components

launched a bitter attack

• Opinion trigger (opinion words)
• Source (opinion holder)
• Polarity positive/negative
• Strength
• Topic (target)

The Australian press
negative
high
Italy
12
Fine-grained opinions

• Five components
• Source (opinion holder)
• e.g. Bethard et al., 2004 Choi et al., 2005
  Kim and Hovy, 2006
• Opinion trigger (opinion words)
• e.g. Yu and Hatzivassiloglou, 2003 Riloff and
  Wiebe, 2003
• Polarity positive/negative
• As above
• Strength
• e.g. Wilson et al. 2004
• Topic (target)
• ????

13
Annotating Topics of Fine-grained Opinions

• Definitions
• Issues
• Approach and Corpus
• IA agreement

14
Examples

• (1)OH John likes Marseille for its weather and
  cultural diversity.
• (2)OH Al thinks that the government should tax
  gas more in order to curb CO2 emissions.

15
Definitions

• (1)OH John likes Marseille for its weather and
  cultural diversity.

16
Definitions

• (1)OH John likes Marseille for its weather and
  cultural diversity.
• Topic city of Marseille

• Topic - the real-world object, event or abstract
  entity that is the subject of the opinion as
  intended by the opinion holder

17
Definitions

• (1)OH John likes TOPIC SPAN Marseille for its
  weather and cultural diversity.
• Topic city of Marseille

• Topic - the real-world object, event or abstract
  entity that is the subject of the opinion as
  intended by the opinion holder
• Topic span - the closest, minimal span of text
  that mentions the topic

18
Definitions

• (1)OH John likes TARGETTOPIC SPAN Marseille
  for its weather and cultural diversity.
• Topic city of Marseille

• Topic - the real-world object, event or abstract
  entity that is the subject of the opinion as
  intended by the opinion holder
• Topic span - the closest, minimal span of text
  that mentions the topic
• Target span - the span of text that covers the
  syntactic surface form comprising the contents of
  the opinion

19
Definitions

• (2)OH Al thinks that the government should tax
  gas more in order to curb CO2 emissions.

20
Definitions

• (2)OH Al thinks that TARGET SPAN the
  government should tax gas more in order to curb
  CO2 emissions.

21
Definitions

• (2)OH Al thinks that TARGET SPAN TOPIC SPAN?
  the government should TOPIC SPAN? tax gas more
  in order to TOPIC SPAN? curb TOPIC SPAN? CO2
  emissions.

22
Definitions

• (2)OH Al thinks that TARGET SPAN the
  government should tax gas more in order to curb
  CO2 emissions.

Context (3) Although he doesnt like government
imposed taxes, he thinks that a fuel tax is the
only effective solution.
23
Definitions

• (2)OH Al thinks that TARGET SPAN the
  government should TOPIC SPAN tax gas more in
  order to curb CO2 emissions.

Context (3) Although he doesnt like government
imposed taxes, he thinks that a fuel tax is the
only effective solution.
24
Related Work

• Product reviews
• E.g. Kobayashi et al. (2004), Yi et al. (2003),
  Popescu and Etzioni (2005), Hu and Liu (2004
• Limit topics to mentions of product names,
  components, and their attributes
• Lexicon look-up
• Focused on methods for lexicon acquisition
• MPQA corpus (Wiebe, Wilson, Cardie, 2004)
• Fine-grained opinions
• Topic annotation deemed too difficult
• Target span annotation is underway
• Kim Hovy (2006)
• Target span extraction using semantic frames
• Limited evaluation

25
Issues in Opinion Topic Identification

• Multiple potential topics mentioned within a
  single target span
• (2)OH Al thinks that TARGET SPAN TOPIC SPAN?
  the government should TOPIC SPAN? tax gas more
  in order to TOPIC SPAN? curb TOPIC SPAN? CO2
  emissions.
• Requires context

Topic of an opinion is the entity that comprises
the main information goal of the opinion based on
the discourse context.
26
Issues in Opinion Topic Identification

• Opinion topics are not always explicitly
  mentioned
• (4) OH John believes the violation of
  Palestinian human rights is one of the main
  factors.
• Topic ISRAELI-PALESTINIAN CONFLICT
• (5) OH I disagree entirely!

27
A Coreference Approach

• Hypothesize that the notion of topic coreference
  will facilitate identification of opinion topics
• Easier than specifying the topic of each opinion
  in isolation

Two opinions are topic-coreferent if they share
the same opinion topic.
28
Opinion Topic Corpus

• Build on the MPQA corpus
• 535 Documents manually annotated for fine-grained
  opinions
• No opinion topic annotation
• Our goal Add the opinion topic information on
  top of the existing opinion annotations
• Created and used a GUI

(www.cs.pitt.edu/mpqa)
29
Annotation Process
30
Annotation Process
31
Annotation Process
32
Interannotator Agreement

• Annotator 1
• 150 of the 535 MPQA documents
• Annotator 2
• 20 of these 150
• IAG measures from noun phrase coreference
  resolution

33
Interannotator Agreement

• Annotator 1
• 150 of the 535 MPQA documents
• Annotator 2
• 20 of these 150
• IAG measures from noun phrase coreference
  resolution

34
Baselines

• all-in-one
• assigns all opinions to the same cluster
• 1 opinion per cluster
• assigns each opinion to its own cluster
• same paragraph
• opinions in the same paragraph are assigned to
  the same cluster

35
Results

• Baselines
• vs. Interannotator agreement

36
Thank you

• Questions?

Annotation instructions more information
available at www.cs.cornell.edu/ves
37
Example
The Australian press has launched a bitter attack
on Italy after seeing their beloved Socceroos
eliminated on a controversial late penalty.
Italian coach Lippi has been blasted for his
favorable comments toward the penalty. Lippi is
preparing his side for the upcoming clash with
Ukraine. He hailed 10-man Italy's determination
to beat Australia and reiterated that the penalty
was rightly given.
38
Example fine-grained opinions
SThe Australian press has launched a bitter
attack on TItaly after seeing Stheir beloved
TSocceroos eliminated on a controversial late
Tpenalty. STItalian coach Lippi has also
been blasted for his favorable comments toward
Tthe penalty. Lippi is preparing his side for
the upcoming clash with Ukraine. SHe hailed
10-man TItaly's determination to beat Australia
and reiterated that Tthe penalty was rightly
given.
39
Motivation

• Sentiment analysis Useful as stand-alone
  application
• Product reviews
• Tracking opinions in the press
• Flame detection, etc.
• Opinion information can benefit many NLP
  applications
• Multi-Perspective Question Answering
• Stoyanov, Cardie, Litman and Wiebe. AAAI WS
  2004 and
• Stoyanov, Cardie and Wiebe. HLT-EMNLP
  2005
• Opinion-Oriented Information Retrieval
• Clustering, etc.

40
Annotation Process
41
Annotation Process
42