Creating Subjective and Objective Sentence Classifiers from Unannotated Texts

1
Creating Subjective and Objective Sentence
Classifiers from Unannotated Texts
(Joint work with Janyce Wiebe at the University
of Pittsburgh)
2
What is Subjectivity?

• Subjective language includes opinions, rants,
  allegations, accusations, suspicions, and
  speculation.
• Distinguishing factual information from
  subjective information could benefit many
  applications, including
• information extraction
• question answering
• summarization
• spam filtering

3
Previous Work on Subjectivity Classification

• Document-level subjectivity classification
  (e.g., Turney 2002 Pang et al. 2002 Spertus
  1997) But most
  documents contain subjective and objective
  sentences. Wiebe et al. 01 reported that 44 of
  sentences in their news corpus were
  subjective!
• Sentence-level subjectivity classification
  Dave et al. 2003 Yu et al. 2003 Riloff, Wiebe,
  Wilson 2003

4
Goals of our research

• Create classifiers that label sentences as
  subjective or objective.
• Learn subjectivity and objectivity clues from
  unannotated corpora.
• Use information extraction techniques to learn
  subjective nouns.
• Use information extraction techniques to learn
  subjective and objective patterns.

5
Outline of Talk

• Learning subjective nouns with extraction
  patterns
• Automatically generating training data with
  high-precision classifiers
• Learning subjective and objective extraction
  patterns
• Naïve Bayes classification and self-training

6
Information Extraction

• Information extraction (IE) systems identify
  facts related to a domain of interest.
• Extraction patterns are lexico-syntactic
  expressions that identify the role of an object.
  For example

ltsubjectgt was killed
assassinated ltdobjgt
murder of ltnpgt
7
Learning Subjective Nouns
Goal to learn subjective nouns from unannotated
texts. Method applying IE-based bootstrapping
algorithms that were designed to learn semantic
categories. Hypothesis extraction patterns can
identify subjective contexts that co-occur with
subjective nouns.
Example expressed ltdobjgt
concern, hope, support
8
Extraction Examples
expressed ltdobjgt condolences, hope, grief,
views, worries indicative of ltnpgt compromise,
desire, thinking inject ltdobjgt vitality,
hatred reaffirmed ltdobjgt resolve, position,
commitment voiced ltdobjgt outrage, support,
skepticism, opposition, gratitude,
indignation show of ltnpgt support, strength,
goodwill, solidarity ltsubjgt was shared anxiety,
view, niceties, feeling
9
Meta-Bootstrapping Riloff Jones 99
Ex hope, grief, joy, concern, worries
Ex expressed ltDOBJgt
Best Extraction Pattern
Ex happiness, relief, condolences
Extractions (Nouns)
10
Basilisk Thelen Riloff 02
11
Subjective Seed Words
12
Subjective Noun Results

• Bootstrapping corpus 950 unannotated FBIS
  documents (English-language foreign news)
• We ran each bootstrapping algorithm for 400
  cycles, generating 2000 words.
• We manually reviewed the words and labeled them
  as strongly subjective or weakly subjective.
• Together, they learned 1052 subjective nouns
  (454 strong, 598 weak).

13
Examples of Strong Subjective Nouns
anguish exploitation pariah antagonism
evil repudiation apologist fallacies
revenge atrocities genius
rogue barbarian goodwill
sanctimonious belligerence
humiliation scum bully ill-treatment
smokescreen condemnation injustice
sympathy denunciation innuendo
tyranny devil insinuation
venom diatribe liar exaggeration
mockery
14
Examples of Weak Subjective Nouns
15
Outline of Talk

• Learning subjective nouns with extraction
  patterns
• Automatically generating training data with
  high-precision classifiers
• Learning subjective and objective extraction
  patterns
• Naïve Bayes classification and self-training

16
Initial Training Data Creation
unlabeled texts
subjective clues
rule-based subjective sentence classifier
rule-based objective sentence classifier
subjective objective sentences
17
Subjective Clues

• entries from manually developed resources
  Levin 93 Ballmer Brennenstuhl 81
• Framenet lemmas with frame element experiencer
  Baker et al. 98
• adjectives manually annotated for polarity
  Hatzivassiloglou McKeown 97
• n-grams learned from corpora Dave et al. 03
  Wiebe et al. 01
• words distributionally similar to subjective seed
  words Wiebe 00
• subjective nouns learned from extraction pattern
  bootstrapping Riloff et al. 03

18
Creating High-Precision Rule-Based Classifiers
GOAL use subjectivity clues from previous
research to build a high-precision (low-recall)
rule-based classifier

• a sentence is subjective if it contains ? 2
  strong subjective clues
• a sentence is objective if
• it contains no strong subjective clues
• the previous and next sentence contain ? 1
  strong subjective clue
• the current, previous, and next sentence together
  contain ? 2 weak subjective clues

19
Data Set

• The MPQA Corpus contains 535 FBIS texts that have
  been manually annotated for subjectivity.
• Our test set consisted of 9,289 sentences from
  the MPQA corpus.
• We consider a sentence to be subjective if it has
  at least one private state of strength medium or
  higher.
• 54.9 of the sentences in our test set are
  subjective.

20
Accuracy of Rule-Based Classifiers
SubjRec SubjPrec SubjF Subj RBC 34.2 90.4
46.6
ObjRec ObjPrec ObjF Obj RBC 30.7 82.4 44.7
21
Generated Data

• We applied the rule-based classifiers to 298,809
  sentences from (unannotated) FBIS documents.
• 52,918 were labeled subjective
• 47,528 were labeled objective
• training set of over 100,000 labeled sentences!

22
Outline of Talk

• Learning subjective nouns with extraction
  patterns
• Automatically generating training data with
  high-precision classifiers
• Learning subjective and objective extraction
  patterns
• Naïve Bayes classification and self-training

23
Representing Subjective Expressions with
Extraction Patterns

• Extraction patterns can represent linguistic
  expressions that are not fixed word sequences.

drove NP up the wall - drove him up the
wall - drove George Bush up the wall - drove
George Herbert Walker Bush up the wall
step on modifiers toes - step on her toes -
step on the mayors toes - step on the newly
elected mayors toes
gave NP a modifiers look - gave his
annoying sister a really really mean look
24
The Extraction Pattern Learner

• Used AutoSlog-TS Riloff 96 to learn extraction
  patterns.
• AutoSlog-TS needs relevant and irrelevant texts
  as input.
• Statistics are generated measuring each patterns
  association with the relevant texts.
• The subjective sentences were called relevant,
  and the objective sentences were called
  irrelevant.

25
ltsubjectgt passive-vp ltsubjgt was
satisfied ltsubjectgt active-vp ltsubjgt
complained ltsubjectgt active-vp dobj ltsubjgt dealt
blow ltsubjectgt active-vp infinitive ltsubjgt
appears to be ltsubjectgt passive-vp
infinitive ltsubjgt was thought to be ltsubjectgt
auxiliary dobj ltsubjgt has position
26
AutoSlog-TS(Step 1)
The World Trade Center, an icon of New York
City, was intentionally attacked very early on
September 11, 2001.
Parser
Extraction Patterns ltsubjgt was attacked icon of
ltnpgt was attacked on ltnpgt
27
AutoSlog-TS (Step 2)
Extraction Patterns ltsubjgt was attacked icon of
ltnpgt was attacked on ltnpgt
28
Identifying Subjective and Objective Patterns
AutoSlog-TS generates 2 statistics for each
pattern F pattern frequency P relevant
frequency / pattern frequency We call a
pattern subjective if F ? 5 and P ? .95 (6364
subjective patterns were learned) We call a
pattern objective if F ? 5 and P ? .15 (832
objective patterns were learned)
29
Examples of Learned Extraction Patterns
Subjective Patterns ltsubjgt believes ltsubjgt was
convinced aggression against ltnpgt to express
ltdobjgt support for ltnpgt
Objective Patterns ltsubjgt increased
production ltsubjgt took effect delegation from
ltnpgt occurred on ltnpgt plans to produce ltdobjgt
30
Patterns with Interesting Behavior
PATTERN FREQ P(Subj Pattern) ltsubjgt asked 128
.63 ltsubjgt was asked 11 1.0
31
Augmenting the Rule-Based Classifiers with
Extraction Patterns
SubjRec SubjPrec SubjF Subj RBC 34.2 90.4
46.6 Subj RBC 58.6 80.9 68.0 w/Patterns
ObjRec ObjPrec ObjF Obj RBC 30.7
82.4 44.7 Obj RBC 33.5 82.1 47.6 w/Patterns
32
Outline of Talk

• Learning subjective nouns with extraction
  patterns
• Automatically generating training data with
  high-precision classifiers
• Learning subjective and objective extraction
  patterns
• Naïve Bayes classification and self-training

33
Naïve Bayes Classifier

• We created an NB classifier using the initial
  training set and several set-valued features
• strong weak subjective clues from RBCs
• subjective objective extraction patterns
• POS tags (pronouns, modals, adjectives, cardinal
  numbers, adverbs)
• separate features for each of the current,
  previous, and next sentences

34
Naïve Bayes Training
35
Naïve Bayes Results
RWW03 77 81 79 (supervised)
RWW03 74 70 72 (supervised)
36
Self-Training Process
37
Self-Training Results
SubjRec SubjPrec SubjF Subj RBC w/Patts 1
58.6 80.9 68.0 Subj RBC w/Patts 2 62.4
80.4 70.3
Naïve Bayes 1 70.6 79.4
74.7 Naïve Bayes 2 86.3 71.3
78.1
Naïve Bayes 1 77.6 68.4
72.7 Naïve Bayes 2 57.6
77.5 66.1
38
Conclusions

• We can build effective subjective sentence
  classifiers using only unannotated texts.
• Extraction pattern bootstrapping can learn
  subjective nouns.
• Extraction patterns can represent richer
  subjective expressions.
• Learning methods can discover subtle distinctions
  between very similar expressions.

39
THE ENDThank you!
40
Related Work

• Genre classification (e.g., Karlgren and
  Cutting 1994 Kessler et al. 1997 Wiebe et al.
  2001)
• Learning adjectives, adj. phrases, verbs, and
  N-grams Turney 2002 Hatzivassiloglou
  McKeown 1997 Wiebe et al. 2001
• Semantic lexicon learning Hearst 1992
  Riloff Shepherd 1997 Roark Charniak 1998
  Caraballo 1999
• Meta-Bootstrapping Riloff Jones 99
• Basilisk Thelen Riloff 02

41
What is Information Extraction?
Extracting facts relevant to a specific topic
from narrative text.
Example Domains Terrorism perpetrator, victim,
target, date, location Management succession
person fired, successor, position,
organization, date Infectious disease outbreaks
disease, organism, victim, symptoms,
location, date
42
Information Extraction from Narrative Text

• Role relationships define the information of
  interest keywords and named entities are not
  sufficient.

Troops were vaccinated against anthrax, cholera,

Researchers have discovered how anthrax toxin
destroys cells and rapidly causes death ...
43
Ranking and Manual Review

• The patterns are ranked using the
  metric
  
• A domain expert reviews the top-ranked patterns
  and assigns thematic roles to the good ones.
  

44
Semantic Lexicons

• A semantic lexicon assigns categories to words.

• Semantic dictionaries are hard to come by,
  especially for specialized domains.
• WordNet Miller 90 is popular but is not always
  sufficient. Roark Charniak 98 found that 3
  of every 5 words learned by their system were
  not present in WordNet.

45
The Bootstrapping Era

KNOWLEDGE !
46
Meta-Bootstrapping
Ex anthrax, ebola, cholera, flu, plague
Ex outbreak of ltNPgt
Best Extraction Pattern
Ex smallpox, tularemia, botulism
Extractions (Nouns)
47
Semantic Lexicon (NP) Results
Iter Company Location Title Location Weapon (Web)
(Web) (Web) (Terror) (Terror) 1 5/5 (1.0) 5/5
(1.0) 0/1 (0) 5/5(1.0) 4/4(1.0)
10 25/32 (.78) 46/50 (.92) 22/31
(.71) 32/50 (.92) 31/44 (.70)
20 52/65 (.80) 88/100 (.88) 63/81
(.78) 66/100 (.66) 68/94 (.72)
30 72/113 (.64) 129/150 (.86) 86/131
(.66) 100/150 (.67) 85/144 (.59)
48
Basilisk
49
The Pattern Pool
Every extraction pattern is scored and the best
patterns are put into a Pattern Pool. The
scoring function is
50
Scoring Candidate Words
Each candidate word is scored by
1. collecting all patterns that extracted it 2.
computing the average number of category
members extracted by those patterns.
51
(No Transcript)
52
Bootstrapping a Single Category
53
Bootstrapping Multiple Categories

54
A Smarter Scoring Function
We incorporated knowledge about competing
semantic categories directly into the scoring
function. The modified scoring function computes
the difference between the score for the target
category and the best score among competing
categories.
diff (wi,ca) AvgLog (wi,ca) - max
(AvgLog(wi,cb))
b ? a