Word Sense and Subjectivity

1
Word Sense and Subjectivity

• Jan Wiebe Rada Mihalcea
• University of Pittsburgh University of
  North Texas

2
Introduction

• Growing interest in the automatic extraction of
  opinions, emotions, and sentiments in text
  (subjectivity)

3
Subjectivity Analysis Applications

• Opinion-oriented question answering How do the
  Chinese regard the human rights record of the
  United States?
• Product review mining What features of the
  ThinkPad T43 do customers like and which do they
  dislike?
• Review classification Is a review positive or
  negative toward the movie?
• Tracking emotions toward topics over time Is
  anger ratcheting up or cooling down toward an
  issue or event?
• Etc.

4
Introduction

• Continuing interest in word sense
• Sense annotated resources being developed for
  many languages
• www.globalwordnet.org
• Active participation in evaluations such as
  SENSEVAL

5
Word Sense and Subjectivity

• Though both are concerned with text meaning, they
  have mainly been investigated independently

6
Subjectivity Labels on Senses

• Alarm, dismay, consternation (fear resulting
  from the awareness of danger)
• Alarm, warning device, alarm system (a
  device that signals the occurrence of some
  undesirable event)

7
Subjectivity Labels on Senses

• Interest, involvement -- (a sense of concern
  with and curiosity about someone or something
  "an interest in music")
• Interest -- (a fixed charge for borrowing
  money usually a percentage of the amount
  borrowed "how much interest do you pay on your
  mortgage?")

8
WSD using Subjectivity Tagging
9
WSD using Subjectivity Tagging
He spins a riveting plot which grabs and holds
the readers interest.
S
Sense 4 Sense 1?
Sense 4 a sense of concern with and curiosity
about someone or something S Sense 1 a fixed
charge for borrowing money O
Subjectivity Classifier
WSD System
Sense 1 Sense 4?
O
The notes do not pay interest.
10
WSD using Subjectivity Tagging
He spins a riveting plot which grabs and holds
the readers interest.
S
Sense 4 Sense 1?
Sense 4 a sense of concern with and curiosity
about someone or something S Sense 1 a fixed
charge for borrowing money O
Subjectivity Classifier
WSD System
Sense 1 Sense 4?
O
The notes do not pay interest.
11
Subjectivity Tagging using WSD
Subjectivity Classifier
He spins a riveting plot which grabs and holds
the readers interest.
S O?
O S?
The notes do not pay interest.
12
Subjectivity Tagging using WSD
Subjectivity Classifier
He spins a riveting plot which grabs and holds
the readers interest.
S O?
Sense 4
WSD System
O S?
Sense 1
The notes do not pay interest.
13
Subjectivity Tagging using WSD
Subjectivity Classifier
He spins a riveting plot which grabs and holds
the readers interest.
S O?
Sense 4
WSD System
O S?
Sense 1
The notes do not pay interest
14
Goals

• Explore interactions between word sense and
  subjectivity
• Can subjectivity labels be assigned to word
  senses?
• Manually
• Automatically
• Can subjectivity analysis improve word sense
  disambiguation?
• Can word sense disambiguation improve
  subjectivity analysis? Future work

15
Outline

• Motivation and Goals
• Assigning Subjectivity Labels to Word Senses
• Manually
• Automatically
• Word Sense Disambiguation using Automatic
  Subjectivity Analysis
• Conclusions

16
Prior Work on Subjectivity Tagging

• Identifying words and phrases associated with
  subjectivity
• Think private state Beautiful positive
  sentiment
• Hatzivassiloglou McKeown 1997 Wiebe 2000
  Kamps Marx 2002 Turney 2002 Esuli
  Sabastiani 2005 Etc
• Subjectivity classification of sentences,
  clauses, phrases, or word instances in context
• subjective/objective positive/negative/neutral
• Riloff Wiebe 2003 Yu Hatzivassiloglou 2003
  Dave et al 2003 Hu Liu 2004 Kim Hovy 2004
  Etc.
• Here subjectivity labels are applied to word
  senses

17
Outline

• Motivation and Goals
• Assigning Subjectivity Labels to Word Senses
• Manually
• Automatically
• Word Sense Disambiguation using Automatic
  Subjectivity Analysis
• Conclusions

18
Annotation Scheme

• Assigning subjectivity labels to WordNet senses
• S subjective
• O objective
• B both

19
Annotators are given the synset and its hypernym

• Alarm, dismay, consternation (fear
  resulting form the awareness of danger)
• Fear, fearfulness, fright (an emotion
  experiences in anticipation of some specific pain
  or danger (usually accompanied by a desire to
  flee or fight))

20
Subjective Sense Definition

• When the sense is used in a text or conversation,
  we expect it to express subjectivity, and we
  expect the phrase/sentence containing it to be
  subjective.

21
Objective Senses Observation

• We dont necessarily expect phrases/sentences
  containing objective senses to be objective
• Would you actually be stupid enough to pay that
  rate of interest?
• Will someone shut that darn alarm off?
• Subjective, but not due to interest or alarm

22
Objective Sense Definition

• When the sense is used in a text or conversation,
  we dont expect it to express subjectivity and,
  if the phrase/sentence containing it is
  subjective, the subjectivity is due to something
  else.

23
Senses that are Both

• Covers both subjective and objective usages
• Example
• absorb, suck, imbibe, soak up, sop up, suck
  up, draw, take in, take up (take in, also
  metaphorically The sponge absorbs water well
  She drew strength from the Ministers Words)

24
Annotated Data

• 64 words 354 senses
• Balanced subset 32 words 138 senses 2 judges
• The ambiguous nouns of the SENSEVAL-3 English
  Lexical Task 20 words 117 senses 2 judges
• Mihalcea, Chklovski Kilgarriff, 2004
• Others 12 words 99 senses 1 judge

25
Annotated Data Agreement Study

• 64 words 354 senses
• Balanced subset 32 words 138 senses 2 judges
• 16 words have both S and O senses
• 16 words do not (8 only S and 8 only O)
• All subsets balanced between nouns and verbs
• Uncertain tags also permitted

26
Inter-Annotator Agreement Results

• Overall
• Kappa0.74
• Percent Agreement85.5

27
Inter-Annotator Agreement Results

• Overall
• Kappa0.74
• Percent Agreement85.5
• Without the 12.3 cases when a judge is U
• Kappa0.90
• Percent Agreement95.0

28
Inter-Annotator Agreement Results

• Overall
• Kappa0.74
• Percent Agreement85.5
• 16 words with S and O senses Kappa0.75
• 16 words with only S or O Kappa0.73
• Comparable difficulty

29
Inter-Annotator Agreement Results

• 64 words 354 senses
• The ambiguous nouns of the SENSEVAL-3 English
  Lexical Task 20 words 117 senses 2 judges
• U tags not permitted
• Even so, Kappa0.71

30
Outline

• Motivation and Goals
• Assigning Subjectivity Labels to Word Senses
• Manually
• Automatically
• Word Sense Disambiguation using Automatic
  Subjectivity Analysis
• Conclusions

31
Related Work

• unsupervised word-sense ranking algorithm of
  McCarthy et al 2004
• That task approximate corpus frequencies of word
  senses
• Our task predict a word-sense property
  (subjectivity)
• method for learning subjective adjectives of
  Wiebe 2000
• That task label words
• Our task label word senses

32
Overview

• Main idea assess the subjectivity of a word
  sense based on information about the subjectivity
  of
• a set of distributionally similar words
• in a corpus annotated with subjective expressions

33
MPQA Opinion Corpus

• 10,000 sentences from the world press annotated
  for subjective expressions
• Wiebe at al., 2005
• www.cs.pitt.edu/mpqa

34
Subjective Expressions

• Subjective expressions opinions, sentiments,
  speculations, etc. (private states) expressed in
  language

35
Examples

• His alarm grew.
• The leaders roundly condemned the Iranian
  Presidents verbal assault on Israel.
• He would be quite a catch.
• That doctor is a quack.

36
Preliminaries subjectivity of word w

37
Subjectivity of word w
Unannotated Corpus (BNC)

insts(DSW) in SE - insts(DSW)
not in SE
insts (DSW)
subj(w)
-1, 1 highly objective, highly subjective
38
Subjectivity of word w
39
Subjectivity of word sense wi
Rather than 1, add or subtract sim(wi,dswj)

sim(wi,dsw1)
-1, 1
-sim(wi,dsw1)
sim(wi,dsw2)
40
Method Step 1

• Given word w
• Find distributionally similar words Lin 1998
• DSW dswj j 1 .. n
• Experiment with top 100 and 160

41
Method Step 2
word w Alarm DSW1 Panic DSW2 Detector
Sense w1 fear resulting from the awareness of danger sim(w1,panic) sim(w1,detector)
Sense w2 a device that signals the occurrence of some undesirable event sim(w2,panic) sim(w2, detector)
42
Method Step 2

• Find the similarity between each word sense and
  each distributionally similar word

• wnss can be any concept-based similarity measure
  between word senses
• we use Jiang Conrath 1997

43
Method Step 2

• Find the similarity between each word sense and
  each distributionally similar word

• wnss can be any concept-based similarity measure
  between word senses
• we use Jiang Conrath 1997

44
Method Step 2

• Find the similarity between each word sense and
  each distributionally similar word

• wnss can be any concept-based similarity measure
  between word senses
• we use Jiang Conrath 1997

45
Method Step 2

• Find the similarity between each word sense and
  each distributionally similar word

• wnss can be any concept-based similarity measure
  between word senses
• we use Jiang Conrath 1997

46
Method Step 2

• Find the similarity between each word sense and
  each distributionally similar word

• wnss can be any concept-based similarity measure
  between word senses
• we use Jiang Conrath 1997

47
Method Step 3

• Input word sense wi of word w
• DSW dswj j 1..n
• sim(wi,dswj)
• MPQA Opinion Corpus
• Output subjectivity score subj(wi)

48
Method Step 3

• totalsim insts(dswj) sim(wi,dswj)
• subj 0
• for each dswj in DSW
• for each instance k in insts(dswj)
• if k is in a subjective expression
• subj sim(wi,dswj)
• else
• subj - sim(wi,dswj)
• subj(wi) subj / totalsim

49
Method Optional Variation
if k is in a subjective expression
subj sim(wi,dswj) else subj -
sim(wi,dswj)
w1 dsw1 dsw2 dsw3 w2 dsw1 dsw2
dsw3 w3 dsw1 dsw2 dsw3
Selected
50
Evaluation

• Calculate subj scores for all word senses, and
  sort them
• While 0 is a natural candidate for division
  between S and O, we perform the evaluation for
  different thresholds in -1,1
• Calculate the precision of the algorithm at
  different points of recall

51
Evaluation

• Automatic assignment of subjectivity for 272 word
  senses (no DSW instances for 82 senses)
• Baseline random selection of S labels
• Number of assigned S labels matches number of S
  labels in the gold standard (recall 1.0)

52
Evaluation precision/recall curves
Number of distri-butionally similar words 160
53
Evaluation

• Break-even point
• Point where precision and recall are equal

54
Outline

• Motivation and Goals
• Assigning Subjectivity Labels to Word Senses
• Manually
• Automatically
• Word Sense Disambiguation using Automatic
  Subjectivity Analysis
• Conclusions

55
Overview

• Augment an existing WSD system with a feature
  reflecting the subjectivity of the context of the
  ambiguous word
• Compare the performance of original and
  subjectivity-aware WSD systems
• The ambiguous nouns of the SENSEVAL-3 English
  Lexical Task
• SENSEVAL-3 data

56
Original WSD System

• Integrates local and topical features
• Local context of three words to the left and
  right, their part-of-speech
• Topical top five words occurring at least three
  times in the context of a word sense
• Ng Lee, 1996, Mihalcea, 2002
• Naïve Bayes classifier
• Lee Ng, 2003

57
Automatic Subjectivity Classifier

• Rule-based automatic sentence classifier from
  Wiebe Riloff 2005
• Included in OpinionFinder available at
• www.cs.pitt.edu/mpqa/

58
Subjectivity Tagging for WSD
Used to tag sentences of the SENSEVAL-3 data that
contain target nouns

Subjectivity Classifier
O


S
atmosphere
Sentencek

59
WSD using Subjectivity Tagging
60
Words with S and O Senses
lt
lt
lt
lt
lt
lt
lt

lt

4.3 error reduction significant (p lt 0.05
paired t-test)
61
Words with Only O Senses
62
Conclusions

• Can subjectivity labels be assigned to word
  senses?
• Manually
• Good agreement Kappa0.74
• Very good when uncertain cases removed
  Kappa0.90
• Automatically
• Method substantially outperforms baseline
• Showed feasibility of assigning subjectivity
  labels to the fine-grained level of word senses

63
Conclusions

• Can subjectivity analysis improve word sense
  disambiguation?
• Improves performance, but mainly for words with
  both S and O senses (4.3 error reduction
  significant (p lt 0.05))
• Performance largely remains the same or degrades
  for words that dont
• Assign subjectivity labels to WordNet WSD system
  should consult WordNet tags to decide when to pay
  attention to the contextual subjectivity feature.

64

• Thank You

65
Refining WordNet

• Semantic Richness
• Find inconsistencies and gaps
• Verb assault attack, round, assail, last out,
  snipe, assault (attack in speech or writing) The
  editors of the left-leaning paper attacked the
  new House Speaker
• But no sense for the noun as in His verbal
  assault was vicious

66
Observation MPQA corpus

• Corpus somewhat noisy for our task
• MPQA annotates subjective expressions
• Objective senses can appear in subjective
  expressions
• Hypothesis subjective senses tend to appear
  more often in subjective expressions than
  objective senses do, and so the appearance of
  words in subjective expressions is evidence of
  sense subjectivity

67
WSD using Subjectivity Tagging
Hypothesis instances of subjective senses are
more likely to be in subjective sentences, so
sentence subjectivity is an informative feature
for WSD of words with both subjective and
objective senses
68
Subjective Sense Examples

• He was boiling with anger
• Seethe, boil (be in an agitated emotional
  state The customer was seething with anger)
• Be (have the quality of being (copula, used
  with an adjective or a predicate noun) John is
  rich This is not a good answer)

69
Subjective Sense Examples

• Whats the catch?
• Catch (a hidden drawback it sounds good
  but whats the catch?)
• Drawback (the quality of being a hindrance he
  pointed out all the drawbacks to my plan)
• That doctor is a quack.
• Quack (an untrained person who pretends to
  be a physician and who dispenses medical advice)
• Doctor, doc, physician, MD, Dr., medico

70
Objective Sense Examples

• The alarm went off
• Alarm, warning device, alarm system (a
  device that signals the occurrence of some
  undesirable event)
• Device (an instrumentality invented for a
  particular purpose the device is small enough
  to wear on your wrist a device intended to
  conserve water
• The water boiled
• Boil (come to the boiling point and change
  from a liquid to vapor Water boils at 100
  degrees Celsius)
• Change state, turn (undergo a transformation or
  a change of position or action)

71
Objective Sense Examples

• He sold his catch at the market
• Catch, haul (the quantity that was caught
  the catch was only 10 fish)
• Indefinite quantity (an estimated quantity)
• The ducks quack was loud and brief
• Quack (the harsh sound of a duck)
• Sound (the sudden occurrence of an audible
  event)