Title: MultiPerspective Question Answering
1Multi-Perspective Question Answering
- ARDA NRRC Summer 2002 Workshop
2Participants
- Janyce Wiebe
- Eric Breck
- Chris Buckley
- Claire Cardie
- Paul Davis
- Bruce Fraser
- Diane Litman
- David Pierce
- Ellen Riloff
- Theresa Wilson
3Part I Overview
4Problem
- Finding and organizing opinions in the world
press and other text
5Our Work will Support
- Finding a range of opinions expressed on a
particular topic, event, issue
- Clustering opinions and their sources
- Attitude (positive, negative, uncertain)
- Basis for opinion (supporting beliefs,
experiences)
- Expressive style (sarcastic, vehement, neutral)
- Building perspective profiles of individuals and
groups over many documents and topics
6Task Conceptualization
- Perspective in language
- Implications for higher-level tasks
7Task Annotation
- Manual annotation scheme for linguistic
expressions of opinions
-
- It is heresy, said Cao. The Shouters claim
- they are bigger than Jesus.
(writer,Cao)
(writer,Cao,Shouters)
(writer,Cao)
(writer,Cao)
8Task Summary Representation of Opinions
- Built upon lower-level annotations
9Task Automate Manual Annotations
- Machine learning
- Identification of opinionated phrases, sources of
opinions,
10Task Organizing Perspective Segments
- Unsupervised clustering
- Text features features from the annotation
scheme higher-level features
11Evaluation
- Exploratory manual clustering
- Evaluation of annotation agreement
- Evaluation of automatic annotations
- Evaluation of automatic clustering
12Part II Conceptualization
13Introduction
- Perspective Framework/filter through which we
see an event
- Language give insight into perspective of writer/
speaker and the people they describe
- Our focus subjectivity (reflections of private
states) as conveyed through language
- Goal Draw from and elaborate on the Annotation
Guidelines to expand our knowledge of how
subjectivity is conveyed via linguistic
expressions.
14Influences on Perspective
- Identity (President, Assistant Secretary of
State)
- Attitudes held by a person
- Micro (towards specific individuals, events)
- Macro (belief systems embraced, culture values)
- Genre of the product (editorial, report, white
paper)
- Context of the writing
- Temporal (writing now about 1960)
- Spatial (writing located in war zone)
- Events (writing now rather than before 9-11)
15Lexical Clues to Subjectivity
- Attitude Expression
-
- Explicit Attitudes Direct realization of
private states
- Implicit Attitudes Expressive subjective
elements
- Other linguistic clues
16Attitude Expression
- The Source (writer, speaker, another person an
institution, a document
- The Object (entities, actions, propositions,
topic, context)
- Attitudes Different typologies
17Attitude Typologies
- Negative (reject) Positive (enjoy) Other
(surprise)
- Opinion (criticize) Emotions (fear) Speculation
(probably)
- Other (must, poor, better)
18Attitude Types
- Content
- Evidential (allegedly, think)
- Deontic (require, should)
- Assessment (like, excellent)
- Process
- Discourse Markers (so, but)
- Operators (very, not)
19Types of AE Combinations
- Reinforced Attitude (A) Two AEs of the same
type (I like her and I admire her as well)
- Weakened Attitude (A-) Two AEs, A and A
- (I like her. I cant stand her politics)
- Entailed Attitude (A)- AE A entails AE B
- (He terrorizes people people fear)
- Modified Attitude (A(B)) AE A modifies AE B
- (He cruelly criticized the people)
20Subjectivity Relation to Ideology
- Ideology Beliefs and opinions held
- Approximation of ideology by grouping AEs for a
given Source
- Grouping Sources with similar ideologies
- Inferring ideology from limited AEs
- Change of an ideology over time
- Prediction of position based on ideology
21Solution Architecture
Annotation Architecture
AnnotationTool
Learning Architecture
LearningAlgorithms
Trained Taggers
Application Architecture
PerspectiveTagging
DocumentRetrieval
DocumentClustering
Question
Other Taggers
22Part III Annotation
23Annotation
- Find opinions, evaluations, emotions,
speculations (private states) expressed in
language
Private state state that is not open to
objective observation or verification. Qui
rk, Greenbaum, Leech, Svartvik (1985). A
Comprehensive Grammar of the English
Language.
24Two Ways of Expressing Private States
- Explicit mentions of private states and speech
events
- The United States fears a spill-over from the
anti-terrorist campaign
- Expressive subjective elements
- The part of the US human rights report about
China is full of absurdities and fabrications.
25Two Ways of Expressing Private States
- Explicit mentions of private states and speech
events Ons
- The United States fears a spill-over from the
anti-terrorist campaign
- Expressive subjective elements
- The part of the US human rights report about
China is full of absurdities and fabrications.
26Nested Sources
27OnlyFactive
OnlyFactiveyes
The US fears a spill-over, said Xirao-Nima, a
professor of foreign affairs at the Central Unive
rsity for Nationalities.
28OnlyFactive
- An example of all OnlyFactiveyes
-
- The government, it added, has amended Pakistan
Citizenship Act 10 of 1951 to enable women of
Pakistani descent to claim Pakistani nationality
for their children born to foreign husbands.
29Example
-
- It is heresy, said Cao. The Shouters claim
- they are bigger than Jesus.
30Example
The Foreign Ministry said Thursday that it was
surprised, to put it mildly
by the U.S. State Departments criticism of
Russias human rights
record and objected in particular to the odious
section on Chechnya.
31Other attributes
- Strengths of various components
- Types of private states
- Objects
32Documents and Annotators
- Over 90 annotated documents
- 100 spans of sources, subjective elements,
ons,
- with type and OnlyFactive
attributes.
- 39 sources, strengths, agent objects
- Participants part-time annotators
33(No Transcript)
34Pilot Agreement Evaluationons and subjective
elements
- Agreement measured by percentage of As
constituents matching Bs, and vice versa
35Overlapping Spans Ons
- Bush has adopted the most pro-Taiwan posture of
any president...
- Bush has adopted the most pro-Taiwan posture of
any president...
36Overlapping Spans Expressive Subjective Elements
-
- some of Mr. Chavezs more alarming faults
- some of Mr. Chavezs more alarming faults
37Untrained Annotators
- 2 groups of 3 untrained annotators (the
participants) each annotated 4 documents without
any communication
38Spans, Untrained Annotators
39Overlapping Spans Expressive Subjective Elements
-
- some of Mr. Chavezs more alarming faults
- some of Mr. Chavezs more alarming faults
40C 74
P 196
D 75
Expressive Subjectivity Agreement
Di 76
T 153
E 29 97
41Spans, All Annotators
42Pilot Agreement EvaluationOnlyFactive
- For the ons identified by both annotators
- Measured agreement for Onlyfactive
- Cohens Kappa
- Krippendorf
- 0.67 Tentative conclusions
- 0.80 Definite conclusions
43OnlyFactive Kappa Values
44Future work
- Continue annotations by trained annotators
- Continue to explore annotations
- Train new annotators, starting with settled
scheme
- Even more detail is possible
45Part IV Summary Representations of Opinions
46Example
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
47Sentence 1
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
48Sentence 1 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
49Sentence 2
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
50Sentence 2 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
neg-attitude (medium) ?
neg-attitude (medium)
neg-attitude (medium)
51Sentence 3
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
52Sentence 3 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
onlyfactive many-countries neg-attitude (medium)
?
extreme
neg-attitude (high,
high, medium)
53Sentence 4
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
54Sentence 4 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
onlyfactive
neg-attitude (medium) ?
55Sentence 5
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
56Sentence 5 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
onlyfactive
onlyfactive
57Sentence 6
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
58Sentence 6 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
?neg-attitude
(medium) ? expr-subj (low)
59Sentence 7
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
60Sentence 7 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
neg-attitude (low)
? expr-subj (low)
neg-attitude (medium)
61Sentence 8
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
62Sentence 8 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
onlyfactive
63Sentence 9
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
64Sentence 9 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
onlyfactive
65Sentence 10
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
66Sentence 10 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and Latin
America have rejected the content of the US Human
Rights Report, calling it a brazen distortion of
the situation, a wrongful and illegitimate move,
and an interference in the internal affairs of
other countries. Recently, the Information Offic
e of the Chinese People's Congress released a
report on human rights in the United States in
2001, criticizing violations of human rights
there. The report quoting data from the Christian
Science Monitor, points out that the murder rate
in the United States is 5.5 per 100,000 people.
In the United States, torture and pressure to
confess crime is common. Many people have been
sentenced to death for crime they did not commit
as a result of an unjust legal system. More than
12 million children are living below the poverty
line. According to the report, one American woman
is beaten every 15 seconds. Evidence show that
human rights violations in the United States have
been ignored for many years.
neg-attitude (low) ?
expr-subj (low)
67All Low-level Annotations
neg-attitude (medium) ?
neg-attitude (medium)
neg-attitude (medium)
onlyfactive many-countries neg-attitude (medium)
?
extreme
neg-attitude (high,
high, medium)
onlyfactive
neg-attitude (medium) ?
onlyfactive
onlyfactive
?neg-attitude
(medium) ? expr-subj (low)
neg-attitude (low)
? expr-subj (low)
neg-attitude (medium)
onlyfactive
onlyfactive
neg-attitude (low) ?
expr-subj (low)
68Creating Summary Representations of Opinions with
Writer As Sole Source
neg-attitude (medium) ?
neg-attitude (medium)
neg-attitude (medium)
onlyfactive
onlyfactive
onlyfactive
?neg-attitude
(medium) ? expr-subj (low)
neg-attitude (low)
? expr-subj (low)
neg-attitude (medium)
onlyfactive
onlyfactive
neg-attitude (low) ?
expr-subj (low)
69Creating Summary Representations of Opinions with
Writer As Sole Source
ATTITUDE value neg strength low
70Creating Summary Representations of Opinions for
Other Sources
onlyfactive many-countries neg-attitude (medium)
?
extreme
neg-attitude (high,
high, medium)
71Creating Summary Representations of Opinions
72Creating Summary Representations of Opinions for
Other Sources
onlyfactive
neg-attitude (medium) ?
73Creating Summary Representations of Opinions
ATTITUDE value neg strength low
ATTITUDE value neg strength high
ATTITUDE value neg strength medium
74Summaries
- Describe collective perspectives with respect to
an issue or event
- Describe the perspective of a particular writer,
individual, government, or news source with
respect to an object or issue
- Describe the temperature and level of rhetoric
about an event or issue
- Create perspective profiles for agents, groups,
news sources and look for outliers and changes
over time
75Part V Automating Manual Annotations
76Solution Architecture
Annotation Architecture
AnnotationTool
Learning Architecture
LearningAlgorithms
Trained Taggers
Application Architecture
PerspectiveTagging
DocumentRetrieval
DocumentClustering
Question
Other Taggers
77 Learning Architecture
Evaluation
Training Data
Learner
Learner
Annotation Database
Gate NE
CASS
Pattern Finder
78 Learning problems
- Identify expressive subjective elements
- Identify private state and speech event phrases
(ons)
- Identify nested sources
- Discriminate facts and views (onlyfactive)
- Classify strengths of opinions
- Classify types of opinions
- Identify objects of opinions
79Pilot experiments
80Future Work
- Feature generation (lexical, collocational,
contextual)
- Feature selection
- Learning strategies (bootstrapping, co-training)
81Part VI Clustering
82Solution Architecture
Annotation Architecture
AnnotationTool
Learning Architecture
LearningAlgorithms
Trained Taggers
Application Architecture
PerspectiveTagging
DocumentRetrieval
DocumentClustering
Question
Other Taggers
83 Application Architecture
Multi-perspective Classifiers
Document Clustering
Documents
Annotation Database
Gate NE
CASS
Feature Generators
84End-User Evaluation Goal
- Establish framework for evaluating tasks that
would be of direct interest to analyst users
- Do an example evaluation
- Most end-uses involve grouping opinions
- Analyzing
- Profiles
- Finding outliers (new info, changes)
- Suggests clustering as a main initial component
85Manual Clustering Exploration
- What kind of clusters do people naturally produce
when given no instructions?
- 6 MPQA participants Penny Lehtola manually
clustered opinions from 1-3 topics
- 19-31 documents per topic
- Multiple opinions per document
- Everybody did one specific topic (Mugabe)
- Targeted 4 hours per participant
- Video-conference to discuss
86Manual Clustering Analysis
- Background strongly influenced cluster type
- Linguist separately clustered every sentence
according to purpose of sentence
- Would be useful for information extraction to
database
- Ex-analyst clustered according to whether
immediate threat existed
- 4 people used the proposed task framework
- Pro-con top-level clusters
- Different sub-cluster strategy
87Manual Clustering Issues
- Treatment of supporting evidence important
- People disagreed how to include it.
- One had separate clustering for evidence
- Some included as part of opinion, others not
- There needs to be some way of linking to opinion
- Random outliers vs important outliers
- People wanted several opinions in sub-cluster
- Pro-con with sub-clusters offers ability to
duplicate sub-cluster on both pro and con side
- Important disagreements appear on other side in a
singleton sub-cluster
88Manual Clusters Agreement
- Among the 4 pro-con two-level clustering
- 50-80 agreement at top level
- Very weak agreement at sub-cluster level
- Even those sub-clustering on common basis
differed
- Does head of government task force speak for the
government?
- Segment boundary agreement about 60 for those
who treated evidence the same way.
89Manual Clustering Overall
- Clustering
- Important, useful
- Everybody does it differently for different
reasons
- Implies evaluation of clusters
- Must be relative to clearly defined task
- Gold standard evaluation very difficult for
anything other than simple clustering
- Outlier evaluation
- When important,must be explicitly addressed,
90User Task Topic
- U1 User states topic of interest and interacts
with IR system
- S1 System retrieves set of relevant documents
along with their perspective annotations
91User Task Question
- U2 User states particular perspective question
on topic.
- Question should
- identify source type (eg, governments,
individuals, writers) of interest.
- Be a yes/no (or pro/con) question for now
92User Task Question Response
- S2System clusters documents
- based on question,text,annotations
- goalgroup together documents with same answer
and perspective (including expressive content).
- System,for now, does not attempt to label each
group with specific answers.
- Target a small number of clusters (2-4?)
93User Task Cluster Feature
- U3 User states constraints on clustered
documents or segments.
- These might be geographic, date, ideological,
political, religous
- S3 System shows subclusters or highlighted
documents
94User Task Results
- U4 User gets impression (visual or statistical)
whether constraints match clusters.
- Easy visualization of exceptions
95FBIS Document Collection
- Large collection of 270,000 foreign news
documents from June, 2001 to May, 2002
- Almost all FBIS documents with a small number of
other relevant docs.
- From MITRE MITAP system
- Collection available for government contractors
96 FBIS Document Collection Features
- English Language
- 60 FBIS translated
- 40 source English
- 20 TV/Radio
- 5 Identified as editorials
97WNC Collection (Docs,Topics)
- Subset of FBIS
- Ran 8 topics on full FBIS retrieving 200
documents each
- Identified 575 of those docs as being publicly
available for a small fee from World News
Connection (wnc.fedworld.gov)
98WNC Distribution
- MITRE/MPQA arranging to have WNC distribute both
data and annotations
- Will require a one-time setup fee then modest
individual fee from each research group.
- WNC is a division of the federal National
Technical Information Service (NTIS)
- Time-frame of public distribution being worked out
99Topics
- 8 topic statements.
- Couple of clauses.
- Each topic has 40-105 identified related docs
within WNC collection.
- 4 topics have complete answer identification
- Answer passage spans identified.
- Generally, 0-4 answer passages per doc
- Average 1 per doc
- Annotations require little training, 5 minutes/doc
1001. Simple evaluation
- Can we usefully cluster passages of documents?
- For each of 4 topics and 40-82 WNC docs
- Find best passage in each doc answering topic
- Cluster passages
- Evaluate using answer passages
- Do Pro or Con answers tend to occur within
the passages of a single cluster?
101Simple Evaluation Details
- Passages Determined statically. 800 characters
broken on sentence boundaries, and overlapping.
- Clusters
- Complete-link agglomerative clusters at top
level
- Average-link agglomerative clusters afterwards.
- Target number of clusters (3)
- No cluster contains more than 2/3 docs.
102Simple Evaluation Comparison
- Base case Determination of best passage from
terms of passage only
- Opinion case Boost similarity of passage if
automatically determined ON occurs.
103Topic 1 Zimbabwe - Base
104Topic 1 Zimbabwe - Opinion
Opinions help a bit
105Topic 3 Kyoto Base/Opinion
Opinions hurt a bit
1062. Retrospective Evaluation
- Is poor performance due to opinions, or to the
clustering?
- Suppose we could find opinion passages perfectly.
Could we then cluster them?
107Retrospective
- Take all passages given by the topic answers.
- Often several passages per document
- Most are shorter than previous experiment.
- Some are the entire document.
- Cluster these passages using same algorithms as
previously.
- Examine whether the Yes and No answers group
in the clusters.
108Retrospective Zimbabwe
109Retrospective Kyoto
110End-User Summary
- We have an end-to-end system that
- Retrieves documents
- Adds opinions annotations using NLP tools
- Clusters documents or passages partly based on
those features
- Evaluations show
- Clustering itself needs work before we can
evaluate the use of perspective.
111Future work
- Evaluate the end-user task described in the
mid-term.
- Needs more annotated WNC docs.
- Current maximum is only 4 annotated docs for any
query. Not enough for even a toy evaluation!
- Needs more investigation of pure clustering
before we can evaluate opinions with it.
- Retrospective evaluation is a good testbed.
112Future Work 2.
- Techniques not yet being used
- Query expansion including explicit opinion
signs
- Collection statistics for opinions and agents
- Just finished first opinion annotations of
entire
- Collection (base heuristic ONs)
- (Hardware for large collection arrived middle of
week 7)
113Summary of Future Work
- More document annotations and studies
- Incorporate summary representations of opinions
into end-user evaluations
- Feature generation, feature selection, and
additional learning strategies for recognizing
expressions of opinions
- Expand end-user evaluation to include richer
aspects of the opinion annotations and summary
representations
- Other end-user evaluation tasks