Title: Participants
1Participants
- Janyce Wiebe
- Eric Breck
- Chris Buckley
- Claire Cardie
- Paul Davis
- Bruce Fraser
- Diane Litman
- David Pierce
- Ellen Riloff
- Theresa Wilson
2Part I Overview
3Problem
- Finding and organizing opinions in the world
press and other text
4Our Work will Support
- Finding a range of opinions expressed on a
particular topic, event, issue - Clustering opinions and their sources
- Attitude (positive, negative, uncertain)
- Basis for opinion (supporting beliefs,
experiences) - Expressive style (sarcastic, vehement, neutral)
- Building perspective profiles of individuals and
groups over many documents and topics
5Task Conceptualization
- Perspective in language
- Implications for higher-level tasks
6Task Annotation
- Manual annotation scheme for linguistic
expressions of opinions -
- It is heresy, said Cao. The Shouters claim
- they are bigger than Jesus.
(writer,Cao)
(writer,Cao,Shouters)
(writer,Cao)
(writer,Cao)
7Task Summary Representation of Opinions
- Built upon lower-level annotations
8Task Automate Manual Annotations
- Machine learning
- Identification of opinionated phrases, sources of
opinions,
9Task Organizing Perspective Segments
- Unsupervised clustering
- Text features features from the annotation
scheme higher-level features
10Evaluation
- Exploratory manual clustering
- Evaluation of annotation agreement
- Evaluation of automatic annotations
- Evaluation of automatic clustering
11Solution Architecture
Annotation Architecture
AnnotationTool
Learning Architecture
LearningAlgorithms
Trained Taggers
Application Architecture
PerspectiveTagging
DocumentRetrieval
DocumentClustering
Question
Other Taggers
12Part II Annotation
13Annotation
- Find opinions, evaluations, emotions,
speculations (private states) expressed in
language
Private state state that is not open to
objective observation or verification.
Quirk, Greenbaum, Leech, Svartvik (1985). A
Comprehensive Grammar of the English Language.
14Two Ways of Expressing Private States
- Explicit mentions of private states and speech
events - The United States fears a spill-over from the
anti-terrorist campaign - Expressive subjective elements
- The part of the US human rights report about
China is full of absurdities and fabrications.
15Two Ways of Expressing Private States
- Explicit mentions of private states and speech
events Ons - The United States fears a spill-over from the
anti-terrorist campaign - Expressive subjective elements
- The part of the US human rights report about
China is full of absurdities and fabrications.
16Nested Sources
17OnlyFactive
OnlyFactiveyes
The US fears a spill-over, said Xirao-Nima, a
professor of foreign affairs at the Central
University for Nationalities.
18OnlyFactive
- An example of all OnlyFactiveyes
-
- The government, it added, has amended Pakistan
Citizenship Act 10 of 1951 to enable women of
Pakistani descent to claim Pakistani nationality
for their children born to foreign husbands.
19Example
-
- It is heresy, said Cao. The Shouters claim
- they are bigger than Jesus.
20Example
The Foreign Ministry said Thursday that it was
surprised, to put it mildly
by the U.S. State Departments criticism of
Russias human rights
record and objected in particular to the odious
section on Chechnya.
21Other attributes
- Strengths of various components
- Types of private states
- Objects
22Documents and Annotators
- Over 90 annotated documents
- 100 spans of sources, subjective elements, ons,
- with type and OnlyFactive
attributes. - 39 sources, strengths, agent objects
- Participants part-time annotators
23(No Transcript)
24Pilot Agreement Evaluationons and subjective
elements
- Agreement measured by percentage of As
constituents matching Bs, and vice versa
25Overlapping Spans Ons
- Bush has adopted the most pro-Taiwan posture of
any president... - Bush has adopted the most pro-Taiwan posture of
any president...
26Overlapping Spans Expressive Subjective Elements
-
- some of Mr. Chavezs more alarming faults
- some of Mr. Chavezs more alarming faults
27Untrained Annotators
- 2 groups of 3 untrained annotators (the
participants) each annotated 4 documents without
any communication
28Spans, Untrained Annotators
29Overlapping Spans Expressive Subjective Elements
-
- some of Mr. Chavezs more alarming faults
- some of Mr. Chavezs more alarming faults
30C 74
P 196
D 75
Expressive Subjectivity Agreement
Di 76
T 153
E 29 97
31Spans, All Annotators
32Pilot Agreement EvaluationOnlyFactive
- For the ons identified by both annotators
- Measured agreement for Onlyfactive
- Cohens Kappa
- Krippendorf
- 0.67 Tentative conclusions
- 0.80 Definite conclusions
33OnlyFactive Kappa Values
34Future work
- Continue annotations by trained annotators
- Continue to explore annotations
- Train new annotators, starting with settled
scheme - Even more detail is possible
35Part III Summary Representations of Opinions
36Example
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
37Sentence 1
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
38Sentence 1 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
39Sentence 2
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
40Sentence 2 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt neg-attitude (medium) ? ltreportgt
ltwritergt neg-attitude (medium)
ltwritergt neg-attitude (medium)
41Sentence 3
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
42Sentence 3 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt onlyfactive ltwriter,
many-countriesgt neg-attitude (medium)
? ltreportgt ltwriter, many-countriesgt
extreme
ltwriter, many-countriesgt neg-attitude (high,
high, medium)
43Sentence 4
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
44Sentence 4 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt onlyfactive
ltwriter, info-officegt
neg-attitude (medium) ? ltUSgt
45Sentence 5
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
46Sentence 5 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt onlyfactive ltwriter, chinarepgt
onlyfactive
47Sentence 6
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
48Sentence 6 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt ?neg-attitude
(medium) ? ltUSgt ltwritergt expr-subj (low)
49Sentence 7
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
50Sentence 7 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt neg-attitude (low)
? ltUSgt ltwritergt expr-subj (low) ltwritergt
neg-attitude (medium)
51Sentence 8
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
52Sentence 8 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt onlyfactive
53Sentence 9
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
54Sentence 9 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt onlyfactive
55Sentence 10
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
56Sentence 10 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt neg-attitude (low) ? ltUSgt
ltwritergt expr-subj (low)
57All Low-level Annotations
ltwritergt neg-attitude (medium) ? ltreportgt
ltwritergt neg-attitude (medium)
ltwritergt neg-attitude (medium)
ltwritergt onlyfactive ltwriter,
many-countriesgt neg-attitude (medium)
? ltreportgt ltwriter, many-countriesgt
extreme
ltwriter, many-countriesgt neg-attitude (high,
high, medium)
ltwritergt onlyfactive
ltwriter, info-officegt
neg-attitude (medium) ? ltUSgt
ltwritergt onlyfactive ltwriter, chinarepgt
onlyfactive
ltwritergt ?neg-attitude
(medium) ? ltUSgt ltwritergt expr-subj (low)
ltwritergt neg-attitude (low)
? ltUSgt ltwritergt expr-subj (low) ltwritergt
neg-attitude (medium)
ltwritergt onlyfactive
ltwritergt onlyfactive
ltwritergt neg-attitude (low) ? ltUSgt
ltwritergt expr-subj (low)
58Creating Summary Representations of Opinions with
Writer As Sole Source
ltwritergt neg-attitude (medium) ? ltreportgt
ltwritergt neg-attitude (medium)
ltwritergt neg-attitude (medium)
ltwritergt onlyfactive
ltwritergt onlyfactive
ltwritergt onlyfactive
ltwritergt ?neg-attitude
(medium) ? ltUSgt ltwritergt expr-subj (low)
ltwritergt neg-attitude (low)
? ltUSgt ltwritergt expr-subj (low) ltwritergt
neg-attitude (medium)
ltwritergt onlyfactive
ltwritergt onlyfactive
ltwritergt neg-attitude (low) ? ltUSgt
ltwritergt expr-subj (low)
59Creating Summary Representations of Opinions with
Writer As Sole Source
ATTITUDE value neg strength low
ltUSgt
60Creating Summary Representations of Opinions for
Other Sources
ltwritergt onlyfactive ltwriter,
many-countriesgt neg-attitude (medium)
? ltreportgt ltwriter, many-countriesgt
extreme
ltwriter, many-countriesgt neg-attitude (high,
high, medium)
61Creating Summary Representations of Opinions
62Creating Summary Representations of Opinions for
Other Sources
ltwritergt onlyfactive
ltwriter, info-officegt
neg-attitude (medium) ? ltUSgt
63Creating Summary Representations of Opinions
ATTITUDE value neg strength low
ATTITUDE value neg strength high
ltUSgt
ATTITUDE value neg strength medium
64Summaries
- Describe collective perspectives with respect to
an issue or event - Describe the perspective of a particular writer,
individual, government, or news source with
respect to an object or issue - Describe the temperature and level of rhetoric
about an event or issue - Create perspective profiles for agents, groups,
news sources and look for outliers and changes
over time
65Part V Automating Manual Annotations
66Solution Architecture
Annotation Architecture
AnnotationTool
Learning Architecture
LearningAlgorithms
Trained Taggers
Application Architecture
PerspectiveTagging
DocumentRetrieval
DocumentClustering
Question
Other Taggers
67 Learning Architecture
Evaluation
Training Data
Learner
Learner
Annotation Database
Gate NE
CASS
Pattern Finder
68 Learning problems
- Identify expressive subjective elements
- Identify private state and speech event phrases
(ons) - Identify nested sources
- Discriminate facts and views (onlyfactive)
- Classify strengths of opinions
- Classify types of opinions
- Identify objects of opinions
69Pilot experiments
Precision Recall F-measure
Baseline 69.9 47.7 56.7
Naïve Bayes 46.7 76.6 58.0
KNN 69.6 63.4 66.4
70Future Work
- Feature generation (lexical, collocational,
contextual) - Feature selection
- Learning strategies (bootstrapping, co-training)
71Part IV Clustering
72Solution Architecture
Annotation Architecture
AnnotationTool
Learning Architecture
LearningAlgorithms
Trained Taggers
Application Architecture
PerspectiveTagging
DocumentRetrieval
DocumentClustering
Question
Other Taggers
73 Application Architecture
Multi-perspective Classifiers
Document Clustering
Documents
Annotation Database
Gate NE
CASS
Feature Generators
74End-User Evaluation Goal
- Establish framework for evaluating tasks that
would be of direct interest to analyst users - Do an example evaluation
- Most end-uses involve grouping opinions
- Analyzing
- Profiles
- Finding outliers (new info, changes)
- Suggests clustering as a main initial component
75Manual Clustering Exploration
- What kind of clusters do people naturally produce
when given no instructions? - 6 MPQA participants Penny Lehtola manually
clustered opinions from 1-3 topics - 19-31 documents per topic
- Multiple opinions per document
- Everybody did one specific topic (Mugabe)
- Targeted 4 hours per participant
- Video-conference to discuss
76Manual Clustering Analysis
- Background strongly influenced cluster type
- Linguist separately clustered every sentence
according to purpose of sentence - Would be useful for information extraction to
database - Ex-analyst clustered according to whether
immediate threat existed - 4 people used the proposed task framework
- Pro-con top-level clusters
- Different sub-cluster strategy
77Manual Clustering Issues
- Treatment of supporting evidence important
- People disagreed how to include it.
- One had separate clustering for evidence
- Some included as part of opinion, others not
- There needs to be some way of linking to opinion
- Random outliers vs important outliers
- People wanted several opinions in sub-cluster
- Pro-con with sub-clusters offers ability to
duplicate sub-cluster on both pro and con side - Important disagreements appear on other side in a
singleton sub-cluster
78Manual Clusters Agreement
- Among the 4 pro-con two-level clustering
- 50-80 agreement at top level
- Very weak agreement at sub-cluster level
- Even those sub-clustering on common basis
differed - Does head of government task force speak for the
government? - Segment boundary agreement about 60 for those
who treated evidence the same way.
79Manual Clustering Overall
- Clustering
- Important, useful
- Everybody does it differently for different
reasons - Implies evaluation of clusters
- Must be relative to clearly defined task
- Gold standard evaluation very difficult for
anything other than simple clustering - Outlier evaluation
- When important,must be explicitly addressed,
80User Task Topic
- U1 User states topic of interest and interacts
with IR system - S1 System retrieves set of relevant documents
along with their perspective annotations
81User Task Question
- U2 User states particular perspective question
on topic. - Question should
- identify source type (eg, governments,
individuals, writers) of interest. - Be a yes/no (or pro/con) question for now
82User Task Question Response
- S2System clusters documents
- based on question,text,annotations
- goalgroup together documents with same answer
and perspective (including expressive content). - System,for now, does not attempt to label each
group with specific answers. - Target a small number of clusters (2-4?)
83User Task Cluster Feature
- U3 User states constraints on clustered
documents or segments. - These might be geographic, date, ideological,
political, religous - S3 System shows subclusters or highlighted
documents
84User Task Results
- U4 User gets impression (visual or statistical)
whether constraints match clusters. - Easy visualization of exceptions
85FBIS Document Collection
- Large collection of 270,000 foreign news
documents from June, 2001 to May, 2002 - Almost all FBIS documents with a small number of
other relevant docs. - From MITRE MITAP system
- Collection available for government contractors
86 FBIS Document Collection Features
- English Language
- 60 FBIS translated
- 40 source English
- 20 TV/Radio
- 5 Identified as editorials
87WNC Collection (Docs,Topics)
- Subset of FBIS
- Ran 8 topics on full FBIS retrieving 200
documents each - Identified 575 of those docs as being publicly
available for a small fee from World News
Connection (wnc.fedworld.gov)
88WNC Distribution
- MITRE/MPQA arranging to have WNC distribute both
data and annotations - Will require a one-time setup fee then modest
individual fee from each research group. - WNC is a division of the federal National
Technical Information Service (NTIS) - Time-frame of public distribution being worked out
89Topics
- 8 topic statements.
- Couple of clauses.
- Each topic has 40-105 identified related docs
within WNC collection. - 4 topics have complete answer identification
- Answer passage spans identified.
- Generally, 0-4 answer passages per doc
- Average 1 per doc
- Annotations require little training, 5 minutes/doc
901. Simple evaluation
- Can we usefully cluster passages of documents?
- For each of 4 topics and 40-82 WNC docs
- Find best passage in each doc answering topic
- Cluster passages
- Evaluate using answer passages
- Do Pro or Con answers tend to occur within
the passages of a single cluster?
91Simple Evaluation Details
- Passages Determined statically. 800 characters
broken on sentence boundaries, and overlapping. - Clusters
- Complete-link agglomerative clusters at top level
- Average-link agglomerative clusters afterwards.
- Target number of clusters (3)
- No cluster contains more than 2/3 docs.
92Simple Evaluation Comparison
- Base case Determination of best passage from
terms of passage only - Opinion case Boost similarity of passage if
automatically determined ON occurs.
93Topic 1 Zimbabwe - Base
Cluster Both Yes No Neither
1 0 3 11 18
2 1 1 5 6
3 2 2 4 4
94Topic 1 Zimbabwe - Opinion
Cluster Both Yes No Neither
1 1 5 8 15
2 1 1 8 8
3 0 1 3 6
Opinions help a bit
95Topic 3 Kyoto Base/Opinion
Cl Both Yes No Nthr
1 0 0 7 7
2 1 3 4 7
3 3 2 3 3
Cl Both Yes No Nthr
1 0 2 2 8
2 2 2 10 8
3 2 1 2 1
Opinions hurt a bit
962. Retrospective Evaluation
- Is poor performance due to opinions, or to the
clustering? - Suppose we could find opinion passages perfectly.
Could we then cluster them?
97Retrospective
- Take all passages given by the topic answers.
- Often several passages per document
- Most are shorter than previous experiment.
- Some are the entire document.
- Cluster these passages using same algorithms as
previously. - Examine whether the Yes and No answers group
in the clusters.
98Retrospective Zimbabwe
Cluster Both Yes No
1 1 1 18
2 0 1 10
3 0 23 34
99Retrospective Kyoto
Cluster Both Yes No
1 1 6 19
2 0 2 1
3 0 4 7
100End-User Summary
- We have an end-to-end system that
- Retrieves documents
- Adds opinions annotations using NLP tools
- Clusters documents or passages partly based on
those features - Evaluations show
- Clustering itself needs work before we can
evaluate the use of perspective.
101Future work
- Evaluate the end-user task described in the
mid-term. - Needs more annotated WNC docs.
- Current maximum is only 4 annotated docs for any
query. Not enough for even a toy evaluation! - Needs more investigation of pure clustering
before we can evaluate opinions with it. - Retrospective evaluation is a good testbed.
102Future Work 2.
- Techniques not yet being used
- Query expansion including explicit opinion
signs - Collection statistics for opinions and agents
- Just finished first opinion annotations of entire
- Collection (base heuristic ONs)
- (Hardware for large collection arrived middle of
week 7)
103Summary of Future Work
- More document annotations and studies
- Incorporate summary representations of opinions
into end-user evaluations - Feature generation, feature selection, and
additional learning strategies for recognizing
expressions of opinions - Expand end-user evaluation to include richer
aspects of the opinion annotations and summary
representations - Other end-user evaluation tasks