Semantic%20Inference%20for%20Question%20Answering

1
Semantic Inference for Question Answering

• and

Sanda Harabagiu Department of Computer
Science University of Texas at Dallas
Srini Narayanan International Computer Science
Institute Berkeley, CA
2
Outline

• Part I. Introduction The need for Semantic
  Inference in QA
• Current State-of-the-art in QA
• Parsing with Predicate Argument Structures
• Parsing with Semantic Frames
• Special Text Relations
• Part II. Extracting Semantic Relations from
  Questions and Texts
• Knowledge-intensive techniques
• Supervised and unsupervised techniques

3
Outline

• Part III. Knowledge representation and inference
• Representing the semantics of answers
• Extended WordNet and abductive inference
• Intentional Structure and Probabilistic Metonymy
• An example of Event Structure
• Modeling relations, uncertainty and dynamics
• Inference methods and their mapping to answer
  types

4
Outline

• Part IV. From Ontologies to Inference
• From OWL to CPRM
• FrameNet in OWL
• FrameNet to CPRM mapping
• Part V. Results of Event Structure Inference for
  QA
• AnswerBank examples
• Current results for Inference Type
• Current results for Answer Structure

5
The need for Semantic Inference in QA

• Some questions are complex!
• Example
• How can a biological weapons program be detected
  ?
• Answer In recent months, Milton Leitenberg, an
  expert on biological weapons, has been looking at
  this murkiest and most dangerous corner of Saddam
  Hussein's armory. He says a series of reports add
  up to indications that Iraq may be trying to
  develop a new viral agent, possibly in
  underground laboratories at a military complex
  near Baghdad where Iraqis first chased away
  inspectors six years ago. A new assessment by the
  United Nations suggests Iraq still has chemical
  and biological weapons - as well as the rockets
  to deliver them to targets in other countries.
  The UN document says Iraq may have hidden a
  number of Scud missiles, as well as launchers and
  stocks of fuel. US intelligence believes Iraq
  still has stockpiles of chemical and biological
  weapons and guided missiles, which it hid from
  the UN inspectors.

6
Complex questions

• Example
• How can a biological weapons program be detected
  ?
• This question is complex because
• It is a manner question
• All other manner questions that were evaluated in
  TREC were asking about 3 things
• Manners to die, e.g. How did Cleopatra die?,
  How did Einstein die?
• Manners to get a new name, e.g. How did
  Cincinnati get its name?
• Manners to say something in another language,
  e.g. How do you say house in Spanish?
• The answer does not contain any explicit manner
  of detection information, instead it talks about
  reports that give indications that Iraq may be
  trying to develop a new viral agent and
  assessments by the United Nations suggesting that
  Iraq still has chemical and biological weapons

7
Complex questions and semantic information

• Complex questions are not characterized only by a
  question class (e.g. manner questions)
• Example How can a biological weapons program be
  detected ?
• Associated with the pattern How can X be
  detected?
• And the topic X biological weapons program
• Processing complex questions is also based on
  access to the semantics of the question topic
• The topic is modeled by a set of discriminating
  relations, e.g. Develop(program)
  Produce(biological weapons) Acquire(biological
  weapons) or stockpile(biological weapons)
• Such relations are extracted from topic-relevant
  texts

8
Alternative semantic representations

• Using PropBank to access a 1 million word corpus
  annotated with predicate-argument
  structures.(www.cis.upenn.edu/ace)
• We can train a generative model for recognizing
  the arguments of each predicate in questions and
  in the candidate answers.
• Example How can a biological weapons program be
  detected ?

Predicate detect Argument 0 detector
Answer(1) Argument 1 detected biological
weapons Argument 2 instrument Answer(2)
Expected Answer Type
9
More predicate-argument structures for questions

• Example From which country did North Korea
  import its missile launch pad metals?
• Example What stimulated Indias missile
  programs?

Predicate import Argument 0 (role importer)
North Korea Argument 1 (role commodity)
missile launch pad metals Argument 2 (role
exporter) ANSWER
Predicate stimulate Argument 0 (role agent)
ANSWER (part 1) Argument 1 (role thing
increasing) Indias missile programs Argument 2
(role instrument) ANSWER (part 2)
10
Additional semantic resources

• Using FrameNet
• frame-semantic descriptions of several thousand
  English lexical items with semantically annotated
  attestations (www.icsi.berkeley.edu/framenet)
• Example What stimulated Indias missile
  programs?

Frame STIMULATE Frame Element CIRCUMSTANCES
ANSWER (part 1) Frame Element EXPERIENCER
Indias missile programs Frame Element STIMULUS
ANSWER (part 2)
Frame SUBJECT STIMULUS Frame Element
CIRCUMSTANCES ANSWER (part 3) Frame Element
COMPARISON SET ANSWER (part 4) Frame Element
EXPERIENCER Indias missile programs Frame
Element PARAMETER nuclear/biological
proliferation
11
Semantic inference for Q/A

• The problem of classifying questions
• E.g. manner questions
• Example How did Hitler die?
• The problem of recognizing answer
  types/structures
• Should manner of death by considered an answer
  type?
• What other manner of event/action should be
  considered as answer types?
• The problem of extracting/justifying/ generating
  answers to complex questions
• Should we learn to extract manner relations?
• What other types of relations should we consider?
• Is relation recognition sufficient for answering
  complex questions? Is it necessary?

12
Manner-of-death

• In previous TREC evaluations 31 questions asked
  about manner of death
• How did Adolf Hitler die?
• State-of-the-art solution (LCC)
• We considered Manner-of-Death as an answer
  type, pointing to a variety of verbs and
  nominalizations encoded in WordNet
• We developed text mining techniques for
  identifying such information based on
  lexico-semantic patterns from WordNet
• Example
• kill sense1 (verb) CAUSE ? die sense1
  (verb)
• Source of the troponyms of the kill sense1
  (verb) concept are candidates for the
  MANNER-OF-DEATH hierarchy
• e.g., drown, poison, strangle, assassinate, shoot

13
Practical Hurdle

• Not all MANNER-OF-DEATH concepts are lexicalized
  as a verb
• ? we set out to determine additional patterns
  that capture such cases
• Goal (1) set of patterns
• (2) dictionaries corresponding to
  such patterns
• ? well known IE technique (IJCAI99,
  RiloffJones)

• Results 100 patterns were discovered

14
Outline

• Part I. Introduction The need for Semantic
  Inference in QA
• Current State-of-the-art in QA
• Parsing with Predicate Argument Structures
• Parsing with Semantic Frames
• Special Text Relations

15
Answer types in State-of-the-art QA systems
Ranked set of passages
Docs
Question
Answer
Question Expansion
Answer Selection
IR
answer type
Answer Type Prediction
Answer Type Hierarchy

• Features
• Answer type
• Labels questions with answer type based on a
  taxonomy
• Classifies questions (e.g. by using a maximum
  entropy model)

16
In Question Answering two heads are better than
one

• The idea originated in the IBMs PIQUANT project
• Traditional Q/A systems employ a pipeline
  approach
• Questions analysis
• Document/passage retrieval
• Answer selection
• Questions are classified based on the expected
  answer type
• Answers are also selected based on the expected
  answer type, regardless of the question class
• Motivated by the success of ensemble methods in
  machine learning, use multiple classifiers to
  produce the final output for the ensemble made of
  multiple QA agents
• A multi-strategy, multi-source approach.

17
Multiple sources, multiple agents
Knowledge Source Portal
WordNet
QGoals
Q-Frame Answer Type
Answering Agents
QUESTION
QPlan Generator
Cyk
Predictive Annot. Answering Agents
Web
Statistical Answering Agents
Semantic Search
Definitional Q. Answering Agents
QPlan Executor
KSP-Based Answering Agents
Keyword Search
Pattern-Based Answering Agents
AQUAINT
TREC
CNS
Answer Resolution
Answers
ANSWER
18
Multiple Strategies

• In PIQUANT, the answer resolution strategies
  consider that different combinations of the
  questions processing, passage retrieval and
  answer selection from different agents is ideal.
• This entails the fact that all questions are
  processed depending on the questions class, not
  the question type
• There are multiple question classes, e.g. What
  questions asking about people, What questions
  asking about products, etc.
• There are only three types of questions that have
  been evaluated yet in systematic ways
• Factoid questions
• Definition questions
• List questions
• Another options is to build an architecture in
  which question types are processed differently,
  and the semantic representations and inference
  mechanisms are adapted for each question type.

19
The Architecture of LCCs QA System
Multiple Definition Passages
Definition Answer
20
Extracting Answers for Factoid Questions

• In TREC 2003 the LCC QA system extracted 289
  correct answers for factoid questions
• The Name Entity Recognizer was responsible for
  234 of them

QUANTITY 55 ORGANIZATION 15 PRICE 3
NUMBER 45 AUTHORED WORK 11 SCIENCE NAME 2
DATE 35 PRODUCT 11 ACRONYM 1
PERSON 31 CONTINENT 5 ADDRESS 1
COUNTRY 21 PROVINCE 5 ALPHABET 1
OTHER LOCATIONS 19 QUOTE 5 URI 1
CITY 19 UNIVERSITY 3
21
Special Case of Names
Questions asking for names of authored works
1934 What is the play West Side Story based on? Answer Romeo and Juliet
1976 What is the motto for the Boy Scouts? Answer Driving Miss Daisy
1982 What movie won the Academy Award for best picture in 1989? Answer Driving Miss Daisy
2080 What peace treaty ended WWI? Answer Versailles
2102 What American landmark stands on Liberty Island? Answer Statue of Liberty
22
NE-driven QA

• The results of the past 5 TREC evaluations of QA
  systems indicate that current state-of-the-art QA
  is determined by the recognition of Named
  Entities
• Precision of recognition
• Coverage of name classes
• Mapping into concept hierarchies
• Participation into semantic relations (e.g.
  predicate-argument structures or frame semantics)

23
Concept Taxonomies

• For 29 of questions the QA system relied on an
  off-line taxonomy with semantic classes such as
• Disease
• Drugs
• Colors
• Insects
• Games
• The majority of these semantic classes are also
  associated with patterns that enable their
  identification

24
Definition Questions

• They asked about
• PEOPLE (most of them starting with Who)
• other types of NAMES
• general concepts
• People questions
• Many use the PERSON name in the format First
  name, Last name
• examples Aaron Copland, Allen Iverson, Albert
  Ghiorso
• Some names had the PERSON name in format First
  name, Last name1, Last name2
• example Antonia Coello Novello
• Other names had the name as a single word ? very
  well known person
• examples Nostradamus, Absalom, Abraham
• Some questions referred to names of kings or
  princes
• examples Vlad the Impaler, Akbar the Great

25
Answering definition questions

• Most QA systems use between 30-60 patterns
• The most popular patterns

Id Pattern Freq. Usage Question
25 person-hyponym QP 0.43 The doctors also consult with former Italian Olympic skier Alberto Tomba, along with other Italian athletes 1907 Who is Alberto Tomba?
9 QP, the AP 0.28 Bausch Lomb, the company that sells contact lenses, among hundreds of other optical products, has come up with a new twist on the computer screen magnifier 1917 What is Bausch Lomb?
11 QP, a AP 0.11 ETA, a Basque language acronym for Basque Homeland and Freedom _ has killed nearly 800 people since taking up arms in 1968 1987 What is ETA in Spain?
13 QA, an AP 0.02 The kidnappers claimed they are members of the Abu Sayaf, an extremist Muslim group, but a leader of the group denied that 2042 Who is Abu Sayaf?
21 AP such as QP 0.02 For the hundreds of Albanian refugees undergoing medical tests and treatments at Fort Dix, the news is mostly good Most are in reasonable good health, with little evidence of infectious diseases such as TB 2095 What is TB?
26
Complex questions

• Characterized by the need of domain knowledge
• There is no single answer type that can be
  identified, but rather an answer structure needs
  to be recognized
• Answer selection becomes more complicated, since
  inference based on the semantics of the answer
  type needs to be activated
• Complex questions need to be decomposed into a
  set of simpler questions

27
Example of Complex Question
How have thefts impacted on the safety of
Russias nuclear navy, and has the theft problem
been increased or reduced over time?
Need of domain knowledge
To what degree do different thefts put nuclear or
radioactive materials at risk?
Question decomposition

• Definition questions
• What is meant by nuclear navy?
• What does impact mean?
• How does one define the increase or decrease of
  a problem?
• Factoid questions
• What is the number of thefts that are likely to
  be reported?
• What sort of items have been stolen?
• Alternative questions
• What is meant by Russia? Only Russia, or also
  former Soviet
• facilities in non-Russian republics?

28
The answer structure

• For complex questions, the answer structure has a
  compositional semantics, comprising all the
  answer structures of each simpler question in
  which it is decomposed.
• Example

Q-Sem How can a biological weapons program be
detected? Question pattern How can X be
detected? X Biological Weapons Program
Conceptual Schemas
INSPECTION Schema Inspect, Scrutinize, Monitor,
Detect, Evasion, Hide, Obfuscate
POSSESSION Schema Acquire, Possess, Develop,
Deliver
Structure of Complex Answer Type
EVIDENCE CONTENT SOURCE QUALITY JUDGE
RELIABILITY
29
Answer Selection

• Based on the answer structure
• Example
• The CONTENT is selected based on
• Conceptual schemas are instantiated when
  predicate-argument structures or semantic frames
  are recognized in the text passages
• The SOURCE is recognized when the content source
  is identified
• The Quality of the Judgements, the Reliability of
  the judgements and the Judgements themselves are
  produced by an inference mechanism

Structure of Complex Answer Type
EVIDENCE CONTENT SOURCE QUALITY JUDGE
RELIABILITY
30
ANSWER Evidence-Combined Pointer to Text
Source
A1 In recent months, Milton Leitenberg, an
expert on biological weapons, has been looking
at this murkiest and most dangerous corner of
Saddam Hussein's armory.
A2 He says a series of reports add up to
indications that Iraq may be trying to develop a
new viral agent, possibly in underground
laboratories at a military complex near Baghdad
where Iraqis first chased away inspectors six
years ago.
Answer Structure
A3 A new assessment by the United Nations
suggests Iraq still has chemical and biological
weapons - as well as the rockets to deliver them
to targets in other countries.
A4The UN document says Iraq may have hidden a
number of Scud missiles, as well as launchers
and stocks of fuel.
A5 US intelligence believes Iraq still has
stockpiles of chemical and biological weapons and
guided missiles, which it hid from the UN
inspectors
Content Biological Weapons Program
develop(Iraq, Viral_Agent(instance_ofnew)) Justif
ication POSSESSION Schema Previous (Intent and
Ability) Prevent(ability, Inspection)
Inspection terminated Status Attempt ongoing
Likelihood Medium Confirmability difficult,
obtuse, hidden
possess(Iraq, Chemical and Biological Weapons)
Justification POSSESSION Schema Previous
(Intent and Ability) Prevent(ability,
Inspection) Status Hidden from Inspectors
Likelihood Medium
possess(Iraq, delivery systems(type rockets
target other countries)) Justification
POSSESSION Schema Previous (Intent and Ability)
Hidden from Inspectors Status Ongoing
Likelihood Medium
31
Answer Structure (continued)
ANSWER Evidence-Combined Pointer to Text
Source
A1 In recent months, Milton Leitenberg, an
expert on biological weapons, has been looking
at this murkiest and most dangerous corner of
Saddam Hussein's armory.
A2 He says a series of reports add up to
indications that Iraq may be trying to develop a
new viral agent, possibly in underground
laboratories at a military complex near Baghdad
where Iraqis first chased away inspectors six
years ago.
A3 A new assessment by the United Nations
suggests Iraq still has chemical and biological
weapons - as well as the rockets to deliver them
to targets in other countries.
A4The UN document says Iraq may have hidden a
number of Scud missiles, as well as launchers
and stocks of fuel.
A5 US intelligence believes Iraq still has
stockpiles of chemical and biological weapons and
guided missiles, which it hid from the UN
inspectors
Content Biological Weapons Program
possess(Iraq, delivery systems(type scud
missiles launchers target other countries))
Justification POSSESSION Schema Previous
(Intent and Ability) Hidden from Inspectors
Status Ongoing Likelihood Medium
possess(Iraq, fuel stock(purpose power
launchers)) Justification POSSESSION Schema
Previous (Intent and Ability) Hidden from
Inspectors Status Ongoing Likelihood Medium
hide(Iraq, Seeker UN Inspectors Hidden CBW
stockpiles guided missiles) Justification
DETECTION Schema Inspection status Past
Likelihood Medium
32
Answer Structure (continued)
ANSWER Evidence-Combined Pointer to Text
Source
A1 In recent months, Milton Leitenberg, an
expert on biological weapons, has been looking
at this murkiest and most dangerous corner of
Saddam Hussein's armory.
A2 He says a series of reports add up to
indications that Iraq may be trying to develop a
new viral agent, possibly in underground
laboratories at a military complex near Baghdad
where Iraqis first chased away inspectors six
years ago.
A3 A new assessment by the United Nations
suggests Iraq still has chemical and biological
weapons - as well as the rockets to deliver them
to targets in other countries.
A4The UN document says Iraq may have hidden a
number of Scud missiles, as well as launchers
and stocks of fuel.
A5 US intelligence believes Iraq still has
stockpiles of chemical and biological weapons and
guided missiles, which it hid from the UN
inspectors
Source UN documents, US intelligence
SOURCE.Type Assesment reports
Source.Reliability Med-high Likelihood Medium
Judge UN, US intelligence, Milton Leitenberg
(Biological Weapons expert)
JUDGE.Type mixed Judge.manner Judge.stage
ongoing
Quality low-medium Reliability low-medium
33
State-of-the-art QALearning surface text
patterns

• Pioneered by Ravichandran and Hovy (ACL-2002)
• The idea is that given a specific answer type
  (e.g. Birth-Date), learn all surface patterns
  that enable the extraction of the answer from any
  text passage
• Patterns are learned by two algorithms
• Relies on Web redundancy

Algorithm 1 (Generates Patterns) Step 1 Select
an answer type AT and a question Q(AT) Step 2
Generate a query (Q(AT) AT) and submit it
to search engine (google, altavista) Step 3
Download the first 1000 documents Step 4 Select
only those sentences that contain the question
content words and the AT Step 5 Pass the
sentences through a suffix tree
constructor Step 6 Extract only the longest
matching sub-strings that contain the AT and the
question word it is syntactically connected
with.
Algorithm 2 (Measures the Precision of
Patterns) Step 1 Query by using only question
Q(AT) Step 2 Download the first 1000
documents Step 3 Select only those sentences
that contain the question word
connected to the AT Step 4 Compute C(a)
patterns matched by the correct
answer C(0)patterns matched by any word Step
6 The precision of a pattern is given by
C(a)/C(0) Step 7 Retain only patterns matching
gt5 examples
34
Results and Problems

• Some results
• Limitations
• Cannot handle long-distance dependencies
• Cannot recognize paraphrases since no semantic
  knowledge is associated with these patterns
  (unlike patterns used in Information Extraction)
• Cannot recognize a paraphrased questions

Answer TypeINVENTOR ltANSWERgt invents ltNAMEgt the
ltNAMEgt was invented by ltANSWERgt ltANSWERgts
invention of the ltNAMEgt ltANSWERgts ltNAMEgt
was ltNAMEgt, invented by ltANSWERgt That ltANSWERgts
ltNAMEgt
Answer TypeBIRTH-YEAR ltNAMEgt (ltANSWERgt-
) ltNAMEgt was born on ltANSWERgt ltNAMEgt was born in
ltANSWERgt born in ltANSWERgt, ltNAMEgt Of ltNAMEgt,
(ltANSWERgt
35
Shallow semantic parsing

• Part of the problems can be solved by using
  shallow semantic parsers
• Parsers that use shallow semantics encoded as
  either predicate-argument structures or semantic
  frames
• Long-distance dependencies are captured
• Paraphrases can be recognized by mapping on IE
  architectures
• In the past 4 years, several models for training
  such parsers have emerged
• Lexico-Semantic resources are available (e.g
  PropBank, FrameNet)
• Several evaluations measure the performance of
  such parsers (e.g. SENSEVAL, CoNNL)

36
Outline

• Part I. Introduction The need for Semantic
  Inference in QA
• Current State-of-the-art in QA
• Parsing with Predicate Argument Structures
• Parsing with Semantic Frames
• Special Text Relations

37
Proposition Bank Overview

• A one million word corpus annotated with
  predicate argument structures Kingsbury, 2002.
  Currently only predicates lexicalized by verbs.
• Numbered arguments from 0 to 5. Typically ARG0
  agent, ARG1 direct object or theme, ARG2
  indirect object, benefactive, or instrument.
• Functional tags ARMG-LOC locative, ARGM-TMP
  temporal, ARGM-DIR direction.

38
The Model

• Consists of two tasks (1) identifying parse tree
  constituents corresponding to predicate
  arguments, and (2) assigning a role to each
  argument constituent.
• Both tasks modeled using C5.0 decision tree
  learning, and two sets of features Feature Set 1
  adapted from Gildea and Jurafsky, 2002, and
  Feature Set 2, novel set of semantic and
  syntactic features Surdeanu, Harabagiu et al,
  2003.

39
Feature Set 1

• POSITION (pos) indicates if constituent appears
  before predicate in sentence. E.g. true for ARG1
  and false for ARG2.
• VOICE (voice) predicate voice (active or
  passive). E.g. passive for PRED.
• HEAD WORD (hw) head word of the evaluated
  phrase. E.g. halt for ARG1.
• GOVERNING CATEGORY (gov) indicates if an NP is
  dominated by a S phrase or a VP phrase. E.g. S
  for ARG1, VP for ARG0.
• PREDICATE WORD the verb with morphological
  information preserved (verb), and the verb
  normalized to lower case and infinitive form
  (lemma). E.g. for PRED verb is assailed, lemma
  is assail.

• PHRASE TYPE (pt) type of the syntactic phrase as
  argument. E.g. NP for ARG1.
• PARSE TREE PATH (path) path between argument
  and predicate. E.g. NP ? S ? VP ? VP for ARG1.
• PATH LENGTH (pathLen) number of labels stored in
  the predicate-argument path. E.g. 4 for ARG1.

40
Observations about Feature Set 1

• Because most of the argument constituents are
  prepositional attachments (PP) and relative
  clauses (SBAR), often the head word (hw) is not
  the most informative word in the phrase.
• Due to its strong lexicalization, the model
  suffers from data sparsity. E.g. hw used lt 3.
  The problem can be addressed with a back-off
  model from words to part of speech tags.
• The features in set 1 capture only syntactic
  information, even though semantic information
  like named-entity tags should help. For example,
  ARGM-TMP typically contains DATE entities, and
  ARGM-LOC includes LOCATION named entities.
• Feature set 1 does not capture predicates
  lexicalized by phrasal verbs, e.g. put up.

41
Feature Set 2 (1/2)

• CONTENT WORD (cw) lexicalized feature that
  selects an informative word from the constituent,
  other than the head. Selection heuristics
  available in the paper. E.g. June for the
  phrase in last June.
• PART OF SPEECH OF CONTENT WORD (cPos) part of
  speech tag of the content word. E.g. NNP for the
  phrase in last June.
• PART OF SPEECH OF HEAD WORD (hPos) part of
  speech tag of the head word. E.g. NN for the
  phrase the futures halt.
• NAMED ENTITY CLASS OF CONTENT WORD (cNE) The
  class of the named entity that includes the
  content word. 7 named entity classes (from the
  MUC-7 specification) covered. E.g. DATE for in
  last June.

42
Feature Set 2 (2/2)

• BOOLEAN NAMED ENTITY FLAGS set of features that
  indicate if a named entity is included at any
  position in the phrase
• neOrganization set to true if an organization
  name is recognized in the phrase.
• neLocation set to true if a location name is
  recognized in the phrase.
• nePerson set to true if a person name is
  recognized in the phrase.
• neMoney set to true if a currency expression is
  recognized in the phrase.
• nePercent set to true if a percentage expression
  is recognized in the phrase.
• neTime set to true if a time of day expression
  is recognized in the phrase.
• neDate set to true if a date temporal expression
  is recognized in the phrase.
• PHRASAL VERB COLLOCATIONS set of two features
  that capture information about phrasal verbs
• pvcSum the frequency with which a verb is
  immediately followed by any preposition or
  particle.
• pvcMax the frequency with which a verb is
  followed by its predominant preposition or
  particle.

43
Results
Features Arg P Arg R Arg F1 Role A
FS1 84.96 84.26 84.61 78.76
FS1 POS tag of head word 92.24 84.50 88.20 79.04
FS1 content word and POS tag 92.19 84.67 88.27 80.80
FS1 NE label of content word 83.93 85.69 84.80 79.85
FS1 phrase NE flags 87.78 85.71 86.73 81.28
FS1 phrasal verb information 84.88 82.77 83.81 78.62
FS1 FS2 91.62 85.06 88.22 83.05
FS1 FS2 boosting 93.00 85.29 88.98 83.74
44
Other parsers based on PropBank

• Pradhan, Ward et al, 2004 (HLT/NAACLJ of ML)
  report on a parser trained with SVMs which
  obtains F1-score90.4 for Argument
  classification and 80.8 for detecting the
  boundaries and classifying the arguments, when
  only the first set of features is used.
• Gildea and Hockenmaier (2003) use features
  extracted from Combinatory Categorial Grammar
  (CCG). The F1-measure obtained is 80
• Chen and Rambow (2003) use syntactic and semantic
  features extracted from a Tree Adjoining Grammar
  (TAG) and report an F1-measure of 93.5 for the
  core arguments
• Pradhan, Ward et al, use a set of 12 new features
  and obtain and F1-score of 93.8 for argument
  classification and 86.7 for argument detection
  and classification

45
Applying Predicate-Argument Structures to QA

• Parsing Questions
• Parsing Answers
• Result exact answer approximately 7 kg of HEU

A(Q) Russias Pacific Fleet has also fallen prey
to nuclear theft in 1/96, approximately 7 kg
of HEU was reportedly stolen from a naval base
in Sovetskaya Gavan.
PAS(A(Q)) Arg1(P1redicate 1) Russias Pacific
Fleet has ArgM-Dis(Predicate 1) also
Predicate 1 fallen Arg1(Predicate 1) prey
to nuclear theft ArgM-TMP(Predicate 2) in
1/96, Arg1(Predicate 2) approximately 7 kg of
HEU was ArgM-ADV(Predicate 2) reportedly
Predicate 2 stolen Arg2(Predicate 2) from
a naval base Arg3(Predicate 2) in Sovetskawa
Gavan
46
Outline

• Part I. Introduction The need for Semantic
  Inference in QA
• Current State-of-the-art in QA
• Parsing with Predicate Argument Structures
• Parsing with Semantic Frames
• Special Text Relations

47
The Model
S
VP
NP
NP
PP
Task 1
She
clapped
her hands
in inspiration
PRED
Agent
Cause
Body Part
Task 2

• Consists of two tasks (1) identifying parse tree
  constituents corresponding to frame elements, and
  (2) assigning a semantic role to each frame
  element.
• Both tasks introduced for the first time by
  Gildea and Jurafsky in 2000. It uses the Feature
  Set 1 , which later Gildea and Palmer used for
  parsing based on PropBank.

48
Extensions

• Fleischman et al extend the model in 2003 in
  three ways
• Adopt a maximum entropy framework for learning a
  more accurate classification model.
• Include features that look at previous tags and
  use previous tag information to find the highest
  probability for the semantic role sequence of any
  given sentence.
• Examine sentence-level patterns that exploit more
  global information in order to classify frame
  elements.

49
Applying Frame Structures to QA

• Parsing Questions
• Parsing Answers
• Result exact answer approximately 7 kg of HEU

A(Q) Russias Pacific Fleet has also fallen prey
to nuclear theft in 1/96, approximately 7 kg
of HEU was reportedly stolen from a naval base
in Sovetskaya Gavan.
FS(A(Q)) VICTIM(P1) Russias Pacific Fleet
has also fallen prey to Goods(P1) nuclear
Target-Predicate(P1) theft in 1/96,
GOODS(P2) approximately 7 kg of HEU was
reportedly Target-Predicate (P2) stolen
VICTIM (P2) from a naval base SOURCE(P2)
in Sovetskawa Gavan
50
Outline

• Part I. Introduction The need for Semantic
  Inference in QA
• Current State-of-the-art in QA
• Parsing with Predicate Argument Structures
• Parsing with Semantic Frames
• Special Text Relations

51
Additional types of relations

• Temporal relations
• TERQUAS ARDA Workshop
• Causal relations
• Evidential relations
• Part-whole relations

52
Temporal relations in QA

• Results of the workshop are accessible from
  http//www.cs.brandeis.edu/jamesp/arda/time/docum
  entation/TimeML-use-in-qa-v1.0.pdf
• A set of questions that require the extraction
  of temporal relations was created (TimeML
  question corpus)
• E.g.
• When did the war between Iran and Iraq end?
• Who was Secretary of Defense during the Golf
  War?
• A number of features of these questions were
  identified and annotated
• E.g.
• Number of TEMPEX relations in the question
• Volatility of the question (how often does the
  answer change)
• Reference to repetitive events
• Number of events mentioned in the question

53
Outline

• Part II. Extracting Semantic Relations from
  Questions and Texts
• Knowledge-intensive techniques
• Unsupervised techniques

54
Information Extraction from texts

• Extracting semantic relations from questions and
  texts can be solved by adapting the IE technology
  to this new task.
• What is Information Extraction (IE) ?
• The task of finding facts about a specified class
  of events from free text
• Filling a table in a database with the
  information sush a database entry can be seen
  as a list of slots of a template
• Events are instances comprising many relations
  that span multiple arguments

55
IE Architecture Overview
Phrasal parser
Domain API
56
Walk-through Example
... a bomb rigged with a trip wire that exploded
and killed him...
57
Learning domain event rulesand domain relations

• build patterns from examples
• Yangarber 97
• generalize from multiple examples annotated text
• Crystal, Whisk (Soderland), Rapier (Califf)
• active learning reduce annotation
• Soderland 99, Califf 99
• learning from corpus with relevance judgements
• Riloff 96, 99
• co-learning/bootstrapping
• Brin 98, Agichtein 00

58
Changes in IE architecture for enabling the
extraction of semantic relations
Document
Tokenizer

• Addition of Relation Layer
• Modification of NE and
• pronominal coreference
• to enable relation coreference
• Add a relation merging
• layer

Entity Coreference
Entity Recognizer
Event Recognizer
Relation Recognizer
Event/Relation Coreference
Relation Merging
EEML File Generation
EEML Results
59
Walk-through Example
Event Murder
The murder of Vladimir Golovlyov, an associate of
the exiled tycoon Boris Berezovsky, was
the second contract killing in the Russian
capital in as many days and capped a week of
setbacks for the Russian leader.
Event Murder
60
Walk-through Example
Event-Entity Relation Victim
Entity-Entity Relation AffiliatedWith
The murder of Vladimir Golovlyov, an associate of
the exiled tycoon Boris Berezovsky, was
the second contract killing in the Russian
capital in as many days and capped a week of
setbacks for the Russian leader.
Event-Entity Relation Victim
Event-Entity Relation EventOccurAt
Entity-Entity Relation GeographicalSubregion
Entity-Entity Relation hasLeader
61
Application to QA

• Who was murdered in Moscow this week?
• Relations EventOccuredAt Victim
• Name some associates of Vladimir Golovlyov.
• Relations AffiliatedWith
• How did Vladimir Golovlyov die?
• Relations Victim
• What is the relation between Vladimir Golovlyov
  and Boris Berezovsky?
• Relations AffliliatedWith

62
Outline

• Part II. Extracting Semantic Relations from
  Questions and Texts
• Knowledge-intensive techniques
• Unsupervised techniques

63
Learning extraction rules and semantic lexicons

• Generating Extraction Patterns AutoSlog (Riloff
  1993), AutoSlog-Ts(Riloff 1996)
• Semantic Lexicon Induction Riloff Shepherd
  (1997), Roark Charniak (1998), Ge, Hale,
  Charniak (1998), Caraballo (1999), Thompson
  Mooney (1999), Meta-Bootstrapping (Riloff Jones
  1999), (Thelen and Riloff 2002)
• Bootstrapping/Co-training Yarowsky (1995), Blum
  and Mitchell (1998), McCallum Nigam (1998)

64
Generating extraction rules

• From untagged text AutoSlog-TS (Riloff 1996)
• The rule relevance is measured by
• Relevance rate log2 (frequency)

STAGE 1
Pre-classified Texts
Concept Nodes ltxgt was bombed by ltygt
Subject World Trade Center Verb was bombed PP
by terrorists
Sentence Analyzer
AutoSlog Heuristics
STAGE 2
Concept Nodes REL ltxgt was bombed 87
bombed by ltygt 84 ltwgt was killed
63 ltzgt saw 49
Pre-classified Texts
Concept Node Dictionary ltxgt was killed ltxgt was
bombed by ltygt
Sentence Analyzer
65
Learning Dictionaries for IE with mutual
bootrapping

• Riloff and Jones (1999)

Generate all candidate extraction rules rom the
training corpus using AutoSlog
Apply the candidate extraction rules to the
training corpus and save the patterns With their
extractions to EPdata
SemLEx seed words Cat_EPlist

• MUTUAL BOOTSTRAPPING LOOP
• Score all extraction rules in Epdata
• best_EP the highest scoring extraction pattern
  not already in Cat_Eplist
• Add best_EP to Cat_Eplist
• Add best_EPs extraction to SemLEx
• Go to step 1.

66
The BASILISK approach (Thelen Riloff)
BASILISK Bootstrapping Approach to SemantIc
Lexicon Induction using Semantic Knowledge
corpus
Key ideas 1/ Collective evidence over a large
set of extraction patterns can reveal
strong semantic associations. 2/ Learning
multiple categories simultaneously can constrain
the bootstrapping process
best patterns
extractions
semantic lexicon
5 best candidate words
67
Learning Multiple Categories Simultaneously

• One Sense per Domain assumption a word belongs
  to a single semantic category within a limited
  domain.
• The simplest way to take advantage of multiple
  categories is to resolve conflicts when they
  arise.
• 1. A word cannot be assigned to category X if it
  has already been assigned to category Y.
• 2. If a word is hypothesized for both category X
  and category Y at the same time, choose the
  category that receives the highest score.

Bootstrapping multiple categories
Bootstrapping a single category
68
Kernel Methods for Relation Extraction

• Pioneered by Zelenko, Aone and Richardella (2002)
• Uses Support Vector Machines and the Voted
  Perceptron Alorithm (Freund and Shapire, 1999)
• It operates on the shallow parses of texts, by
  using two functions
• A matching function between the nodes of the
  shallow parse tree and
• A similarity function between the nodes
• It obtains very high F1-score values for relation
  extraction (86.8)

69
Outline

• Part III. Knowledge representation and inference
• Representing the semantics of answers
• Extended WordNet and abductive inference
• Intentional Structure and Probabilistic Metonymy
• An example of Event Structure
• Modeling relations, uncertainty and dynamics
• Inference methods and their mapping to answer
  types

70
Three representations

• A taxonomy of answer types in which Named Entity
  Classes are also mapped.
• A complex structure that results from schema
  instantiations
• Answer type generated by the inference on the
  semantic structures

71
Possible Answer Types
TOP
PERSON LOCATION DATE TIME PRODUCT NUMERICAL
MONEY ORGANIZATION MANNER REASON

VALUE
DEGREE DIMENSION RATE DURATION PERCENTAGE
COUNT
72
Examples
PRODUCT
PERSON
PERSON
PRODUCT
73
Outline

• Part III. Knowledge representation and inference
• Representing the semantics of answers
• Extended WordNet and abductive inference
• Intentional Structure and Probabilistic Metonymy
• An example of Event Structure
• Modeling relations, uncertainty and dynamics
• Inference methods and their mapping to answer
  types

74
Extended WordNet

• eXtended WordNet is an ongoing project at the
  Human Language Technology Research Institute,
  University of Texas at Dallas. http//xwn.hlt.utda
  llas.edu/)
• The goal of this project is to develop a tool
  that takes as input the current or future
  versions of WordNet and automatically generates
  an eXtended WordNet that provides several
  important enhancements intended to remedy the
  present limitations of WordNet.
• In the eXtended WordNet the WordNet glosses are
  syntactically parsed, transformed into logic
  forms and content words are semantically
  disambiguated.

75
Logic Abduction

• Motivation
• Goes beyond keyword based justification by
  capturing
• syntax based relationships
• links between concepts in the question and the
  candidate answers

Axiom Builder
Justification
QLF ALF XWN axioms NLP
axioms Lexical chains
Answer Ranking
Ranked answers
Success
Answer explanation
Proof fails
Relaxation
76
COGEX the LCC Logic Prover for QA

• Inputs to the Logic Prover
• A logic form provides a mapping of the question
  and candidate answer text into first order logic
  predicates.
• Question
• Where did bin Laden 's funding come from other
  than his own wealth ?
• Question Logic Form
• ( _multi_AT(x1) ) bin_NN_1(x2) Laden_NN(x3)
  _s_POS(x5,x4) nn_NNC(x4,x2,x3)
  funding_NN_1(x5) come_VB_1(e1,x5,x11)
  from_IN(e1,x1) other_than_JJ_1(x6)
  his_PRP_(x6,x4) own_JJ_1(x6) wealth_NN_1(x6)

77
Justifying the answer

• Answer
• ... Bin Laden reportedly sent representatives to
  Afghanistan opium farmers to buy large amounts of
  opium , probably to raise funds for al - Qaida
  ....
• Answer Logic Form
• Bin_NN(x14) Laden_NN(x15)
  nn_NNC(x16,x14,x15) reportedly_RB_1(e2)
  send_VB_1(e2,x16,x17) representative_NN_1(x17)
  to_TO(e2,x21) Afghanistan_NN_1(x18)
  opium_NN_1(x19) farmer_NN_1(x20)
  nn_NNC(x21,x19,x20) buy_VB_5(e3,x17,x22)
  large_JJ_1(x22) amount_NN_1(x22)
  of_IN(x22,x23) opium_NN_1(x23)
  probably_RB_1(e4) raise_VB_1(e4,x22,x24)
  funds_NN_2(x24) for_IN(x24,x26) al_NN_1(x25)
  Qaida_NN(x26) ...

78
Lexical Chains

• Lexical Chains
• Lexical chains provide an improved source of
  world knowledge by supplying the Logic Prover
  with much needed axioms to link question keywords
  with answer concepts.
• Question
• How were biological agents acquired by bin
  Laden?
• Answer
• On 8 July 1998 , the Italian newspaper Corriere
  della Serra indicated that members of The World
  Front for Fighting Jews and Crusaders , which was
  founded by Bin Laden , purchased three chemical
  and biological_agent production facilities in
• Lexical Chain
• ( v - buy1, purchase1 ) HYPERNYM ( v - get1,
  acquire1 )

79
Axiom selection

• XWN Axioms
• Another source of world knowledge is a general
  purpose knowledge base of more than 50,000 parsed
  and disambiguated glosses that are transformed
  into logic form for use during the course of a
  proof.
• Gloss
• Kill is to cause to die
• GLF
• kill_VB_1(e1,x1,x2) -gt cause_VB_1(e1,x1,x3)
  to_TO(e1,e2) die_VB_1(e2,x2,x4)

80
Logic Prover

• Axiom Selection
• Lexical chains and the XWN knowledge base work
  together to select and generate the axioms needed
  for a successful proof when all the keywords in
  the questions are not found in the answer.
• Question
• How did Adolf Hitler die?
• Answer
• Adolf Hitler committed suicide
• The following Lexical Chain is detected
• ( n - suicide1, self-destruction1,
  self-annihilation1 ) GLOSS ( v - kill1 ) GLOSS
  ( v - die1, decease1, perish1, go17, exit3,
  pass_away1, expire2, pass25 ) 2
• The following axioms are loaded into the Usable
  List of the Prover
• exists x2 all e1 x1 (suicide_nn(x1) -gt
  act_nn(x1) of_in(x1,e1) kill_vb(e1,x2,x2)).
• exists x3 x4 all e2 x1 x2 (kill_vb(e2,x1,x2)
  -gt cause_vb_2(e1,x1,x3) to_to(e1,e2)
  die_vb(e2,x2,x4)).

81
Outline

• Part III. Knowledge representation and inference
• Representing the semantics of answers
• Extended WordNet and abductive inference
• Intentional Structure and Probabilistic Metonymy
• An example of Event Structure
• Modeling relations, uncertainty and dynamics
• Inference methods and their mapping to answer
  types

82
Intentional Structure of Questions

• Example Does have
  ?
• x y
• Predicate-argument have/possess (Iraq, biological
  weapons)
• structure Arg-0 Arg-1
• Question Pattern possess (x,y)
• Intentional Structure

biological weapons
Iraq
Evidence ??? Coercion Means of Finding ??? Coercion Source ??? Coercion Consequence ??? Coercion
83
Coercion of Pragmatic Knowledge

• 0Evidence (1-possess (2-Iraq, 3-biological
  weapons)
• A form of logical metonymy

Lapata and Lascarides (Computational Linguistics,2003) ?allows coercion of interpretations by collecting possible meanings from large corpora. Examples Mary finished the cigarette ? Mary finished smoking the cigarette. Arabic is a difficult language ? Arabic is a language that is difficult to learn ? Arabic is a language that is difficult to process automatically
84
The Idea

• Logic metonymy is in part processed as verbal
  metonymy. We model, after Lapata and Lascarides,
  the interpretation of verbal metonymy as
• where vthe metonymic verb (enjoy)
• oits object (the cigarette)
• ethe sought-after interpretation
  (smoking)

85
A probabilistic model

• By choosing the ordering , the
  probability may be factored as
• where we make the estimations

This is a model of interpretation and coercion
86
Coercions for intentional structures

• 0Evidence (1-possess (2-Iraq, 3-biological
  weaponry)

vdiscover (1,2,3) Vstockpile (2,3) Vuse (2,3) V0 (1,2,3)
edevelop (?,3) eacquire (? ,3)
einspections (? ,2, 3) eban (? , 2, from 3)
Topic Coercion
87
Outline

• Part III. Knowledge representation and inference
• Representing the semantics of answers
• Extended WordNet and abductive inference
• Intentional Structure and Probabilistic Metonymy
• An example of Event Structure
• Modeling relations, uncertainty and dynamics
• Inference methods and their mapping to answer
  types

88
ANSWER Evidence-Combined Pointer to Text
Source
A1 In recent months, Milton Leitenberg, an
expert on biological weapons, has been looking
at this murkiest and most dangerous corner of
Saddam Hussein's armory.
A2 He says a series of reports add up to
indications that Iraq may be trying to develop a
new viral agent, possibly in underground
laboratories at a military complex near Baghdad
where Iraqis first chased away inspectors six
years ago.
Answer Structure
A3 A new assessment by the United Nations
suggests Iraq still has chemical and biological
weapons - as well as the rockets to deliver them
to targets in other countries.
A4The UN document says Iraq may have hidden a
number of Scud missiles, as well as launchers
and stocks of fuel.
A5 US intelligence believes Iraq still has
stockpiles of chemical and biological weapons and
guided missiles, which it hid from the UN
inspectors
Content Biological Weapons Program
develop(Iraq, Viral_Agent(instance_ofnew)) Justif
ication POSSESSION Schema Previous (Intent and
Ability) Prevent(ability, Inspection)
Inspection terminated Status Attempt ongoing
Likelihood Medium Confirmability difficult,
obtuse, hidden
possess(Iraq, Chemical and Biological Weapons)
Justification POSSESSION Schema Previous
(Intent and Ability) Prevent(ability,
Inspection) Status Hidden from Inspectors
Likelihood Medium
possess(Iraq, delivery systems(type rockets
target other countries)) Justification
POSSESSION Schema Previous (Intent and Ability)
Hidden from Inspectors Status Ongoing
Likelihood Medium
89
ANSWER Evidence-Combined Pointer to Text
Source
A1 In recent months, Milton Leitenberg, an
expert on biological weapons, has been looking
at this murkiest and most dangerous corner of
Saddam Hussein's armory.
A2 He says a series of reports add up to
indications that Iraq may be trying to develop a
new viral agent, possibly in underground
laboratories at a military complex near Baghdad
where Iraqis first chased away inspectors six
years ago.
Answer Structure
A3 A new assessment by the United Nations
suggests Iraq still has chemical and biological
weapons - as well as the rockets to deliver them
to targets in other countries.
A4The UN document says Iraq may have hidden a
number of Scud missiles, as well as launchers
and stocks of fuel.
A5 US intelligence believes Iraq still has
stockpiles of chemical and biological weapons and
guided missiles, which it hid from the UN
inspectors
Content Biological Weapons Program
develop(Iraq, Viral_Agent(instance_ofnew)) Justif
ication POSSESSION Schema Previous (Intent and
Ability) Prevent(ability, Inspection)
Inspection terminated Status Attempt ongoing
Likelihood Medium Confirmability difficult,
obtuse, hidden
possess(Iraq, Chemical and Biological Weapons)
Justification POSSESSION Schema Previous
(Intent and Ability) Prevent(ability,
Inspection) Status Hidden from Inspectors
Likelihood Medium
possess(Iraq, delivery systems(type rockets
target other countries)) Justification
POSSESSION Schema Previous (Intent and Ability)
Hidden from Inspectors Status Ongoing
Likelihood Medium
90
ANSWER Evidence-Combined Pointer to Text
Source
A1 In recent months, Milton Leitenberg, an
expert on biological weapons, has been looking
at this murkiest and most dangerous corner of
Saddam Hussein's armory.
A2 He says a series of reports add up to
indications that Iraq may be trying to develop a
new viral agent, possibly in underground
laboratories at a military complex near Baghdad
where Iraqis first chased away inspectors six
years ago.
Answer Structure
A3 A new assessment by the United Nations
suggests Iraq still has chemical and biological
weapons - as well as the rockets to deliver them
to targets in other countries.
A4The UN document says Iraq may have hidden a
number of Scud missiles, as well as launchers
and stocks of fuel.
Temporal Reference/Grounding
A5 US intelligence believes Iraq still has
stockpiles of chemical and biological weapons and
guided missiles, which it hid from the UN
inspectors
Content Biological Weapons Program
develop(Iraq, Viral_Agent(instance_ofnew)) Justif
ication POSSESSION Schema Previous (Intent and
Ability) Prevent(ability, Inspection)
Inspection terminated Status Attempt ongoing
Likelihood Medium Confirmability difficult,
obtuse, hidden
possess(Iraq, Chemical and Biological Weapons)
Justification POSSESSION Schema Previous
(Intent and Ability) Prevent(ability,
Inspection) Status Hidden from Inspectors
Likelihood Medium
possess(Iraq, delivery systems(type rockets
target other countries)) Justification
POSSESSION Schema Previous (Intent and Ability)
Hidden from Inspectors Status Ongoing
Likelihood Medium
91
Answer Structure (continued)
ANSWER Evidence-Combined Pointer to Text
Source
A1 In recent months, Milton Leitenberg, an
expert on biological weapons, has been looking
at this murkiest and most dangerous corner of
Saddam Hussein's armory.
A2 He says a series of reports add up to
indications that Iraq may be trying to develop a
new viral agent, possibly in underground
laboratories at a military complex near Baghdad
where Iraqis first chased away inspectors six
years ago.
Present Progressive Perfect
A3 A new assessment by the United Nations
suggests Iraq still has chemical and biological
weapons - as well as the rockets to deliver them
to targets in other countries.
A4The UN document says Iraq may have hidden a
number of Scud missiles, as well as launchers
and stocks of fuel.
A5 US intelligence believes Iraq still has
stockpiles of chemical and biological weapons and
guided missiles, which it hid from the UN
inspectors
Content Biological Weapons Program
possess(Iraq, delivery systems(type scud
missiles launchers target other countries))
Justification POSSESSION Schema Previous
(Intent and Ability) Hidden from Inspectors
Status Ongoing Likelihood Medium
Present Progressive Continuing
possess(Iraq, fuel stock(purpose power
launchers)) Justification POSSESSION Schema
Previous (Intent and Ability) Hidden from
Inspectors Status Ongoing Likelihood Medium
hide(Iraq, Seeker UN Inspectors Hidden CBW
stockpiles guided missiles) Justification
DETECTION Schema Inspection status Past
Likelihood Medium
92
ANSWER Evidence-Combined Pointer to Text
Source
A1 In recent months, Milton Leitenberg, an
expert on biological weapons, has been looking
at this murkiest and most dangerous corner of
Saddam Hussein's armory.
A2 He says a series of reports add up to
indications that Iraq may be trying to develop a
new viral agent, possibly in underground
laboratories at a military complex near Baghdad
where Iraqis first chased away inspectors six
years ago.
Answer Structure
A3 A new assessment by the United Nations
suggests Iraq still has chemical and biological
weapons - as well as the rockets to deliver them
to targets in other countries.
A4The UN document says Iraq may have hidden a
number of Scud missiles, as well as launchers
and stocks of fuel.
Uncertainty and Belief
A5 US intelligence believes Iraq still has
stockpiles of chemical and biological weapons and
guided missiles, which it hid from the UN
inspectors
Content Biological Weapons Program
develop(Iraq, Viral_Agent(instance_ofnew)) Justif
ication POSSESSION Schema Previous (Intent and
Ability) Prevent(ability, Inspection)
Inspection terminated Status Attempt ongoing
Likelihood Medium Confirmability difficult,
obtuse, hidden