PASCAL Pump Priming Project on Textual Entailment

1
PASCAL Pump Priming Projecton Textual Entailment

• Bar Ilan (Israel)CNTS (Antwerp)IDIAP
  (Martigny)XRCE (Grenoble)
• 2004-2005

2
Project Outline Lexical Entailment

• Bar Ilan defines entailment framework and lexical
  entailment models (predicting words conceptual
  presence in a context)
• IDIAP develops an alternative lexical model,
  motivated in classical IR terms (LM style)
• CNTS develops a subtitling application which
  incorporates the above lexical models to predict
  subtitle entailment (with BIU and IDIAP
  involvement)
• XRCE develops an IR model which incorporates
  lexical entailment reasoning (with BIU)

3
Bar IlanProblem definition and
datasetsProbabilistic SettingTwo probabilistic
lexical models

• Ido DaganOren Glickman

4
Applied Textual Entailment Abstracting
Variability Inference
Hypothesis (h) Elizabeth Dole was born in North
Carolina

• QA Where was Elizabeth Dole Born?
• Answer Iowa
• IE born in
• Summarization
• Identify redundancies

Applied Textual Entailment
Text (t) Doles wife, Elizabeth, is a native of
NC
5
Foundations and Infrastructure

• considerable time and effort were spent on
• understanding the problem
• coming up with task definition
• understanding the relation to applications
• designing evaluation datasets
• cf. the Recognizing Textual Entailment PASCAL
  challenge

6
Probabilistic Textual Entailment

• Definition
• t probabilistically entails h (t ?h) if
• P(Trh 1 t) gt P(Trh 1)
• t increases the likelihood of h being true
• Positive PMI t provides information on hs
  truth
• P(Trh 1 t) entailment confidence (required to
  be high)
• This project entailment modeled only at the
  lexical level

7
Lexical Alignment (1st Model)

• Based on the probabilistic setting
• Assumes each word in h is entailed from one word
  in t (similar to early translation models)
  estimate correspondingly P(w1w2)
• Utilizes word co-occurrence probabilities
• Published at ACL-05 entailment workshop (PASCAL
  related event)

8
Results on RTE 1 dataset

• Top accuracy at PASCAL RTE challenge (Southampton)

9
Contextual entailment (2nd model) as a text
categorization task

• Derive entailment from all context words
• rather than word-to-word alignment
• View lexical entailment as a text categorization
  task
• where every word meaning is a category
• Does car fit in this text?
• estimate probability that word meaning is
  entailed from whole text
• Unsupervised application of Naïve Bayes
• EM Applied to estimate hidden truth values
• Published at AAAI-05

10
IDIAPLexical Models

• Samy BengioMikaela Keller

11
Document Representation for Entailment

• Models for Document Representation Documents are
  usually represented in the "bag of words" space
  (huge dimensionality, very sparse).
• A richer representation of documents (more
  compact and more general) could improve the
  solutions to tasks defined by the Textual
  Entailment framework.
• We have explored two different approaches
• a probabilistic approach
• a stochastic gradient approach

12
First intent Theme Topic Mixture Model (TTMM)

• Presented at PASCAL Workshop on Learning Methods
  for Text Understanding and Mining, 2004.
• TTMM is a graphical model in the spirit of PLSA
  and LDA.
• TTMM can be trained by EM on unlabeled data.
• Yields results similar to PLSA and LDA on Reuters
  text classification task.
• Unfortunately, TTMM does not scale well to large
  databases.

13
Second intent Neural Network for Text
Representation (NNTR).

• Presented at Learning_at_Snowbird'2005 and
  ICANN'2005.
• Same spirit as NN for Language Modeling (Bengio
  Y. et al, JMLR, 2003)
• Based on Multi Layered Perceptrons
• MLPword MLPdocument transpose word/document to
  rich distributed space.
• MLPword/document estimates the joint target (same
  context? Or not)
• Margin criterion, stochastic Gradient descent on
  unlabeled data.
• Yields better performance than TFIDF and PLSA on
  a retrieval task and a filtering task using the
  TDT2-based TREC-8 and TREC-9 databases.
• Suitable to adapt directly to assess Lexical
  Entailment probability (of word given document,
  analogically to BIU model see Subtitling part
  next)

14
Architecture and Results
15
CNTSSubtitling Application forEvaluation of
Entailment

• Walter DaelemansAnja Hoethker

16
Subtitling project MUSA

• Serves as applicative evaluation framework for
  BIU and IDIAP entailment models
• EU IST, 5th framework Sep. 2002-Feb 2005
• Goals
• Conversion of audio streams into TV subtitles
  (monolingual)
• Translation of subtitles into Greek or French
• CNTS contribution sentence compression

17
Entailment in subtitle generation

• Subtitles have to comply to time and space
  restrictions
• Due to these limitations text may have to be
  compressed
• Transcript entails subtitle
• As much as possible, subtitle should entail
  transcript (no information loss, or minimal loss
  if needed)

18
Example

• It was clear that he wasnt very impressed.
• Clearly, he wasnt impressed.

19
Compression methods

• Sentences can be compressed through
• Deletion of irrelevant chunks (apply linguistic
  rules based on shallow parser annotation)
• Substitution with shorter text while preserving
  information
• Look for shorter synonyms in WordNet, or
  paraphrases in a lexicon extracted from parallel
  corpus

20
Rate compression possibilities

• Usually only little compression (a few
  words/characters) is required
• Entailment models can be used to rate the
  different WordNet suggestions (Bar-Ilan, IDIAP,
  n-gram model,)
• Compression that produces the highest entailment
  score should be chosen as subtitle

21
Evaluation

• The output of the entailment models is evaluated
  relative to the human judgments of the candidate
  substitutions
• Results showed that the two posterior lexical
  entailment models, which score the substitution
  according to the context of its application, were
  inferior than prior (context-independent)
  scores for the substitution.
• In subtitling application, broader context is
  less effective, but better models may still
  improve results.

22
Results
23
XRCE Textual Entailment for IR

• Stephane ClinchantEric GaussierCyril Goutte

24
Textual Entailment for Information Retrieval

• XRCEs planned contribution
• Does TE yield improved results in IR?
• Can we leverage IR models to improve estimation
  of entailment probabilities?
• Springsummer 2005 work on 1. use the language
  modelling approach to IR and TE probabilities.

25
The language model approach to IRCroftLafferty
(2003) Language Modeling for Information
Retrieval

• Evaluate the distance of the query from the
  document

Standard document model
Query model
26
Textual entailment as translation

• Three approaches to textual entailment
  probability

1. Lexical entailment (Bar Ilan Univ. model)
Problem overestimate probability of frequent
words!
2. Lexical similarity (cf. XRCE model for
cross-lingual similarity)
Problem is too low reinforce
the diagonal
3. Set similarity, eg
27
Information Retrieval results

• Carried out with Lemur toolkit
  http//www.lemurproject.org/
• tf.idf simple tf.idf weighting for ad hoc
  retrieval
• KL standard language model approach with KL
  divergence
• KLTE LM approach with textual entailment
  probabilities

relevant avg precision
tf.idf 963 34.59
KL 977 53.18
KLTE 991 56.10
Corpus CLEF2003, 160K documents,1006 relevant on
54 topics
28
Meetings

• ACL 2004 (BIU, XRCE, CNTS)
• COLING 2004 (BIU, XRCE, IDIAP)
• Nov. 2004 at XRCE (Several days, with BIU and
  IDIAP)
• Southampton 2005 (BIU, CNTS)
• ACL 2005 (BIU, XRCE, CNTS)
• Common WIKI and mailing list
• 15 papers and reports

29
Follow up activities XRCE

• Participation in CLEF 2007 Domain Specific IR
  Task
• Lexical entailment models find relations between
  terms that pseudo feedback can not find
• Improvement from 45 to 50 MAP w. lexical
  entailment

30
Follow up activities CNTS

• DAESO Joint project with Tilburg University
  (Emiel Krahmer and Erwin Marsi)
• Semantic entailment in Dutch.
• For sentence fusion in multi-document
  summarization.

31
Follow up activities BIU

• Further work and evaluation study on lexical
  entailment and reference (EMNLP 2006)
• Lexical entailment for implicit word sense
  disambiguation in context for lexical
  substitution (ACL 06)
• Applying lexical entailment for IR (project
  funded by Ministry of Defense)
• An unsupervised lexical-entailment approach for
  standard text categorization datasets (academic-
  industrial consortium funded by Ministry of Trade
  Industry)

32
Takeout New Research Problems

• Identify prior likelihood of lexical entailment
  substitution between words
• Identify posterior likelihood of lexical
  entailment in context (local and global context
  features)
• Possible evaluation tasks
• Unsupervised learning for standard text
  categorization datasets, taking only category
  name as input
• Apply purely lexical methods to RTE datasets

Thank You!
33
A Generative Model

• We assume a probabilistic generative model
• A generation event of a text along with a hidden
  possible world (truth assignments) based on a
  corresponding joint probability space

John was born in France (t)
John Speaks French ? 1John was born in Paris
? 0 John was born in Milan ? 0 John is married
to Alice ? 1 (w)
34
Project Next Steps

• Complete the implementation and integration of
  subtitling application and conduct tests
• Complete IR experiments
• Improve models according to results and mutual
  feedback
• Write 2 joint papers based on the models and
  experiments for the two application areas