Summarisation and the RTE Challenge

1
Summarisation and the RTE Challenge
Nicola Stokes (with some modifications by Ido
Dagan) NICTA Victoria Research Lab, Department
of Computer Science and Software
Engineering, University of Melbourne
2
MDS and RTE

• Why are summarisation researchers interested in
  RTE?
• Provides us with a means of evaluating a core
  component in summarisation systems Redundant
  Information Identification
• For example, only one of these sentences should
  be included in the summary
• The highly contagious disease, which broke out
  in the district of Gulu, causes its victims to
  bleed to death.
• This is a highly contagious and deadly
  disease.
• However, full entailment relationships are
  rarely encountered by MDS systems (Stokes,
  Newman, 2005)
• Therefore, current RTE Challenge summarisation
  data does not correspond directly to the task
  Organizers view

3
What Next?

• The RTE Challenge has two options
• Eliminate summarisation task data as it isnt
  representative
• OR
• Broaden challenge scope with instances of partial
  information overlap
• Motivation to do the latter
• Address summarization needs more completely
• and other applications too (QA, IR)
• Data can be easily obtained from online automatic
  MDS systems

4
Building an Informational Equivalence Dataset

• Informational Equivalence Two sentences are
  informationally equivalent if they share at least
  one information unit.
• An Information Unit is defined as a unit of text
  that contains at least one subject-verb
  relationship.
• Some sample data (50 positive 50 negative
  sentence pairs)
• http//www.cs.mu.oz.au/nstokes/TE/MDS_corpus_
  1.0.xml

5
Positive Informational Equivalence

• TEXT A Some 70,000 passengers have been
  prevented from travelling after hundreds of
  British Airways staff took unofficial action in
  support of 600 sacked catering workers.
• TEXT B British Airways said Friday that all of
  its striking Heathrow Airport workers were
  returning to work after a 24-hour walkout that
  saw all flights cancelled at the airport and
  about 70,000 travellers stranded.

6
Common Information Unit (Auxiliary)

• Annotator 1 70000 passengers were stranded
  after a strike by British Airways staff
• Annotator 2 70,000 British Airways passengers
  stranded after 24-hour staff walkout.
• Annotator 3 70,000 passengers were stranded
  after British Airway staff took strike action
• Our proposal mark entailed entailing
  information units instead

7
Positive Informational Equivalence

• H Some 70,000 passengers have been prevented
  from travelling after hundreds of British Airways
  staff took unofficial action in support of 600
  sacked catering workers.
• T British Airways said Friday that all of its
  striking Heathrow Airport workers were returning
  to work after a 24-hour walkout that saw all
  flights cancelled at the airport and about 70,000
  travellers stranded.

8
Proposal for next years SUM task

• RTE data Collect examples from output of
  summarisation systems
• RTE task Keep next years task a binary
  classification task. But we will also investigate
  the possibility of an alignment style evaluation
  for future challenges. Try partial entailment
  annotation, following Pyramid style

9
More Information

• Paper N. Stokes and E. Newman, (2005).
  Multi-document Summarisation and the PASCAL
  Textual Entailment Challenge. Proceedings of the
  Australasian Language Technology Workshop 2005
  (ALTW 2005), Australasian Language Technology
  Association. pp. 215-223.
• Sample MDS dataset for next years challenge
• http//www.cs.mu.oz.au/nstokes/TE/MDS_corpus_1.0.
  xml
• WWW http//www.cs.mu.oz.au/nstokes/
• Email nstokes_at_cs.mu.oz.au

10
Negative Informational Equivalence

• Text A Discovery was loaded with nearly 7,000
  pounds of garbage that had accumulated in the
  space station since it was last visited by a
  shuttle in December 2002.
• Text B The Discovery crew spent nine of their
  first 13 days in orbit transferring supplies to
  the space station.