CSI 5386 Project Presentation : Learning From Low Level Features

1
CSI 5386 Project Presentation Learning From Low
Level Features Pyramid Scoring Methods

• Marcel Wirantono

2
Overview

• Pyramid evaluation of Summary
• Learning from low level features
• Conclusion (Currently)

3
Pyramid Method

• Difference with older methods (unigram
  co-occurrence, Lin Hovy 2002)- more than 1
  model summary- Stability on human written
  summary
• ExampleA1 In 1998 two Libyans indicted in 1991
  for the Lockerbie bombing were still in Libya.B1
  Two Libyans were indicted in 1991 for blowing up
  a Pan Am jumbo jet over Lockerbie, Scotland in
  1988.C1 Two Libyans, accused by the United
  States and Britain of bombing a New York bound
  Pan Am jet over Lockerbie, Scotland in 1988,
  killing 270 people, for 10 years were harbored by
  Libya who claimed the suspects could not get a
  fair trail in America or Britain.D2 Two Libyan
  suspects were indicted in 1991.

4
Pyramid Method

• 2 SCUs obtained- SCU1 (w4) two Libyans were
  officially accused of the Lockerbie bombing
  (A1,B1,C1,D2)- SCU2 (w3) the indictment of the
  two Lockerbie suspects was in 1991(A1,B1,D2)

• Target Concepts
• A summary is evaluated by their total SCUs
  weight
• Goal of a summary is to maximize the number of
  SCUs and maximize the weight of the SCUs included.

5
The Data

• 20 Document sets ( 18000 sentences with 1100
  sentences that contain at least 1 SCU)
• Data are treated at sentence level i.e. 1
  sentence 1 instance
• Due to the nature of the pyramid scoring
  evaluation most SCUs are that of a lower weight.
  (Maximum SCU weight or the highest pyramid level
  is 7 and average SCU weight 2.7)
• An extra target feature is included i.e. the
  number of peer summary that uses the same
  sentence.

6
The Features

• Line number (position in the document)
• Number of characters
• Number of words
• Paragraph number
• The Sequence /position in the paragraph

• Number of connective words like conjunctions i.e.
  afterwards
• Number of words that suggest causality i.e.
  hence, because
• Number of content phrase that are true phrases
  (more than 2 words)
• Number of proper nouns
• Number of content words (words that do not appear
  in the stopwords list)

7
Data Visualization

• Blue represents tabulation of instances that
  contain at least 1 SCU
• Red represents tabulation of instances that
  contain no SCU

8
Data Visualization

• Blue represents tabulation of instances that
  have more than 3 as an average SCU weights
• Red represents tabulation of instances that have
  more than 3 as an average SCU weights(instances
  with no SCUs are ignored)

9
Result ( Naïve Bayes and Decision Tree C.4.5 for
classifying instances with SCU vs. non SCU
instances)
10
Conclusion

• Due to the nature of the attributes and severity
  of the imbalance learning to identify potential
  summary sentences is hard
• Future Works (part of this project) -Finding
  more low level features that better separate
  these instances-Introducing cost to the learning
  problem that will improve recall at the cost of
  precision (treating them as trade
  off)-reformulate the problem by trying to learn
  from sentences with high SCUs- reformulate the
  problem as nominal learning using linear
  regression and produce weights and threshold that
  can separate potential summary sentences with non
  important sentences - evaluating performance by
  cross validation using unused document sets (for
  10 C.V. each time learn with 18 different sets)
• Ideal Conclusion would be to at least produce a
  system or equation or algorithm that can capture
  all the potential sentences and filter the rest
  (tolerance in the number of unimportant sentence
  should be reasonable but not fixed)

11
References

• Aaron Harnly, Ani Nenkova, Rebecca Passonneau,
  Owen RambowAutomation of summary evaluation by
  the pyramid method, RANLP-2005, Borovets,
  Bulgaraia
• Ani Nenkova and Rebecca PassonneauEvaluating
  Content Selection in Summarization the Pyramid
  Method,NAACL-HLT 2004
• Terry Copeck, Stan Szpakowicz, Leveraging
  Pyramids,2005
• Ian H. Witten and Eibe Frank (2005) "Data Mining
  Practical machine learning tools and techniques",
  2nd Edition, Morgan Kaufmann, San Francisco,
  2005.