Learning WithinSentence Semantic Coherence

1
Learning Within-Sentence Semantic Coherence

• Elena Eneva
• Rose Hoberman
• Lucian Lita
• Carnegie Mellon University

2
Semantic (in)Coherence

• Trigram content words unrelated
• Effect on speech recognition
• Actual Utterance THE BIRD FLU HAS AFFECTED
  CHICKENS FOR YEARS BUT ONLY RECENTLY BEGAN MAKING
  HUMANS SICK
• Top Hypothesis THE BIRD FLU HAS AFFECTED
  SECONDS FOR YEARS BUT ONLY RECENTLY BEGAN MAKING
  HUMAN SAID
• Our goal model semantic coherence

3
A Whole Sentence Exponential Model Rosenfeld
1997
def

• P0(s) is an arbitrary initial model (typically
  N-gram)
• fi(s)s are arbitrary computable properties of s
  (aka features)
• Z is a universal normalizing constant

4
A Methodology for Feature Induction

• Given corpus T of training sentences
• Train best-possible baseline model, P0(s)
• Use P0(s) to generate corpus T0 of pseudo
  sentences
• Pose a challenge find (computable) differences
  that allow discrimination between T and T0
• Encode the differences as features fi(s)
• Train a new model

5
Discrimination Task
Are these content words generated from a trigram
or a natural sentence?

• - - - feel - - sacrifice - - sense - - - - - - -
  - -meant - - - - - - - - trust - - - - truth
• - - kind - free trade agreements - - - living - -
  ziplock bag - - - - - - university japan's daiwa
  bank stocks step

6
Building on Prior Work

• Define content words (all but top 50)
• Goal model distribution of content words in
  sentence
• Simplify model pairwise co-occurrences (content
  word pairs)
• Collect contingency tables calculate measure of
  association for them

7
Q Correlation Measure
Derived from Co-occurrence Contingency Table

• Q values range from 1 to 1

8
Density Estimates

• We hypothesized
• Trigram sentences wordpair correlation
  completely determined by distance
• Natural sentences wordpair correlation
  independent of distance
• kernel density estimation
• distribution of Q values in each corpus
• at varying distances

9
Q Distributions
---- Trigram Generated Broadcast News
Density
Q Value
10
Likelihood Ratio Feature
she is a country singer searching for fame and
fortune in nashville Q(country,nashville) 0.76
Distance 8 Pr (Q0.76d8,BNews) 0.32
Pr(Q0.76d8,Trigram) 0.11 Likelihood ratio
0.32/0.11 2.9
11
Simpler Features

• Q Value based
• Mean, median, min, max of Q values for content
  word pairs in the sentence (Cai et al 2000)
• Percentage of Q values above a threshold
• High/low correlations across large/small
  distances
• Other
• Word and phrase repetition
• Percentage of stop words
• Longest sequence of consecutive stop/content
  words

12
Datasets

• LM and contingency tables (Q values) derived from
  103 million words of BN
• From remainder of BN corpus and sentences sampled
  from trigram LM
• Q value distributions estimated from 100,000
  sentences
• Decision tree trained and test on 60,000
  sentences
• Disregarded sentences with
• Mike Stevens says its not real
• Weve been hearing about it

13
Experiments

• Learners
• C5.0 decision tree
• Boosting decision stumps with Adaboost.MH
• Methodology
• 5-fold cross validation on 60,000 sentences
• Boosting for 300 rounds

14
Results
15
Shannon-Style Experiment

• 50 sentences
• ½ real and ½ trigram-generated
• Stopwords replaced by dashes
• 30 participants
• Average accuracy of 73.77 6
• Best individual accuracy 84
• Our classifier
• Accuracy of 78.9 0.42

16
Summary

• Introduced a set of statistical features which
  capture aspects of semantic coherence
• Trained a decision tree to classify with accuracy
  of 80
• Next step incorporate features into exponential
  LM

17
Future Work

• Combat data sparsity
• Confidence intervals
• Different correlation statistic
• Stemming or clustering vocabulary
• Evaluate derived features
• Incorporate into an exponential language model
• Evaluate the model on a practical application

18
Agreement among Participants
19
Expected Perplexity Reduction

• Semantic coherence feature
• 78 of broadcast news sentences
• 18 of trigram-generated sentences
• Kullback-Leibler divergence .814
• Average perplexity reduction per word .0419
  (2.814/21) per sentence?
• Features modify probability of entire sentence
• Effect of feature on per-word probability is small

20
Distribution of Likelihood Ratio
Density
Likelihood Value
21
Discrimination Task

• Natural Sentence
• but it doesn't feel like a sacrifice in a sense
  that you're really saying this is you know i'm
  meant to do things the right way and you trust it
  and tell the truth
• Trigram-Generated
• they just kind of free trade agreements which
  have been living in a ziplock bag that you say
  that i see university japan's daiwa bank stocks
  step though

22
Q Values at Distance 1
---- Trigram Generated Broadcast News
Density
Q Value
23
Q Values at Distance 3
Density
Q Value
24
Outline

• The problem of semantic (in)coherence
• Incorporating this into the whole-sentence
  exponential LM
• Finding better features for this model using
  machine learning
• Semantic coherence features
• Experiments and results