Probabilistic Parsing II

1
Probabilistic Parsing II

• (many slides adapted from slides by
• Michael Collins)

TexPoint fonts used in EMF. Read the TexPoint
manual before you delete this box. AAAAA
2
A REVIEW OF WHERE WEVE BEEN RECENTLY.
3
How well do PCFGs work?

• Not very well
• a PCFG adequate to parse over 90 of the MIT
  Voyager Corpus was successful in picking the
  correct parse on only 35 of a reserved test set.
• Sample Sentences -- The MIT Voyager Corpus
• I'm currently at MIT
• What kind of food does LaGroceria serve
• Where is the closest library to MIT
• What's the closest ice cream parlor to Harvard
  University
• Is there a subway stop by the Mount Auburn
  Hospital
• Can you show me the intersection of Cambridge
  Street and Hampshire Street
• Which subway stop is closest to the library at
  forty five Pearl Street

4
Voyager Experiments Results

• Results parsing reserved Voyager corpus. Ref
  Magerman Marcus 1991

5
PP Attachment V-NP-PP forced choice
S
VP
PP
NP
NP
V
P

• He joined the board as a nonexecutive
  director
• Quintuple (V-attach, vjoined, n1board, pas,
  n2director)
• Training set 20,801 quintuples (V- or N-attach,
  v, n1, p, n2)
• Test set 3097 quintuples
• Development set 4059 quintuples

6
Core Statistical Approach

• Estimate
• If
• Noun-attach
• Else
• Verb-attach
• Estimation using Maximum Likelihood Estimate

7
The final algorithm w/ backoff
8
Q How to make PCFGs sensitive to -Lexical
relationships -Larger Contexts
9
Lexical relationships paths in the parse tree
10
A lexicalized tree Lexical relationships are
now coded locally in the tree itself
11
The SPATTER Parser (Magerman 95)
12
A Lexicalized PCFG
13
Factoring rule expansion Charniak 97
14
Smoothed Estimation
15
Smoothed Estimation II
16
Smoothed Estimation III
17
Independence Assumptions
18
Head Probabilities
19
Rule Probabilities
20
Estimating Head Probabilities
21
Old Generative Parsing Results (lt40 w)
22
Discriminative Parsing

• Adapted from slides by Chris Manning (Seven
  lectures on Statistical Parsing)
• Ryan Gabbard (CIS 391, Penn)

23
Universal Machine Learning Diagram
24
Generative v. Discriminative Models

• Generative question How can we model the joint
  distribution of the classes and the features?
• Why waste energy on stuff we dont care about?
  Lets optimize the job were trying to do
  directly!
• Discriminative question What features
  distinguish the classes from one another?

25
Example
Modeling what sort of bizarre distribution
produced these training points is hard, but
distinguishing the classes is a piece of cake!
chart from MIT tech report 507, Tony Jebara
26
Formally The Classification Problem

• Given a training set of iid samples T(X1,Y1)
  (Xn,Yn) of input and class variables from an
  unknown distribution D(X,Y), estimate a function
  that predicts the class from the input
  variables
• Goal a hypothesis with minimum expected
  loss
• Under 0-1 loss the hypothesis with minimum
  expected loss is the Bayes optimal classifier

27
Discriminative Parsing as a classification problem

• The observed Xs are the sentences.
• The class Y of a sentence is its parse tree
• The model has a large (infinite!) space of
  variables, but we can still assign them
  probabilities
• The way we can do this is by breaking whole parse
  trees into component parts

28
Approaches to Solving Classification Problems

• Generative. Try to estimate the probability
  distribution of the data D(X,Y)
• specify a parametric model family
• choose parameters by maximum likelihood on
  training data
• estimate conditional probabilities by Bayes rule
• You use the generative model backwards
• classify new instances to the most probable class
  Y according to

29
Approaches to Solving Classification Problems

• 2. Discriminative. Try to estimate the
  conditional distribution D(YX) from data.
• specify a parametric model family
• estimate parameters by maximum conditional
  likelihood of training data
• classify new instances to the most probable class
  Y according to
• 3. Discriminative. Distribution-free. Try to
  estimate directly from data so that
  its expected loss will be minimized.

30
Motivating discriminative estimation (1)
100 6
2
A training corpus of 108 (imperative) sentences.
Follows an example by Mark Johnson
31
Motivating discriminative parsing (2)

• In discriminative models, it is easy to
  incorporate different kinds of features
• Often just about anything that seems
  linguistically interesting
• In generative models, its often difficult, and
  the model suffers because of false independence
  assumptions
• This ability to add informative features is the
  real power of discriminative models for NLP.

32
Discriminative Parsers

• Discriminative Dependency Parsing
• Not as computationally hard (tiny grammar
  constant)
• Explored considerably recently. E.g. McDonald et
  al. 2005
• Make parser action decisions discriminatively
• E.g. with a shift-reduce parser
• Dynamic program Phrase Structure Parsing
• Resource intensive! Most work on sentences of
  length lt15
• The need to be able to dynamic program limits the
  feature types you can use
• Post-Processing Parse reranking
• Just work with output of k-best generative parser

• Distribution-free methods
• Probabilistic model methods

33
Discriminative models

• Shift-reduce parser Ratnaparkhi (98)
• Learns a distribution P(TS) of parse trees given
  sentences using the sequence of actions of a
  shift-reduce parser
• Uses a maximum entropy model to learn conditional
  distribution of parse action given history
• Suffers from independence assumptions that
  actions are independent of future observations
• Higher parameter estimation cost to learn local
  maximum entropy models
• Lower but still good accuracy 86 - 87 labeled
  precision/recall

34
Discriminative dynamic-programmed parsers

• Taskar et al. (2004 EMNLP) show how to do joint
  discriminative SVM-style (max margin) parsing
  building a phrase structure tree also conditioned
  on words in O(n3) time
• In practice, totally impractically slow. Results
  were never demonstrated on sentences longer than
  15 words
• Turian et al. (2006 NIPS) do a decision-tree
  based discriminative parser
• Research continues.

35
Discriminative Models Distribution Free
Re-ranking (Collins 2000)

• Represent sentence-parse tree pairs by a feature
  vector F(X,Y)
• Learn a linear ranking model with parameters
  using the boosting loss

13 error reduction
Still very close in accuracy to generative model
Charniak 2000
36
Charniak and Johnson (2005 ACL)Coarse-to-fine
n-best parsing and MaxEnt discriminative reranking

• Builds a maxent discriminative reranker over
  parses produced by (a slightly bugfixed and
  improved version of) Charniak (2000).
• Gets 50 best parses from Charniak (2000) parser
• Doing this exploits the coarse-to-fine idea to
  heuristically find good candidates
• Maxent model for reranking uses heads, etc. as
  generative model, but also nice linguistic
  features
• Conjunct parallelism
• Right branching preference
• Heaviness (length) of constituents factored in
• Gets 91 LP/LR F1 (on all sentences! up to 80
  wd)

37
The Latest Parsing Results