Statistical Natural Language Processing

1
Statistical Natural Language Processing
2
What is NLP?

• Natural Language Processing (NLP), or
  Computational Linguistics, is concerned with
  theoretical and practical issues in the design
  and implementation of computer systems for
  processing human languages
• It is an interdisciplinary field which draws on
  other areas of study such as computer science,
  artificial intelligence, linguistics and logic

3
Applications of NLP

• natural language interfaces to databases
• programs for classifying and retrieving documents
  by content
• explanation generation for expert systems
• machine translation
• advanced word-processing tools

4
What makes NLP a computational challenge?

• Ambiguous nature of Natural Language.
• There are varied applications for language
  technology
• Knowledge representation is a difficult task.
• There are different levels of information encoded
  in our language

5
What is statistical NLP?

• Statistical NLP aims to perform statistical
  inference for the field of NLP
• Statistical inference consists of taking some
  data generated in accordance with some unknown
  probability distribution and making inferences.

6
Motivations for Statistical NLP

• Cognitive modeling of the human language
  processing has not reached a stage where we can
  have a complete mapping between the language
  signal and the information contents.
• Complete mapping is not always required.
• Statistical approach provides the flexibility
  required for making the modeling of a language
  more accurate.

7
Idea behind Statistical NLP

• View language processing as a noisy channel
  information transmission.
• The approach requires a model that characterizes
  the transmission by giving for every message the
  probability of the observed output

8
Statistical Modeling and Classification

• Primitive acoustic features
• Quantization
• Maximum likelihood and related rules
• Class conditional density function
• Hidden Markov Model Methodology

9
Details.

• Primitive acoustic features are used to estimate
  the speech spectrum on the basis of its
  statistical properties.
• By means of quantization a typical speech signal
  can be represented as a sequence of symbols and
  can be mapped using statistical decision rules
  into a multidimensional acoustic feature space,
  thus classifying the signal.

10
Maximum Likelihood

• Although there is no direct method for computing
  the probability of a phonetic unit given its
  acoustic features,we can use Bayes rule to
  estimate the probability of a phonetic class
  given its features from the likelihood of the
  features given the class. This method leads to
  the maximum likelihood classifier which assigns
  an unknown vector to that class whose probability
  density function conditioned on the class has the
  maximum value.
• Another variant of the maximum likelihood
  methodology is clustering.

11
Hidden Markov Models

• A Hidden Markov Model, is a set of states
  (lexical categories in our case) with directed
  edges labeled with transition probabilities that
  indicate the probability of moving to the state
  at the end of the directed edge, given that one
  is now in the state at the start of the edge. The
  states are also labeled with a function which
  indicates the probabilities of outputting
  different symbols if in that state (while in a
  state, one outputs a single symbol before moving
  to the next state). In our case, the symbol
  output from a state/lexical category is a word
  belonging to that lexical category.

12
Hidden Markov Models (cont.)



13
Conditional Class Density Function

• All statistical methods of speech recognition
  depend on the class conditional density function.
• These, in turn, depend on the existence of a
  sufficiently large, correctly labeled training
  set and well understood statistical estimation
  techniques

14
How does statistics help

• Disambiguation may be achieved by using
  stochastic context free grammars
• It helps in providing degrees of grammaticality
• Naturalness
• Structural preference
• Error Tolerance

15
Example using stochastic CFG

• for example consider the sentence
• John Walks
• The grammar is as follows
• 1 S -gt NP V 0.7
• 2 S -gt NP 0.3
• 3 NP -gt N 0.8
• 4 NP -gt N N 0.2
• 5 N -gt John 0.6
• 6 N -gt Walks 0.4
• 7 V -gt Walks 1.0
• The numbers on the right represent the weights
  for each rule.The weight of the analysis is the
  product of the weights of the rules used in the
  derivation.
• Predicting the right sentence that is perceived
  is based on these weights.

16
Degrees of grammaticality

• Traditional approaches to NLP do not accommodate
  gradations of grammaticality. A sentence is
  either correct or not.
• In some cases acceptability may vary with the
  structure and context of the sentence.

17
Structural Preference

• Consider the sentence
• The emergency crews hate most is domestic
  violence.
• The correct interpretation is
• The emergency that the crews hate most is
  domestic violence.
• These preferences can be seen more as structural
  preferences rather than parsing preferences.
• Statistical approaches can easily handle such
  structural preferences.

18
Error Tolerance

• A remarkable property of human language
  comprehension is error tolerance.
• Many sentences that the traditional approach
  classifies as ungrammatical can actually be
  interpreted by statistical NLP techniques.

19
Conclusions

• Free and commercial software is now available
  that provides a lot of NLP features. (e.g.
  Microsoft XP has a speech recognition software by
  which users can control menus and execute
  commands)
• A lot of research is going into developing new
  applications and investigating new techniques and
  approaches that will make Statistical NLP more
  feasible in the near future.