Foundations of language and speech technology

1
Foundations of language and speech technology

• Course 2004-2005

2
Schedule

• Until January 1 hmm speech recognition
• Attempt to built a simple speech recogniser with
  material from CGN, using the HTK toolkit
• After January 1 advanced text-to-speech
• Attempt to built a system for unrestricted
  text-to-speech synthesis including prosody

3
Planning first weeks

• 25/11 general foundations
• Manning Schütze, H 1 H 2
• 29/11 DTW HMM
• Manning Schütze, H 9
• 2/12 HTK toolkit
• HTK book
• 6/12 HTK toolkit

4
Preliminaries 1

• counting rules
• empiricists rationalists
• performance competence

5
Preliminaries 2

• Learning
• Methods
• Bootstrap
• Automation
• Status of the method
• Reflects human mind
• Because it works

6
Probabilities

• Binary choices?
• categorical perception?
• (un)grammaticality?
• handling unseen data
• Sources of variation (ambiguities)
• Many more in speech than in text
• name examples

7
Data explorations

• Need of large text and speech corpora
• Enriched for bootstrapping systems
• Primitive statistics counts

8
Counts 1

• Zipfs law
• principle of least effort
• Also in human speech production and perception
• Information theory
• frequency x rank constant
• Approximately
• Underlying mechanisms?
• Mandelbrot
• f P (r ?) -?

9
Counts 2

• Power relation
• Very general phenomenon
• Additive
• Results, among others, from binomial
  distributions
• Random choice out of limited number of items
• Most words are rare
• Bad news for frequency-based approaches

10
Assignment

• Frequency distributions of first names
• Available on the website
• Solve
• Constant
• Explain deviations (for low and high ranks)
• Any idea why a power relation may apply in this
  case?

11
Mathematical foundations

• Probability 0,1
• Conditional probability P(AB)
  P(A?B)/P(B)P(A?B) P(AB)P(B)
• Bayes theoremP(BA) P(B?A)/P(A)
  P(AB)P(B)/P(A)

12
pmf

• Probability mass function p(x)
• also probability density function
• random variable X has pmf p(x)
• Expectation E(X) ? x p(x)
• Variance Var(X) E((X-E(X))2)
• Standard deviation sqrt(Var(X))

13
Joint distributions

• p(x,y) P (xX, yY)
• chain rulep(w,x,y,z) p(w)p(wx)p(yw,x)p(zw,x,
  y)

14
Distributions

• Binomial distribution
• Multinomial distribution
• Normal distribution
• Gaussian
• mean, variance

15
Likelihood ratio

• which model best explains a data sequence s
  model ? or model ??P(?s)/P(?s)and expand P
  with Bayes rule

16
Entropy

• Self-information
• How many bits you need to transfer a certain type
  of information
• H(p) H(X) ? p(x) log2 p(x)
  expectation

17
Joint and conditional entropy

• Joint H(X,Y) ? ? p(x,y) log2 p(x,y)
• ConditionalH(XY) ? ? p(x,y) log2 p(xy)
• H(X,Y) H(X) H(YX)

18
Mutual information

• I(XY) H(X) H(XY) H(Y) H(YX)reduction
  in uncertainty of X, given Y,or reversily
• I(XX) H(X)mutual information of X
  self-information

19
Uncertainty

• Entropy the less uncertainty, the lower
• The more a model encaptures data (a language) the
  lower the entropy of the model will be
• Perplexity 2 H(x,m)
• the average number of different words one may
  expect to show after any word, given a languge
  model

20
Perplexity