Probability theory

1
Probability theory

• LING 570
• Fei Xia
• Week 2 10/01/07

2
Misc.

• Patas account and dropbox
• Course website, Collect it, and GoPost.
• Mailing list
• Received message on Thursday?
• Questions about hw1?

3
Outline

• Quiz 1
• Unix commands
• Linguistics
• Elementary Probability theory MS 2.1

4
Quiz 1

• Five areas weight ave
• Programming 4.0 (3.74)
• Try Perl or Python
• Unix commands 1.2 (0.99)
• Probability 2.0 (1.09)
• Regular expression 2.0 (1.62)
• Linguistics knowledge 0.8 (0.71)

5
Results

• 9.0-10 4
• 8.0-8.9 8
• lt 8.0 8

6
Unix commands

• ls (list), cp (copy), rm (remove)
• more, less, cat
• cd, mkdir, rmdir, pwd
• chmod to change file permission
• tar, gzip to tar/zip files
• ssh, sftp to log on or ftp files
• man to learn a command

7
Unix commands (cont)

• compilers javac, gcc, g, perl,
• ps, top,
• which
• Pipe
• cat input_file eng_tokenizer.sh make_voc.sh gt
  output_file
• sort, unique, awk, grep
• grep the voc awk print 2 sort uniq
  c sort -nr

8
Examples

• Set the permission of foo.pl so it is readable
  and executable by the user and the group.
• rwx rwx rwx gt 101 101 000
• chmod 550 foo.pl
• Move a file, foo.pl, from your home dir to /tmp
• mv /foo.pl /tmp

9
Linguistics POS tags

• Open class Noun, verb, adjective, adverb
• Auxiliary verb/modal can, will, might, ..
• Temporal noun tomorrow
• Adverb adjly, always, still, not,
• Closed class Preposition, conjunction,
  determiner, pron,
• Conjunction CC (and), SC (if, although)
• Complementizer that,

10
Linguistics syntactic structure

• Two kinds
• Phrase structure (a.k.a. parse tree)
• Dependency structure
• Examples
• John said that he would call Mary tomorrow

11
Outline

• Quiz 1
• Unix commands
• Linguistics
• Elementary Probability theory

12
Probability Theory
13
Basic concepts

• Sample space, event, event space
• Random variable and random vector
• Conditional probability, joint probability,
  marginal probability (prior)

14
Sample space, event, event space

• Sample space (O) the set of all possible
  outcomes.
• Ex toss a coin three times
• HHH, HHT, HTH, HTT,
• Event an event is a subset of O.
• Ex an event is
• HHT, HTH, THH
• Event space (2O) the set of all possible events.

15
Probability function

• A probability function (a.k.a. a probability
  distribution) distributes a probability mass of 1
  throughout the sample space ?.
• It is a function from 2? ! 0,1 such that
• P(?) 1
• For any disjoint sets Aj 2 2?, P(? Aj) ?
  P(Aj)
• - Ex P(HHT, HTH, HTT)
• P(HHT) P(HTH) P(HTT)

16
The coin example

• The prob of getting a head is 0.1 for one toss.
  What is the prob of getting two heads out of
  three tosses?
• P(Getting two heads)
• P(HHT, HTH, THH)
• P(HHT) P(HTH) P(THH)
• 0.10.10.9 0.10.90.10.90.10.1
• 30.10.10.9

17
Random variable

• The outcome of an experiment need not be a
  number.
• We often want to represent outcomes as numbers.
• A random variable X is a function O?R.
• Ex the number of heads with three tosses
  X(HHT)2, X(HTH)2, X(HTT)1,

18
The coin example (cont)

• X the number of heads with three tosses
• P(X2)
• P(HHT, HTH, THH)
• P(HHT) P(HTH) P(THH)

19
Two types of random variables

• Discrete X takes on only a countable number of
  possible values.
• Ex Toss a coin three times. X is the number of
  heads that are noted.
• Continuous X takes on an uncountable number of
  possible values.
• Ex X is the speed of a car

20
Common trick 1 Maximum likelihood estimation

• An example toss a coin 3 times, and got two
  heads. What is the probability of getting a head
  with one toss?
• Maximum likelihood (ML)
• ? arg max? P(data ?)
• In the example,
• P(X2) 3 p p (1-p)
• e.g., the prob is 3/8 when p1/2, and is 12/27
  when p2/3
• 3/8 lt 12/27

21
Random vector

• Random vector is a finite-dimensional vector of
  random variables XX1,,Xk.
• P(x) P(x1,x2,,xn)P(X1x1,., Xnxn)
• Ex P(w1, , wn, t1, , tn)

22
Notation

• X, Y, Xi, Yi are random variables.
• x, y, xi are values.
• P(Xx) is written as P(x)
• P(Xx Yy) is written as P(x y).

23
Three types of probability

• Joint prob P(x,y) prob of Xx and Yy happening
  together
• Conditional prob P(x y) prob of Xx given a
  specific value of Yy
• Marginal prob P(x) prob of Xx for all
  possible values of Y.

24
An example

• There are two coins. Choose a coin and then toss
  it. Do that 10 times.
• Coin 1 is chosen 4 times one head and three
  tails.
• Coin 2 is chosen six times four heads and two
  tails.
• Lets calculate the probabilities.

25
Probabilities

• P(C1) 4/10, P(C2) 6/10
• P(Xh) 5/10, P(Xt) 5/10
• P(Xh C1) ¼, P(Xh C2) 4/6
• P(Xt C1) ¾, P(Xt C2) 2/6
• P(Xh, C1) 1/10, P(Xh, C2) 4/10
• P(Xt, C1) 3/10, P(Xt C2) 2/10

26
Relation between different types of probabilities

• P(Xh, C1)
• P(C1) P(Xh C1)
• 4/10 ¼ 1/10
• P(Xh)
• P(Xh, C1) P(Xh, C2)
• 1/10 4/10 5/10

27
Common trick 2Chain rule
28
Common trick 3 joint prob ?Marginal prob
29
Common trick 4Bayes rule
30
Independent random variables

• Two random variables X and Y are independent iff
  the value of X has no influence on the value of Y
  and vice versa.
• P(X,Y) P(X) P(Y)
• P(YX) P(Y)
• P(XY) P(X)
• Our previous examples P(X, C) ! P(X) P(C)

31
Conditional independence

• Once we know C, the value of A does not affect
  the value of B and vice versa.
• P(A,B C) P(AC) P(BC)
• P(AB,C) P(A C)
• P(BA, C) P(B C)

32
Independence and conditional independence

• If A and B are independent, are they conditional
  independent?
• Example
• Burglar, Earthquake
• Alarm

33
Common trick 5Independence assumption
34
An example

• P(w1 w2 wn)
• P(w1) P(w2 w1) P(w3 w1 w2)
• P(wn w1 , wn-1)
• ¼ P(w1) P(w2 w1) . P(wn wn-1)
• Why do we make independence assumption which we
  know are not true?

35
Summary of elementaryprobability theory

• Basic concepts sample space, event space, random
  variable, random vector
• Joint / conditional /marginal probability
• Independence and conditional independence
• Five common tricks
• Max likelihood estimation
• Chain rule
• Calculating marginal probability from joint
  probability
• Bayes rule
• Independence assumption

36
Outline

• Quiz 1
• Unix commands
• Linguistics
• Elementary Probability theory

37
Next time

• JM Chapt 2
• Formal language and formal grammar
• Regular expression
• Hw1 is due at 3pm on Wed.