Using Probabilistic Information

1
Using Probabilistic Information

• Read J M Chapter 5, pages 141 - 156

2
Why Do We Need Probabilities?(or Why Arent We
Sure?)

• Noisy channel

I haf to go. geneology

• Sentences are flat. Knowledge is structured.

Joe hit the ball with the bat.

• It would be too inefficient to have to say
  everything.

He bought it.

• Our programs still dont know as much as people
  do.

3
Conditional Probability
Definition P(A B) P(A ? B)
P(B)
Intuition
B A
A B A B
A B
A ?A
B ?B
4
Using Conditional Probability for Recognition
Our task find the object (word, structure, or
whatever) that is most likely given our
observation.
w? argmax P(wO) argmax P(w ? O)
w ? V w ? V P(O)
Example P(wordhave soundhaf)
P(wordhave ? soundhaf)

P(sound haf)

• But what do we actually know
• P(sound haf word have)
• P(word have)

5
Bayes Theorem
P(A B) P(A ? B)

P(B)
P(A ? B)
? P(A) P(B) P(A)
P(A ? B)
? P(A) P(A) P(B)
P(B A)
? P(A) P(B)
6
Using Bayes Theorem
P(A B) P(B A) ? P(A)
P(B)
Example P(wordhave soundhaf)
P(soundhaf wordhave) ?
P(wordhave)
P(sound haf)
But, if we are comparing candidate
interpretations for haf, we can ignore the
denominator since they are all the same.
7
Spelling Correction Choices
Common assumption just one mistake (covers about
80 of nonword errors). Four kinds of mistakes
insertion, deletion, transposition,
substitution. Example
8
Spelling Correction Priors
c? argmax P(ct) argmax P(tc) P(c)
c ? C c ? C
Example observed word acress
Note P(c)s include adding .5 for smoothing.
9
Spelling Correction Conditional Probs
c? argmax P(ct) argmax P(tc) P(c)
w ? V w ? V
Example What is P(deleting t following
c)? Answer We need to collect data from a
training set and encode them in some useful way
confusion matrices contain counts
delx,y, insx,y, subx,y, transx,y From
these counts, we can compute probabilities Deleti
on P(tc) (which involves deleting the ith
character, which happens to be x, where the i-1st
character is y delci-1,ci / countci-1ci
10
Spelling Correction All Together
Typed word acress Intended word ?
11
Spelling Correction the Britany Example
P(word britney O britne)
P(Obritne wordBritney) ?
P(wordbritney)
P(O britne)
The data below shows some of the misspellings
detected by our spelling correction system for
the query britney spears , and the count of
how many different users spelled her name that
way. Each of these variations was entered by at
least two different unique users within a three
month period, and was corrected to britney
spears by our spelling correction system (data
for the correctly spelled query is shown for
comparison).
From http//www.google.com/jobs/britney.html
12
488941 britney spears 40134 brittany
spears 36315 brittney spears 24342 britany
spears  7331 britny spears  6633 briteny
spears  2696 britteny spears  1807 briney
spears  1635 brittny spears  1479 brintey
spears  1479 britanny spears  1338 britiny
spears  1211 britnet spears  1096 britiney
spears   991 britaney spears   991 britnay
spears
811 brithney spears   811 brtiney
spears   664 birtney spears   664 brintney
spears   664 briteney spears   601 bitney
spears   601 brinty spears   544 brittaney
spears   544 brittnay spears   364 britey
spears   364 brittiny spears   329 brtney
spears   269 bretney spears   269 britneys
spears   244 britne spears   244 brytney
spears  
13
  220 breatney spears   220 britiany
spears   199 britnney spears   163 britnry
spears   147 breatny spears   147 brittiney
spears   147 britty spears   147 brotney
spears   147 brutney spears   133 britteney
spears   133 briyney spears   121 bittany
spears   121 bridney spears   121 britainy
spears   121 britmey spears  
  109 brietney spears   109 brithny
spears   109 britni spears   109 brittant
spears    98 bittney spears    98 brithey
spears    98 brittiany spears    98 btitney
spears    89 brietny spears    89 brinety
spears    89 brintny spears    89 britnie
spears    89 brittey spears    89 brittnet
spears    89 brity spears    89 ritney spears
14
80 bretny spears    80 britnany
spears    73 brinteny spears    73 brittainy
spears    73 pritney spears    66 brintany
spears    66 britnery spears    59 briitney
spears    59 britinay spears    54 britneay
spears    54 britner spears    54 britney's
spears    54 britnye spears    54 britt
spears    54 brttany spears    
48 bitany spears    48 briny spears    48
brirney spears    48 britant spears    48
britnety spears    48 brittanny spears    48
brttney spears    44 birttany spears    44
brittani spears    44 brityney spears    44
brtitney spears    39 brienty spears    39
brritney spears    36 bbritney spears    36
briitany spears
15
  36 britanney spears    36 briterny
spears    36 britneey spears    36 britnei
spears    36 britniy spears    32 britbey
spears    32 britneu spears
   2 brtittny spears     2 brttiny
spears     2 brtttany spears     2 brydney
spears     2 brynty spears     2 brythey
spears     2 bryttney spears     2 btiany
spears     2 btirtney spears     2 btitiney
spears     2 btittny spears     2 btritany
spears     2 buttney spears     2 grittney
spears     2 prietny spears     2 pritany
spears     2 prittany spears
16
Other Examples of Bayes Theorem - Glasses

• We observe Joe wearing glasses. We want to
  decide whether it is more likely that Joe is a
  salesman or a librarian. Here are the facts (L
  means librarian, S means salesman, G means
  glasses)
• P(G) .1
• P(L) .0001
• P(S) .01
• P(GL) 1
• P(GS) .05
• P(LG) P(GL) P(L) / P(G)
• 1 0.0001/.1
• 0.001
• P(SG) P(GS) P(S) / P(G)
• .05 0.01/.1
• 0.005

17
Other Examples of Bayes Theorem - Drugs

• We want to compute the probability that Joe uses
  heroin given that he tests positive for it. Here
  are the facts (H means heroin use, E means a
  positive test for heroin)
• Sensitivity P(EH) 0.95
• Specificity 1 - P(EH) 0.90
• Baseline "prior" probability P(H) 0.03.
• P(HE) P(EH) P(H)
• P(E)
• 0.95 0.03/0.030.95
  0.970.1
• 0.1255.