Bayesian Models

1
Bayesian Models
2
Agenda

• Project
• WebCT
• Late HW
• Math
• Independence
• Conditional Probability
• Bayes Formula Theorem
• Steyvers, et al 2003

3
Independence

• Two events A and B are independent if the
  occurrence of A has no influence on the
  probability of the occurrence of B.
• Independent It doesnt matter who is elected
  president, the world will still be a mess.
• Not independent If candidate B is elected
  president, the probability that the world will be
  a mess is 99. If candidate A is elected, the
  probability that the world will be a mess will be
  lowered to 98.

4
Independence

• A and B are independent if P(A?B) P(A) x P(B).
• Independent
• Pick a card from a deck.
• A The card is an ace, P(A) 1/13.
• B The card is a spade, P(B) 1/4
• P(A?B) 1/13 x 1/4 1/52.
• Not independent
• Draw two cards from a deck without replacement.
• A The first card is a space, P(A) 1/4
• B The second card is a spade, P(B) 1/4
• P(A?B) (13 x 12) / (52 x 51) lt 1/4 x 1/4.

5
Conditional Probability

• Example
• What is the probability that a husband will vote
  Democrat given that his wife does?
• P(HusbandDemocratWifeDemocrat)
• This is different from
• What is the probability that a husband will vote
  democrat?
• What is the probability that a hustband and wife
  will vote democrat?

6
Conditional Probability

• P(BA) is the conditional probability that B will
  occur given that A has occurred P(BA) P(B?A)
  / P(A).

All possible events
P(A) A occurs
P(B) B occurs
P(B?A) A and B occur
7
Conditional Probability

• Suppose we roll two dice
• A The sum is 8
• A (2,6), (3,5), (4,4), (5,3), (6,2)
• P(A) 5/36
• B The first die is 3
• B (3,1), (3,2), (3,3), (3,4), (3,5), (3,6)
• P(B) 6/36
• A?B (3,5)
• P(BA) (1/36)/(5/36) 1/5.

8
Conditional Probability
36
5
6
1
P(A) 5/36 P(B) 6/36 P(B?A) 1/36 P(BA)
(1/36)/(5/36) 1/5
P(BA) P(B?A)/P(A)
9
Bayes Formula
Suppose families have 1, 2, or 3 children with
1/3 probability each. Bobby has no brothers.
What is the probability he is an only child?
10
Bayes Formula
Suppose families have 1, 2, or 3 children with
1/3 probability each. Bobby has no brothers.
What is the probability he is an only child?
Let child1, child2, and child3 be the events
that A family has 1, 2, or 3 children,
respectively. Let boy1 be the event that a
family has only 1 boy. Want to compute
P(child1boy1) P(child1?boy1)/P(boy1)
11
Bayes Formula
Suppose families have 1, 2, or 3 children with
1/3 probability each. Bobby has no brothers.
What is the probability he is an only child?
Want to compute P(child1boy1)
P(child1?boy1)/P(boy1) We need to compute
P(child1?boy1) and P(boy1)
12
Bayes Formula
We need to compute P(child1?boy1) and
P(boy1) Because P(BC) P(C?B) / P(C), we can
write P(C?B) P(C) P(BC). So,
P(child1?boy1) P(child1)P(boy1 child1) 1/3
x 1/2 1/6.
13
Bayes Formula
We need to compute P(child1?boy1) and
P(boy1) P(boy1) P(child1?boy1)
P(child2?boy1) P(child3?boy1) We know
P(child1?boy1) 1/6. Likewise, P(child2?boy1)
1/6 P(child3?boy1) 1/8 P(boy1) 1/6 1/6
1/8
14
Bayes Formula
Suppose families have 1, 2, or 3 children with
1/3 probability each. Bobby has no brothers.
What is the probability he is an only child?
P(child1boy1) P(child1?boy1)/P(boy1)
(1/6) / (1/6 1/6 1/8) 4/11
15
Bayes Formula
Suppose families have 1, 2, or 3 children with
1/3 probability each. Bobby has no brothers.
What is the probability he is an only child?
1 Child 120
2 Children 120
3 Children 120
1 boy 60
1 boy 60
1 boy 45
16
Bayes Formula
1 Child 120
2 Children 120
3 Children 120
1 boy 60
1 boy 60
1 boy 45
P(child1boy1) P(child1?boy1)/P(boy1)
(60/360) / ((606045)/360) 60/165 4/11
17
Bayes Formula
Event1
Event2
Eventn

Sub- event
Sub- event
Sub- event
Known P(Eventi), ?P(Eventi) 1, and
P(SubeventEventi) Compute P(Event1Subevent)
18
Bayes Formula
Event1
Event2
Eventn

Sub- event
Sub- event
Sub- event
P(Event1Subevent) P(Event1?Subevent) /
P(Subevent) P(Eventi?Subevent)
P(Eventi)P(SubeventEventi) P(Subevent) ?
P(Eventi?Subevent) ? P(Eventi)P(SubeventEve
nti)
19
Bayes Formula
Event1
Event2
Eventn

Sub- event
Sub- event
Sub- event
P(Event1)P(SubeventEvent1) P(Eve
nt1Subevent) ----------------------------------
------- ? P(Eventi)P(SubeventEventi)
20
Bayes Formula
1 of the population has a disease. 99 of the
people who have the disease have the
symptoms. 10 who dont have the disease have the
symptoms. Let D A person has the
disease. Let S A person has the
symptoms. P(D) .01 and so P(D) .99 P(SD)
.99 and P(SD) .10 What is P(DS)?
21
Bayes Formula
P(D) P(SD) P(DS)
--------------------------------------------
P(D) P(SD) P(D)
P(SD) (.01 x .99) / (.01 x .99 .99 x
.10) .091
22
Bayes Formula
P(Event1)P(SubeventEvent1) P(Eve
nt1Subevent) ----------------------------------
------- ? P(Eventi)P(SubeventEventi)
If there are a very large portion of events, the
denominator may be very hard to calculate. If,
however, you are only interested in relative
probabilities
23
Bayes Formula
P(Event1)P(SubeventEvent1) P(Eve
nt1Subevent) ----------------------------------
------- ? P(Eventi)P(SubeventEventi)
? P(Event1Subevent)
P(Event1)P(SubeventEvent1) ----------------------
----- ----------------------------------------
P(Event2Subevent) P(Event2)P(SubeventEvent2
) This is called the odds.
24
Bayes Formula

• Lets say you have 2 hypotheses (or models), H1
  and H2, under consideration.
• The log odds of these two hypotheses given
  experimental data, D, is

25
Bayes Formula
Posterior odds Relative belief in 2 hypotheses
given the data. Quantity of interest.
26
Bayes Formula
Prior odds Relative belief in 2 hypotheses
Before observing any data. Often assumed to be
1 (0 in log odds).
27
Bayes Formula
Likelihoods Relative probability of the
data, given the two hypotheses. Usually computed
from your models