Bayesian Models

Agenda Project WebCT Late HW Math –Independence –Conditional Probability –Bayes Formula & Theorem Steyvers, et al 2003

Independence Two events A and B are independent if the occurrence of A has no influence on the probability of the occurrence of B. –Independent: “It doesn’t matter who is elected president, the world will still be a mess.” –Not independent: “If candidate B is elected president, the probability that the world will be a mess is 99%. If candidate A is elected, the probability that the world will be a mess will be lowered to 98%.”

Independence A and B are independent if P(A  B) = P(A) x P(B). – Independent: Pick a card from a deck. A = “The card is an ace”, P(A) = 1/13. B = “The card is a spade”, P(B) = 1/4 P(A  B) = 1/13 x 1/4 = 1/52. –Not independent: Draw two cards from a deck without replacement. A = “The first card is a space”, P(A) = 1/4 B = “The second card is a spade”, P(B) = 1/4 P(A  B) = (13 x 12) / (52 x 51) < 1/4 x 1/4.

Conditional Probability Example: –What is the probability that a husband will vote Democrat given that his wife does? – P(Husband Democrat |Wife Democrat ) –This is different from: What is the probability that a husband will vote democrat? What is the probability that a hustband and wife will vote democrat?

All possible events Conditional Probability P(B|A) is the conditional probability that B will occur given that A has occurred: P(B|A) = P(B  A) / P(A). P(A) = A occurs P(B) = B occurs P(B  A) = A and B occur

Conditional Probability Suppose we roll two dice –A = “The sum is 8” A = {(2,6), (3,5), (4,4), (5,3), (6,2)} P(A) = 5/36 –B = “The first die is 3” B = {(3,1), (3,2), (3,3), (3,4), (3,5), (3,6)} P(B) = 6/36 –A  B = {(3,5)} –P(B|A) = (1/36)/(5/36) = 1/5.

Conditional Probability P(A) = 5/36 P(B) = 6/36 P(B  A) = 1/36 P(B|A) = (1/36)/(5/36) = 1/5 P(B|A) = P(B  A)/P(A)

Bayes Formula Suppose families have 1, 2, or 3 children with 1/3 probability each. Bobby has no brothers. What is the probability he is an only child?

Bayes Formula Suppose families have 1, 2, or 3 children with 1/3 probability each. Bobby has no brothers. What is the probability he is an only child? Let child 1, child 2, and child 3 be the events that A family has 1, 2, or 3 children, respectively. Let boy 1 be the event that a family has only 1 boy. Want to compute P(child 1 |boy 1 ) = P(child 1  boy 1 )/P(boy 1 )

Bayes Formula Suppose families have 1, 2, or 3 children with 1/3 probability each. Bobby has no brothers. What is the probability he is an only child? Want to compute P(child 1 |boy 1 ) = P(child 1  boy 1 )/P(boy 1 ) We need to compute P(child 1  boy 1 ) and P(boy 1 )

Bayes Formula We need to compute P(child 1  boy 1 ) and P(boy 1 ) Because P(B|C) = P(C  B) / P(C), we can write: P(C  B) = P(C) P(B|C). So, P(child 1  boy 1 ) = P(child 1 )P(boy 1 | child 1 ) = 1/3 x 1/2 = 1/6.

Bayes Formula We need to compute P(child 1  boy 1 ) and P(boy 1 ) P(boy 1 ) = P(child 1  boy 1 ) + P(child 2  boy 1 ) + P(child 3  boy 1 ) We know P(child 1  boy 1 ) = 1/6. Likewise, P(child 2  boy 1 ) = 1/6 P(child 3  boy 1 ) = 1/8 P(boy 1 ) = 1/6 + 1/6 + 1/8

Bayes Formula Suppose families have 1, 2, or 3 children with 1/3 probability each. Bobby has no brothers. What is the probability he is an only child? P(child 1 |boy 1 ) = P(child 1  boy 1 )/P(boy 1 ) = (1/6) / (1/6 + 1/6 + 1/8) = 4/11

Bayes Formula 1 Child Children Children 120 Suppose families have 1, 2, or 3 children with 1/3 probability each. Bobby has no brothers. What is the probability he is an only child? 1 boy 60 1 boy 60 1 boy 45

Bayes Formula 1 Child Children Children boy 60 1 boy 60 1 boy 45 P(child 1 |boy 1 ) = P(child 1  boy 1 )/P(boy 1 ) = (60/360) / (( )/360) = 60/165 = 4/11

Bayes Formula Event 1 Event 2 Event n Sub- event Sub- event Sub- event … Known: P(Event i ),  P(Event i ) = 1, and P(Subevent|Event i ) Compute: P(Event 1 |Subevent)

Bayes Formula Event 1 Event 2 Event n Sub- event Sub- event Sub- event … P(Event 1 |Subevent) = P(Event 1  Subevent) / P(Subevent) P(Event i  Subevent) = P(Event i )P(Subevent|Event i ) P(Subevent) =  P(Event i  Subevent) =  P(Event i )P(Subevent|Event i )

Bayes Formula Event 1 Event 2 Event n Sub- event Sub- event Sub- event … P(Event 1 )P(Subevent|Event 1 ) P(Event 1 |Subevent) =  P(Event i )P(Subevent|Event i )

Bayes Formula 1% of the population has a disease. 99% of the people who have the disease have the symptoms. 10% who don’t have the disease have the symptoms. Let D = “A person has the disease”. Let S = “A person has the symptoms”. P(D) =.01 and so P(~D) =.99 P(S|D) =.99 and P(S|~D) =.10 What is P(D|S)?

Bayes Formula P(D) P(S|D) P(D|S) = P(D) P(S|D) + P(~D) P(S|~D) = (.01 x.99) / (.01 x x.10) =.091

Bayes Formula P(Event 1 )P(Subevent|Event 1 ) P(Event 1 |Subevent) =  P(Event i )P(Subevent|Event i ) If there are a very large portion of events, the denominator may be very hard to calculate. If, however, you are only interested in relative probabilities…

Bayes Formula P(Event 1 )P(Subevent|Event 1 ) P(Event 1 |Subevent) =  P(Event i )P(Subevent|Event i )  P(Event 1 |Subevent) P(Event 1 )P(Subevent|Event 1 ) = P(Event 2 |Subevent) P(Event 2 )P(Subevent|Event 2 ) This is called the “odds”.

Bayes Formula Let’s say you have 2 hypotheses (or models), H 1 and H 2, under consideration. The log odds of these two hypotheses given experimental data, D, is:

Bayes Formula Posterior odds = Relative belief in 2 hypotheses given the data. Quantity of interest.

Bayes Formula Prior odds = Relative belief in 2 hypotheses Before observing any data. Often assumed to be 1 (0 in log odds).

Bayes Formula Likelihoods = Relative probability of the data, given the two hypotheses. Usually computed from your models…