1. Bayesian Models

2. Agenda Project WebCT Late HW Math –Independence –Conditional Probability –Bayes Formula & Theorem Steyvers, et al 2003

3. Independence Two events A and B are independent if the occurrence of A has no influence on the probability of the occurrence of B. –Independent: “It doesn’t matter who is elected president, the world will still be a mess.” –Not independent: “If candidate B is elected president, the probability that the world will be a mess is 99%. If candidate A is elected, the probability that the world will be a mess will be lowered to 98%.”

4. Independence A and B are independent if P(A  B) = P(A) x P(B). – Independent: Pick a card from a deck. A = “The card is an ace”, P(A) = 1/13. B = “The card is a spade”, P(B) = 1/4 P(A  B) = 1/13 x 1/4 = 1/52. –Not independent: Draw two cards from a deck without replacement. A = “The first card is a space”, P(A) = 1/4 B = “The second card is a spade”, P(B) = 1/4 P(A  B) = (13 x 12) / (52 x 51) < 1/4 x 1/4.

5. Conditional Probability Example: –What is the probability that a husband will vote Democrat given that his wife does? – P(Husband Democrat |Wife Democrat ) –This is different from: What is the probability that a husband will vote democrat? What is the probability that a hustband and wife will vote democrat?

6. All possible events Conditional Probability P(B|A) is the conditional probability that B will occur given that A has occurred: P(B|A) = P(B  A) / P(A). P(A) = A occurs P(B) = B occurs P(B  A) = A and B occur

7. Conditional Probability Suppose we roll two dice –A = “The sum is 8” A = {(2,6), (3,5), (4,4), (5,3), (6,2)} P(A) = 5/36 –B = “The first die is 3” B = {(3,1), (3,2), (3,3), (3,4), (3,5), (3,6)} P(B) = 6/36 –A  B = {(3,5)} –P(B|A) = (1/36)/(5/36) = 1/5.

8. Conditional Probability 36 5 6 1 P(A) = 5/36 P(B) = 6/36 P(B  A) = 1/36 P(B|A) = (1/36)/(5/36) = 1/5 P(B|A) = P(B  A)/P(A)

9. Bayes Formula Suppose families have 1, 2, or 3 children with 1/3 probability each. Bobby has no brothers. What is the probability he is an only child?

10. Bayes Formula Suppose families have 1, 2, or 3 children with 1/3 probability each. Bobby has no brothers. What is the probability he is an only child? Let child 1, child 2, and child 3 be the events that A family has 1, 2, or 3 children, respectively. Let boy 1 be the event that a family has only 1 boy. Want to compute P(child 1 |boy 1 ) = P(child 1  boy 1 )/P(boy 1 )

11. Bayes Formula Suppose families have 1, 2, or 3 children with 1/3 probability each. Bobby has no brothers. What is the probability he is an only child? Want to compute P(child 1 |boy 1 ) = P(child 1  boy 1 )/P(boy 1 ) We need to compute P(child 1  boy 1 ) and P(boy 1 )

12. Bayes Formula We need to compute P(child 1  boy 1 ) and P(boy 1 ) Because P(B|C) = P(C  B) / P(C), we can write: P(C  B) = P(C) P(B|C). So, P(child 1  boy 1 ) = P(child 1 )P(boy 1 | child 1 ) = 1/3 x 1/2 = 1/6.

13. Bayes Formula We need to compute P(child 1  boy 1 ) and P(boy 1 ) P(boy 1 ) = P(child 1  boy 1 ) + P(child 2  boy 1 ) + P(child 3  boy 1 ) We know P(child 1  boy 1 ) = 1/6. Likewise, P(child 2  boy 1 ) = 1/6 P(child 3  boy 1 ) = 1/8 P(boy 1 ) = 1/6 + 1/6 + 1/8

14. Bayes Formula Suppose families have 1, 2, or 3 children with 1/3 probability each. Bobby has no brothers. What is the probability he is an only child? P(child 1 |boy 1 ) = P(child 1  boy 1 )/P(boy 1 ) = (1/6) / (1/6 + 1/6 + 1/8) = 4/11

15. Bayes Formula 1 Child 120 2 Children 120 3 Children 120 Suppose families have 1, 2, or 3 children with 1/3 probability each. Bobby has no brothers. What is the probability he is an only child? 1 boy 60 1 boy 60 1 boy 45

16. Bayes Formula 1 Child 120 2 Children 120 3 Children 120 1 boy 60 1 boy 60 1 boy 45 P(child 1 |boy 1 ) = P(child 1  boy 1 )/P(boy 1 ) = (60/360) / ((60+60+45)/360) = 60/165 = 4/11

17. Bayes Formula Event 1 Event 2 Event n Sub- event Sub- event Sub- event … Known: P(Event i ),  P(Event i ) = 1, and P(Subevent|Event i ) Compute: P(Event 1 |Subevent)

18. Bayes Formula Event 1 Event 2 Event n Sub- event Sub- event Sub- event … P(Event 1 |Subevent) = P(Event 1  Subevent) / P(Subevent) P(Event i  Subevent) = P(Event i )P(Subevent|Event i ) P(Subevent) =  P(Event i  Subevent) =  P(Event i )P(Subevent|Event i )

19. Bayes Formula Event 1 Event 2 Event n Sub- event Sub- event Sub- event … P(Event 1 )P(Subevent|Event 1 ) P(Event 1 |Subevent) = -----------------------------------------  P(Event i )P(Subevent|Event i )

20. Bayes Formula 1% of the population has a disease. 99% of the people who have the disease have the symptoms. 10% who don’t have the disease have the symptoms. Let D = “A person has the disease”. Let S = “A person has the symptoms”. P(D) =.01 and so P(~D) =.99 P(S|D) =.99 and P(S|~D) =.10 What is P(D|S)?

21. Bayes Formula P(D) P(S|D) P(D|S) = -------------------------------------------- P(D) P(S|D) + P(~D) P(S|~D) = (.01 x.99) / (.01 x.99 +.99 x.10) =.091

22. Bayes Formula P(Event 1 )P(Subevent|Event 1 ) P(Event 1 |Subevent) = -----------------------------------------  P(Event i )P(Subevent|Event i ) If there are a very large portion of events, the denominator may be very hard to calculate. If, however, you are only interested in relative probabilities…

23. Bayes Formula P(Event 1 )P(Subevent|Event 1 ) P(Event 1 |Subevent) = -----------------------------------------  P(Event i )P(Subevent|Event i )  P(Event 1 |Subevent) P(Event 1 )P(Subevent|Event 1 ) --------------------------- = ---------------------------------------- P(Event 2 |Subevent) P(Event 2 )P(Subevent|Event 2 ) This is called the “odds”.

24. Bayes Formula Let’s say you have 2 hypotheses (or models), H 1 and H 2, under consideration. The log odds of these two hypotheses given experimental data, D, is:

25. Bayes Formula Posterior odds = Relative belief in 2 hypotheses given the data. Quantity of interest.

26. Bayes Formula Prior odds = Relative belief in 2 hypotheses Before observing any data. Often assumed to be 1 (0 in log odds).

27. Bayes Formula Likelihoods = Relative probability of the data, given the two hypotheses. Usually computed from your models…