0 0 Review Probability Axioms –Non-negativity P(A)≥0 –Additivity P(A U B) =P(A)+ P(B), if A and B are disjoint. –Normalization P(Ω)=1 Independence of two events A and B –P(A∩B)=P(A)P(B) –Two coin tosses A={first toss is a head}, B={second toss is a head} –Disjoint vs independent P(A∩B)=0, if P(A)>0 and P(B)>0, P(A∩B) < P(A)P(B). They are never independent. A={first toss is a head}, B={first toss is a tail} P(A)=0.5, P(B)=0.5, P(A∩B)=0

Conditional Probability 1.4 Total Probability Theorem and Bayes’ Rule

2 2 Conditional Probability A way to reason about the outcome given partial information Example1 –To toss a fair coin 100 times, what’s the probability that the first toss was a head? Fair coin 1/2 –To toss a fair coin 100 times, if 99 tails come up, what’s the probability that the first toss was a head? Very small? Example2 –A fair coin and an unfair coin (1/4 tail, 3/4 head) The first toss is fair, if the outcome is a head, use the fair coin for the 2 nd toss, if the outcome is a tail, use the unfair coin for the 2 nd toss. What’s the probability that the 2nd toss was a tail? –½x½ + ½x¼ = What’s the probability that the 2nd toss was a tail if we know that the first toss was a tail? –1/4

3 3 Conditional Probability Definition of a conditional probability –The probability of event A given event B (P(B)>0 ) –P(A|B)=P(A) if A and B are independent A new probability law (recall the definition of probability laws) –

4 4 Conditional Probability Examples –Two rolls of a die, what’s the probability that the first roll was a 1? –Fair dice 1/6 –Two rolls of a die, the sum of the two rolls is 6, what’s the probability that the first roll was a 1? B: (1,5) (2,4) (3,3) (4,2) (5,1), A and B: (1,5) P(A|B)= (1/36)/(5/36)=1/5 –Two rolls of a die, the sum of the two rolls is 6, what’s the probability that the first roll was EVEN? B: (1,5) (2,4) (3,3) (4,2) (5,1), A and B: (2,4) (4,2) P(A|B)= (2/36)/(5/36)=2/5

5 5 Conditional Probability The new universe is B P(A 1 )> P(A 2 ), does it mean that P(A 1 |B)> P(A 2 |B)? No! –An Example: Two rolls of a die B: the sum of the two rolls is 4, (1,3) (2,2) (3,1) A 1 : the first roll was 1 or 2 A 2 : the first roll was 3, 4, 5 or 6 –P(A 1 )=1/3 P(A 2 )=2/3 P(B)= 3/36 =1/12 –P(A 1 ∩ B) = 2/36 = 1/18 P(A 2 ∩ B) = 1/36 –P(A 1 | B)= (1/18)/(1/12) =2/3 P(A 2 | B)= (1/36)/(1/12) =1/3

6 6 Conditional Probability The Chain Rule

7 7 Conditional Independence Conditional independence, A and C are independent conditional on B, P(B)>0 P(A∩C|B)=P(A|B) P(C|B) Example (conditional independence ≠ independence): unfair coins, coin 1- (0.9 head, 0.1 tail) coin 2- (0.1 head, 0.9 tail), coin 3 is fair. –Toss coin 3 first. If it’s head, toss coin 1 twice. If it’s tail, toss coin 2 twice. –A= X H X, the event that the 2nd toss is a head –C= X X H, the event that the 3rd toss is a head –B= H X X, the event that the first toss is a head

8 8 Total Probability Theorem A 1, A 2, … A n be a partition of Ω –Recall the definition of a partition Total Probability Theorem

9 9 An example –A fair coin and an unfair coin (1/4 tail, 3/4 head) The first toss is fair, if the outcome is a head, use the fair coin for the 2nd and 3rd toss, if the outcome is a tail, use the unfair coin. –B={ the 2nd and 3rd tosses are both tails} –A 1 ={the first toss is an head}, A 2 ={the first toss is a tail}. A 1 and A 2 is a partition of the universe.. P(A 1 )=P(A 2 )= 1/2 –P(B|A 1 )= 1/4, P(B|A 2 )= 1/16

10 Bayes’ Rule A 1, A 2, … A n be a partition of Ω Bayes’ Rule

11 Bayes’ Rule An Example Question: –How likely is there a tumor given that a shade is observed? –P(A 2 |B)

12 Bayes’ Rule Bayes’ Rule from scratch

13 Sending a bit through a noisy channel Sender has a bit b- either 0 or 1 with equal probability to send to the receiver –p=0.1 –Question1: if the sender sends b once, and the receiver receives 1, what can the receiver say about b? –Question2: if the sender sends b 3 times, and the receiver receives 1,1,1 what can the receiver say about b? –Question3: if the sender sends b 3 times, and the receiver receives 1,0,1 what can the receiver say about b?