Bayes Theorem and Review of Concepts Learnt 1 Krishna.V.Palem Kenneth and Audrey Kennedy Professor of Computing Department of Computer Science, Rice University

Content Bayes’ Theorem: Proof and In-class exercise solution Conditional Probability : Statistical Independence Revisited General Product Rule Review of concepts learnt so far Discussion of last year’s class projects 2

What is Bayes Theorem? Bayes' theorem relates the conditional and unconditional probabilities of events A and B, where B has a non-zero probability: Each term in Bayes' theorem has a conventional name: P(A) is the prior probability or unconditional probability of A. It is "prior" in the sense that it does not take into account any information about B. P(A|B) is the conditional probability of A, given B. P(B|A) is the conditional probability of B given A. P(B) is the prior or marginal probability of B 3

Alternate Form of Bayes Theorem Consider that A has two events : A 1 and A 2 If we want to compute the probability of A1 given B, then But, P(B) can be written as Hence, we get More generally, Bayes theorem can be written as 4

Understanding Bayes Theorem Bayes theorem is often used to compute posterior probabilities given observations. For example, a patient may be observed to have certain symptoms. Bayes' theorem can be used to compute the probability that a proposed diagnosis is correct, given that observation. Intuitively, Bayes’ theorem in this form describes the way in which one's beliefs about observing ‘A’ are updated by having observed ‘B’. 5

In class problem in Bayes’ Theorem 6 Suppose you walk home and notice that the grass is wet. You want to know if the grass is wet because it rained or because your sprinkler was on Looking at the news paper, you know the probability of raining today. Going by your memory, you somehow “estimate” the probability that the sprinkler was on today What is given? Probability it rained today Probability that your sprinkler was on today. Probability that the grass is wet given it rained or sprinkler was on Let us ignore the cases: “Sprinkler was on and it rained” and “Sprinkler was off and it did not rain”.

In class problem in Bayes Theorem 7 EventProbability Sprinkler On0.2 Sprinkler Off0.8 EventProbability Raining0.6 No Rain0.4 SprinklerRainGrass is wetGrass is dry OffRain OnNo Rain Probability of sprinkler being on Probability of raining Probability of grass being wet given different states of sprinkler and whether it rained Question: Given that the grass is wet, what is the probability that the sprinkler was on and it was not raining?

Solution to the problem Based on the problem statement, we ignore the cases when it rains and sprinkler is on or it rains and sprinkler is off. Let Sprinkler be represented as S, rain by R and grass by G. There are 2 cases of grass being wet: A 1 - Sprinkler =On and Rain=Off A 2 - Sprinkler = Off and Rain = On B – Grass is wet We need to find P(A 1 |B) Applying Bayes theorem from Slide 4, we have 8

Solution to the problem From the table, we get P(A 1 ) = P(Sprinkler =On and Rain=Off) = P(Sprinker = on) * P(Rain = Off) - (as the events are independent) = 0.2 * 0.4 = 0.08 P(A 2 ) = P(Sprinkler = Off and Rain=On) - (as the events are independent) = P(Sprinker = Off) * P(Rain = On) = 0.8 * 0.6 = 0.48 P(B|A 1 ) = 0.7 (given) P(B|A 2 ) = 0.9 (given) Applying Bayes theorem we get, = (0.08*0.7) / [(0.08*0.7)+(0.48*0.9)] =

In-Class Exercise Now, consider the same problem statement and attempt the following question: Question: Given that the grass is wet, what is the probability that it was raining and the sprinkler was not on? Note: Use Bayes theorem to solve this problem Answer:

Content Bayes’ Theorem – Proof and In-class exercise solution General Product Rule Review of concepts learnt so far Discussion of last year’s class projects 11

General Product Rule All along, we have been using product rule as given below P(A and B and C and …) = P(A)P(B)P(C)…. The above formula is a “Special case” of the general Product Rule. All the problems we have been dealing with have consisted of “Independent” Events Rolling of a pair of dies Tossing of coins Therefore, P(A and B and C and ….) = P(A)P(B)P(C)….. But what if they were not independent? Will the same formula work? NO!! So is there a general product rule which can be applied? YES!! 12

Product Rule Suppose we are interested in simultaneous occurrence of event A, B and C. Suppose these events are all dependent on each other P(A and B and C) = P(A)P(B|A)P(C|A,B) In general for n different dependent events A 1, A 2, A 3 ….A n Can we derive it? 13

Proof for Product Rule Let us consider just 2 “Dependent” events A 1 and A 2 Definition of conditional probability is P(A 2 | A 1 ) = P(A 1 and A 2 )/P(A 1 ) So P(A 1 and A 2 ) = P(A 2 | A 1 ) P(A 1 ) Now let us add a third event A 3 We need to represent P(A 1 and A 2 and A 3 ) in terms of P(A 1 and A 2 ). How about P(A 3 | A 1 A 2 )? P(A 3 | A 1 and A 2 ) = P(A 1 and A 2 and A 3 )/P(A 1 and A 2 ) Rearranging the terms of the equation, we get P(A 1 and A 2 and A 3 ) = P(A 3 | A 1 and A 2 ) P(A 1 and A 2 ) = P(A 3 | A 1 and A 2 ) P(A 2 | A 1 ) P(A 1 ) 14

General Product Rule In general we can extend this to n events In general for n different dependent events A 1, A 2, A 3 ….A n P(A 1 and A 2 and A 3 …. A n ) = P(A 1 )P(A 2 |A 1 )P(A 3 |A 1,A 2 )P(A 4 |A 1,A 2,A 3 )……………… P(A n |A 1,A 2,A 3,….,A n-1 ) 15 %

Content Bayes’ Theorem – Proof and In-class exercise solution Conditional Probability : Statistical Independence Revisited General Product Rule Review of concepts learnt so far Discussion of last year’s class projects 16

Summary of topics covered The notion of ‘chance’ dates back to primitive age a variety of animals, from bees to primates, embrace risk for a chance at a reward Numbers have evolved from crude representation to more simple representation like the Hindu-Arabic numbers Need for compactness The advent of ‘numbers’ provided people an opportunity to quantify chance. Numbers laid the foundation to analyzing random experiments and processes. This tool to analyze random experiments is probability. 17

Definitions and properties of probability Outcome: It’s the result of a single trial of an experiment Example: When you roll a die, one of the outcomes is a ‘6’. Event: It’s a collection of one or more outcome. Example: An event could be rolling 2 ‘6’s consecutively. Probability: The likelihood of the event occurring. Example: Probability of rolling 2 ‘6’s consecutively is 1/36 Properties of probability 0 ≤ P(x) ≤ 1 P(NULL SET) = 0 & P(SUPER SET OF EVENTS) = 1 P(All events except EVENT A) = 1 – P(EVENT A) 18

Random Variable Random Variable: A function that maps probability to real numbers Example: Event of seeing 2 ‘6’s can be mapped to number 12. This mapping is done by a random variable 19 In general, let the variable ‘x’ represent the event xEvent 1Event 1 2Event 2 3Event 3 4Event 4 Probability (Event i) Here ‘x’ is called a random variable. Where i={1,2,3,4}

Conditional Probability Conditional Probability: It’s the probability of an event given some additional information or given the information that another event occurred 20 Event 1 Event 2 Event 3 … Event Even Event Odd Knowledge of the outcome of the roll of a die in terms of whether it is an even number or an odd number allows us to predict the actual outcome more precisely p p1p1 p2p2 p3p3 … p P even P odd So P(Event 1|Event Even) is the conditional probability of Event 1 given Event Even Experiment

Conditional probability is defined as follows If event A is dependent on another event B, then the probability of event A given knowledge about event B is For the die problem P(Die rolled a 2 | Die rolled an even number) = P(Die rolled 2 and Die rolled even) = 1/6 = 1/3!! P(Event A | Event B) = P(Event A and Event B occurring) P(Event B occurring) 21 P (Die rolled even)1 / 2 Conditional Probability

Mutual Exclusivity: Two events are mutually exclusive if the events cannot occur at the same time. Example: Heads and tails in a single toss are mutually exclusive events P(Heads Or tails) = P(Heads) + P(Tails) Sum Rule: If X 1, X 2, …. X n are N mutually exclusive events, then P(X 1 or X 1 or X 1 or …. X 1 ) = P(X 1 ) + P(X 2 ) + …. + P(X n ) Product Rule: If X 1, X 2, …. X n are N independent events, then P(X 1 & X 1 & X 1 & …. X 1 ) = P(X 1 )P(X 2 )…. P(X n ) 22 Definitions and properties of probability

Independence Independence: Two events are independent if occurrence of one event does not affect the occurrence of the other. Example: Getting a 6 on a die does not affect getting a 2 on the other die Formally, P(B|A) = P(B) Alternately, “ The probability of two independent events occurring simultaneously is equal to the product of probability of individual events” 23 Random variables X and Y are independent if and only if For every

Definitions and properties of probability Distribution: A compact representation of all the events and their outcomes, typically in the form of an equation. 24 Xp(X) 11/6 All other number 5/6 P(just one throw) = P(getting a 1 in first throw) = 1/6 P(just two throws) = P(getting number other than 1 in first throw)* P(getting number 1 on second) = (5/6)*(1/6) P(just N throws) = P(getting number other than 1 in first N-1 throws)*P(getting 1 on Nth throw) = (5/6) N-1 (1/6) This is a “geometric distribution”

Bayes Theorem Bayes Theorem: Shows the relationship between a conditional probability with its reverse form Each term in Bayes' theorem has a conventional name: P(A) is the prior probability or unconditional probability of A. It is "prior" in the sense that it does not take into account any information about B. P(A|B) is the conditional probability of A, given B. P(B|A) is the conditional probability of B given A. P(B) is the prior or marginal probability of B 25

Transition Graphs Model or Transition Graphs: A set of events can be modeled as a transition graph with the nodes representing the outcomes and the edges having the associated probabilities Eg: Snakes and Ladders Game /

Content Bayes’ Theorem – Proof and In-class exercise solution Conditional Probability : Statistical Independence Revisited General Product Rule Review of concepts learnt so far Discussion of last year’s class projects 27

Field Goal Percentage vs. Free Throw Percentage A case study of applied probability on sports By Yung-Seok Kevin Choi It is advantageous to quantify different aspects of a game to strategize In basketball two of the primary statistics are Field Goal Percentage ratio of field goals achieved to field goals attempted Free Throw Percentage ratio of free throws achieved to free throws attempted Solution Collect the data from a basketball reference website MATLAB to analyze and parse data The data was organized into different bins MATLAB to compute conditional probabilities such as Made statements such as “given a guard’s free throw percentage in bin 6, we are 85 percent confident that his field goal percentage falls between bins 6 and 11.” Problem Statement: In basketball, is field goal percentage related to free throw percentage and how does this relationship differ between positions?

Impartiality in Mafia By Jose Luis Garcia There is an interactive social game called Mafia The basic idea of the game is that there are two groups (Townspeople and Mafia) with each having different advantages who attempt to use these strengths in order to incapacitate the rival group and thus win the game. Is it the case that one of the two groups has more chance of winning than the other Specific Analysis What is the probability of being assigned to either mafia or townspeople? How many turns on average does it take to finish a game? Is there a group favored to win and if so which group? How is the win/loss probability related to the number of players? Solution Mathematical model of the game Define the variables (number of players, win probability etc.) MATLAB script to run multiple trials of the game Based on the ratio of townspeople to the mafia Define the probability of winning of the townspeople A card is chosen at random from a given stack (MATLAB script) Compute winning and losing probabilities based on multiple trials One such Conclusion As the number of players is increased the probability of Mafia winning is also increased. Problem Statement: Analyze the impartiality of an interactive social game called Mafia

Probability in Currency Exchange By Thomas Roinesdal The values of international currencies fluctuate every day Data about trading between among different currencies is available It will be very advantageous if one can predict the future value of the exchange rate of a currency based on the historical trends. Solution Model Consider pairs of currencies Example: 1 US dollar = 1.4 Singapore dollar Collect historical data of trends in the exchange rate Compute the relative frequency of different exchange rates Calculate conditional probabilities based on this data by applying Bayes Theorem Conclusion Based on historical data of many currency exchange rates, there was less than 30% chance of predicting accurately a future value of a given currency But it was shown that some currencies are more correlated than others Problem Statement: Is it possible to predict a currency given the historical values of other currencies?

Schedule for Help Sessions & Mini- Projects 16 Sept – Help Session 1 30 Sept – Final Project Proposal 5 Oct – Help Session 2 7 Oct – Mini Project 1 31

END 32