1. EE1J2 Mathematics for Applied Computing Lecture 9: Bayes’ Rule and Statistical Inference  Lecture content –Motivation –Conditional probabilities –Bayes’ rule –Simple statistical inference

2. Motivation  So far we have learnt how to estimate the probability that a random variable x takes a particular value v –The random variable x might correspond to throwing a dice and the values v are the possible outcomes –Or, in a much more complex example, the random variable x might correspond to speaking a word and the values v might be the possible acoustic signals which result when that word is spoken  Focussing on the second example, a much more interesting question is “given a value v, which word was spoken?” (in other words, automatic speech recognition)  This is inference, and it’s the topic of this lecture

3. Adding the ‘black dice’  I’ve added 6 black die to our set of 48 white die  We now have two classes –Class W: white die –Class B: black die  The black die are not fair, they are weighted. The figure shows their PMF, estimated over 300 throws

4. Conditional Probabilities  The probability of throwing a particular value now depends on the colour of the dice. – P(x=v | W) is the conditional probability that x takes the value v given a white dice  For example, based on the PMF for the black dice and the fact that a white dice is fair: – P(x=6 | W) = 1/6 = 0.167 – P(x=6 | B) = 0.8

5. More on joint probabilities  Now suppose that the 48 white and 6 black die are put into a bag.  A person selects a dice at random. What is the probability that the selected dice is white and a 6 is thrown?  This is another joint probability, denoted by P(x=6, W)  We can decompose this probability into the probability that a white die is chosen, times the probability that a 6 is thrown given that the die is white  In other words, P(x=6, W) = P(x=6|W) x P(W) – P(W) = 48/54 = 0.889 (because 48 of the 54 die are white), and – P(x=6|W) = 1/6 = 0.167  So, P(x=6, W) = 0.167 x 0.889 = 0.148

6. Bayes’ rule  For any value v we have: – P(x=v, W) = P(x=v|W) x P(W)  But the order is not important, so we could just as easily have written: – P(x=v, W) = P(W| x=v) x P(x=v)  Combining these two expressions gives: – P(W| x=v) x P(x=v) = P(x=v|W) x P(W)  This results in Bayes’ rule:

7. Inference  Bayes’ rule is important because it allows us to do inference  Consider the following question: –“A person chooses a dice randomly from the bag from the previous slide, throws it, and gets a 6. Is the dice black or white?”  From Bayes’ rule:  So, our best guess is that the die is white  Notice that the probabilities sum to 1, and that if I’m only interested in which is the larger of the two I can ignore the denominator

8. Inference (continued)  Now suppose that the person throws the die twice, and gets a six both times:  So, given more evidence the best guess is that the die is black  Do you understand why ?

9. Bayes’ rule - terminology  Provided that we know the relevant probabilities, we can apply this technique whenever we are trying to decide which class C has produced a value v  Bayes’ rule  Standard terminology: – P(C) is called the prior (or a priori) probability of the class C – P(x=v|C) is called the class conditional probability of v – P(C|x=v) is called the posterior (or a posteriori) probability of the class C

10. Another example  A man goes fishing to 3 different lakes: L 1, L 2 and L 3. –Because of varying difficulty in getting to the lakes, he visits the lakes with probabilities P(L 1 )=0.7, P(L 2 )=0.2, P(L 3 )=0.1 –Each lake contains the same 4 species of fish: carp, tench, pike and perch. The man stays at the lake until he has caught exactly 4 fish. –The probabilities of a fish belonging to each species from each lake are as follows: − P(carp|L 1 )=0.1, P(tench|L 1 )=0.8, P(perch|L 1 ) = 0.05, P(pike|L 1 )=0.05 − P(carp|L 2 )=0.7, P(tench|L 2 )=0.1, P(perch|L 2 ) = 0.1, P(pike|L 2 )=0.1 − P(carp|L 3 )=0.3, P(tench|L 3 )=0.3, P(perch|L 3 ) = 0.2, P(pike|L 3 )=0.2  The man returns from a days fishing with 1 tench, 2 carp and 1 pike. Which lake did he go to?

11. Example continued  Now suppose that our man camps by the same lake for three days and fishes each day: –On day 1 he catches 2 carp, 1 tench and 1 perch –On day 2 he catches 1 carp, 1 tench and 2 pike –On day 3 he catches 1 carp and 3 tench  Which lake did he go to?  What assumptions have we made, and are they reasonable?

12. Summary  The main objectives of this lecture were: –To introduce you to Bayes’ rule –To show how Bayes’ rule can be used to do simple statistical inference –To point out that this is the basic principle which underlies some very significant practical problems, e.g. automatic speech recognition, general pattern recognition, information retrieval, etc