1. Bayesian Prior and Posterior Study Guide for ES205 Yu-Chi Ho Jonathan T. Lee Nov. 24, 2000

2. 2 Outline  Conditional Density  Bayes Rule  Conjugate Distribution  Example  Other Conjugate Distributions  Application

3. 3 Conditional Density  The conditional probability density of w happening given x has occurred, assume p x (x)  0: N

4. 4 Bayes Rule  Replace the joint probability density function with the bottom equation from page 3: N

5. 5 Conjugate Distribution  W: parameter of interest in some system  X: the independent and identical observation on the system  Since we know the model of the system, the conditional density of X|W could be easily computed, e.g., N

6. 6 Conjugate Distribution (cont.)  If the prior distribution of W belong to a family, for any size n and any values of the observations in the sample, the posterior distribution of W must also belong to the same family. This family is called a conjugate family of distributions. N

7. 7 Example  An urn of white and red balls with unknown w being the fraction of the balls that are red.  Assume we can take n sample, X 1, …, X n, from the urn, with replacement, e.g, n i.i.d. samples. This is a Bernoulli distribution. N

8. 8 Example (cont.)  Total number of red ball out of n trials, Y = X 1 + … + X n, has the binomial distribution  Assume the prior dist. of w is beta distribution with parameters  and  N

9. 9 Example (cont.)  The posterior distribution of W is which is also a beta distribution.

10. 10 Example (cont.)  Updating formula: ’ =  + y Posterior (new) parameter = prior (old) parameter + # of red balls ’ =  + (n – y) Posterior (new) parameter = prior (old) parameter + # of white balls N

11. 11 Other Conjugate Distributions  The observations forms a Poisson distribution with an unknown value of the mean w.  The prior distribution of w is a gamma distribution with parameters  and .  The posterior is also a gamma distribution with parameters and  + n.  Updating formula: ’ =  + y ’ =  + n N

12. 12 Other Conjugate Distributions (cont.)  The observations forms a negative binomial distribution with a specified r value and an unknown value of the mean w.  The prior distribution of w is a beta distribution with parameters  and .  The posterior is also a beta distribution with parameters  + rn and.  Updating formula: ’ =  + rn ’ =  + y N

13. 13 Other Conjugate Distributions (cont.)  The observations forms a normal distribution with an unknown value of the mean w and specified precision r.  The prior distribution of w is a normal distribution with mean  and precision .  The posterior is also a normal distribution with mean and precision  + nr.  Updating formula: N

14. 14 Other Conjugate Distributions (cont.)  The observations forms a normal distribution with the specified mean m and unknown precision w.  The prior distribution of w is a gamma distribution with parameters  and .  The posterior is also a gamma distribution with parameters and.  Updating formula: ’ =  + n/2 ’ =  + ½ N

15. 15 Summary of the Conjugate Distributions ObservationsPriorPosterior BernoulliBeta PoissonGamma Negative binominal Beta Normal Gamma N

16. 16 Application  Estimate the state of the system based on the observations: Kalman filter. N

17. 17 References: DeGroot, M. H., Optimal Statistical Decisions, McGraw-Hill, 1970. Ho, Y.-C., Lecture Notes, Harvard University, 1997. Larsen, R. J. and M. L. Marx, An Introduction to Mathematical Statistics and Its Applications, Prentice Hall, 1986.