1. INTRODUCTION TO MACHINE LEARNING Bayesian Estimation

2. Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2  Estimating parameters of a model from the data  Regression  Classification  Have some prior knowledge on possible parameter range  Before looking at the data  Distribution of the parameter

3. Generative Model Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 3

4. Bayes Rule Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 4

5. Multinomial variable Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 5  Sample of multinomial data taking one of K state  Sample Likelihood  Good way to specify prior distribution on state probabilities q

6. Dirichlet Distribution Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 6  Probability of each combination of state probabilities  Parameters: approximate proportions of data in state q i

7. Posteriori Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 7  Likelihood  Posteriori

8. Conjugate Prior Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 8  Posteriori and prior have the same form  Sequential learning  Instance by instance  Calculate posteriori for the current item  Make it prior for the next item

9. Continuous Variable Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 9  Instances are Gaussian Distributed with unknown parameters  Conjugate prior

10. Continuous Variable Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 10 Posteriori Mean is weighted combination of sample mean and prior mean More samples, estimate is closer to m Little prior uncertainty=>closer to prior mean

11. Precision/Variance Prior Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 11  More convenient to work with precision  Conjugate prior is a Gamma Distribution

12. Precision Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 12  Posteriori is a weighted sum of prior and sample statistics

13. Parameter Estimation Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 13  Used prior to refine distribution parameter estimates  User prior to refine parameter of some function of the input  Regression  Classification discriminant

14. Regression Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 14

15. Regression Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 15  Maximum Likelihood  Prediction  Gaussian Prior

16. Prior on weights Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 16

17. Examples 17