1. CS 59000 Statistical Machine learning Lecture 7 Yuan (Alan) Qi Purdue CS Sept. 16 2008 Acknowledgement: Sargur Srihari’s slides

2. Outline Review of noninformative priors, nonparametric methods, and nonlinear basis functions Regularized regression Bayesian regression Equivalent kernel Model Comparison

3. The Exponential Family (1) where ´ is the natural parameter and so g(´) can be interpreted as a normalization coefficient.

4. Property of Normalization Coefficient From the definition of g(´) we get Thus

5. Conjugate priors For any member of the exponential family, there exists a prior Combining with the likelihood function, we get Prior corresponds to º pseudo-observations with value Â.

6. Noninformative Priors (1) With little or no information available a-priori, we might choose a non-informative prior. ¸ discrete, K -nomial : ¸ 2 [a,b] real and bounded: ¸ real and unbounded: improper! A constant prior may no longer be constant after a change of variable; consider p(¸) constant and ¸=´ 2 :

7. Noninformative Priors (2) Translation invariant priors. Consider For a corresponding prior over ¹, we have for any A and B. Thus p(¹) = p(¹ { c) and p(¹) must be constant.

8. Noninformative Priors (4) Scale invariant priors. Consider and make the change of variable For a corresponding prior over ¾, we have for any A and B. Thus p(¾) / 1/¾ and so this prior is improper too. Note that this corresponds to p(ln ¾) being constant.

9. Nonparametric Methods (1) Parametric distribution models are restricted to specific forms, which may not always be suitable; for example, consider modelling a multimodal distribution with a single, unimodal model. Nonparametric approaches make few assumptions about the overall shape of the distribution being modelled.

10. Nonparametric Methods (2) Histogram methods partition the data space into distinct bins with widths ¢ i and count the number of observations, n i, in each bin. Often, the same width is used for all bins, ¢ i = ¢. ¢ acts as a smoothing parameter. In a D -dimensional space, using M bins in each dimen- sion will require M D bins!

11. Nonparametric Methods (3) Assume observations drawn from a density p(x) and consider a small region R containing x such that The probability that K out of N observations lie inside R is Bin(K j N,P ) and if N is large If the volume of R, V, is sufficiently small, p(x) is approximately constant over R and Thus

12. Nonparametric Methods (5) To avoid discontinuities in p(x), use a smooth kernel, e.g. a Gaussian Any kernel such that will work. h acts as a smoother.

13. K -Nearest-Neighbours for Classification (1) Given a data set with N k data points from class C k and, we have and correspondingly Since, Bayes’ theorem gives

14. K -Nearest-Neighbours for Classification (2) K = 1 K = 3

15. Basis Functions

16. Examples of Basis Functions (1)

17. Maximum Likelihood Estimation (1)

18. Maximum Likelihood Estimation (2)

19. Sequential Estimation

20. Regularized Least Squares

21. More Regularizers

22. Visualization of Regularized Regression

23. Bayesian Linear Regression

24. Posterior Distributions of Parameters

25. Predictive Posterior Distribution

26. Examples of PredictiveDistribution

27. Question Suppose we use Gaussian basis functions. What will happen to the predictive distribution if we evaluate it at places far from all training data points?

28. Equivalent Kernel Given Predictive mean is where

29. Equivalent kernel Basis Function: Equivalent kernel: Gaussian Polynomial Sigmoid

30. Covariance between two predictions Predictive mean at nearby points will be highly correlated, whereas for more distant pairs of points the correlation will be smaller.

31. Bayesian Model Comparison Suppose we want to compare models. Given a training set, we compute Model evidence (also known as marginal likelihood): Bayes factor:

32. Likelihood, Parameter Posterior & Evidence Likelihood and evidence Parameter posterior distribution and evidence

33. Crude Evidence Approximation Assume posterior distribution is centered around its mode

34. Evidence penalizes over-complex models Given M parameters Maximizing evidence leads to a natural trade- off between data fitting & model complexity.

35. Evidence Approximation & Empirical Bayes Approximating the evidence by maximizing marginal likelihood. Where hyperparameters maximize the evidence. Known as Empirical Bayes or type2 maximum likelihood

36. Model Evidence and Cross-Validation Root-mean-square error Model evidence Fitting polynomial regression models

37. Next class Linear Classification