1. CS 59000 Statistical Machine learning Lecture 10 Yuan (Alan) Qi Purdue CS Sept. 25 2008

2. Outline Review of Fisher’s linear discriminant, percepton, probabilistic generative models, Probabilistic discriminative models: Logistic regression Probit regression

3. Fisher Linear Discriminant

4. Within Class and Between Class Scatter Matrices

5. Generative eigenvalue problem

6. Fisher’s Linear Discriminant

7. Perceptron

8. Generalized Linear Model Minimize where M denotes the set of all misclassified patterns

9. Stochastic Gradient Descent

10. Probabilistic Generative Models

11. Gaussian Class-Conditional Densities Conditional densities of data: The posterior distribution for label/class:

12. Maximum Likelihood Estimation Linked to Fisher’s linear discriminant

13. Discrete features Naïve Bayes classification:

14. Probabilistic Discriminative Models Instead of modeling Model directly

15. Generative vs Condition Models Discussion

16. Logistic Regression Let Likelihood function

17. Maximum Likelihood Estimation Note that

18. Newton-Raphson Optimization for Linear Regression Let H denote Hessian matrix It converges in one iteration for linear regression.

19. Newton-Raphson Optimization for Logistic Regression Gradient and Hessian of the error function:

20. Newton-Raphson Optimization for Logistic Regression Iterative reweighted least squares (IRLS): Solving a series of weighted least-square problems

21. Other discriminative models Generative models Logistic regression How about other discriminative functions?

22. Probit Regression Probit function:

23. Labeling Noise Model Robust to outliers and labeling errors

24. Generalized Linear Models

25. Generalized linear model: Activation function: Link function:

26. Canonical Link Function If we choose the canonical link function: Gradient of the error function:

27. Laplace Approximation for Posterior Gaussian approximation around mode:

28. Illustration of Laplace Approximation

29. Evidence Approximation

30. Bayesian Information Criterion Approximation of Laplace approximation: More accurate evidence approximation needed

31. Bayesian Logistic Regression