1. Probabilistic Generative Models Rong Jin

2. Probabilistic Generative Model Classify instance x into one of K classes Class prior Density function for class C k

3. Probabilistic Generative Model Classification decision Key is to decide parameters

4. Probabilistic Generative Model Given training data

5. Probabilistic Generative Model

7. Singularity of covariance matrix Overfitting problem Solutions Diagonalize the covariance matrix Smoothing/regularization

8. Naïve Bayes Difficult to estimate for high dimensional data x Naïve Bayes approximation Distribution of 1 D Diagonalize the covariance matrix

9. Naïve Bayes Text categorization : word histogram of a document

10. Naïve Bayes Bad approximation Good classification accuracy Text categorization for 20 Newsgroups

11. Naïve Bayes It is the ratio that matters

12. Decision Boundary Consider text categorization of two classes Linear decision boundary

13. Decision Boundary Consider two class classification Gaussian density function Shared covariance matrix Linear decision boundary

14. Decision Boundary Generative models essentially create linear decision boundaries Why not directly model the linear decision boundary

15. Assumption of Generative Models It misses the factor How important is ?

16. Ambiguous Training Data Training data : training data only indicates the set of class labels to which the true class assignment belongs to