1. Mehdi Ghayoumi MSB rm 132 mghayoum@kent.edu Ofc hr: Thur, 11-12 a Machine Learning

2. Announcements Hw1 Feb-11 and Hw2 Mar-11 Programming HW1 Feb-25 and Programming Hw2 Apr-1 Quiz 1 March-11 and Quiz 2 Apr-8 First project report Mar-16,18. Second project report Apr-13,15. April 29 – Exam review May 4 – Final Exam

3. Machine Learning

4. Supervised Learning – Classification: – Regression: Unsupervised Learning – Clustering: – ART and SOM –PCA

5. Machine Learning Supervised vs. Unsupervised Learning Hierarchical Clustering – Hierarchical Agglomerative Clustering (HAC) Non-Hierarchical Clustering – K-means – Mixtures of Gaussians and EM-Algorithm

6. Machine Learning Clustering: Partition unlabeled examples into disjoint subsets of clusters, such that: – Examples within a cluster are similar – Examples in different clusters are different

7. Machine Learning

12. Cluster based on similarities/distances Distance measure between instances x r and x s Minkowski (L p ) (Euclidean for p = 2) City-block distance Hierarchical Clustering Machine Learning

13. Agglomerative Clustering Assume a similarity function that determines the similarity of two instances: sim(x,y). How to compute similarity of two clusters each possibly containing multiple instances? –Single Link: Similarity of two most similar members. –Complete Link: Similarity of two least similar members. Machine Learning

14. Single Link Agglomerative Clustering Use maximum similarity of pairs: Can result in “straggly” (long and thin) clusters due to chaining effect. Machine Learning

15. Single Link Example Machine Learning

16. Complete Link Agglomerative Clustering Use minimum similarity of pairs: Makes more “tight,” spherical clusters that are typically preferable. Machine Learning

17. Complete Link Example Machine Learning

19. Direct clustering methods require a specification of the number of clusters, k, desired. Randomly choose k instances as seeds, one per cluster. Form initial clusters based on these seeds. Iterate, repeatedly reallocating instances to different clusters to improve the overall clustering. Stop when clustering converges or after a fixed number of iterations. Machine Learning

20. Assumes instances are real-valued vectors. Clusters based on centroids, center of gravity, or mean of points in a cluster, c: Reassignment of instances to clusters is based on distance to the current cluster centroids. Machine Learning

21. Hard clustering typically assumes that each instance is given a “hard” assignment to exactly one cluster. Does not allow uncertainty in class membership or for an instance to belong to more than one cluster. Soft clustering gives probabilities that an instance belongs to each of a set of clusters. Allow uncertainty in class membership or for an instance to belong to more than one cluster. Machine Learning

22. Probabilistic method for soft clustering. Direct method that assumes k clusters: {c 1, c 2,… c k } Soft version of k-means. Assumes a probabilistic model of categories that allows computing P(c i | E) for each category, c i, for a given example, E. For text, typically assume a naïve-Bayes category model. –Parameters  = {P(c i ), P(w j | c i ): i  {1,…k}, j  {1,…,|V|}} Machine Learning

23. Iterate following two steps until convergence: –Expectation (E-step): Compute P(c i | E) for each example given the current model, and probabilistically re-label the examples based on these posterior probability estimates. –Maximization (M-step): Re-estimate the model parameters, , from the probabilistically re-labeled data. Machine Learning

24. Unlabeled Examples +  +  +  +   + Assign random probabilistic labels to unlabeled data Initialize: Machine Learning

25. Prob. Learner +  +  +  +   + Give soft-labeled training data to a probabilistic learner Initialize: Machine Learning

26. Prob. Learner Prob. Classifier +  +  +  +   + Produce a probabilistic classifier Initialize: Machine Learning

27. Prob. Learner Prob. Classifier Relabel unlabled data using the trained classifier +  +  +  +   + E Step: Machine Learning

28. Prob. Learner +  +  +  +   + Prob. Classifier Continue EM iterations until probabilistic labels on unlabeled data converge. Retrain classifier on relabeled data M step: Machine Learning

31. How to evaluate clustering? Improving search to converge faster and avoid local minima. Overlapping clustering. Ensemble clustering. Clustering structured relational data. Semi-supervised methods other than EM Machine Learning

32. Thank you!