1. Bregman Divergences in Clustering and Dimensionality Reduction COMS 6998-4: Learning and Empirical Inference Irina Rish IBM T.J. Watson Research Center Slide credits: Srujana Merugu, Arindam Banerjee, Sameer Agarwal

2. Outline Intro to Bregman Divergences Clustering with Bregman Divergences  k-means: quick overview  From Euclidean distance to Bregman divergences  Some rate-distortion theory Dimensionality Reduction with Bregman Divergences  PCA: quick overview  Probabilistic Interpretation of PCA; exponential family  From Euclidean distance to Bregman divergences Conclusions

3. Distance (distortion) measures in learning Euclidean distance – most commonly used  Nearest neighbor, k-means clustering, least squares regression, PCA, distance metric learning, etc But…is it always an appropriate type of distance? No!  Nominal attributes (e.g. binary)  Distances between distributions Probabilistic interpretation:  Euclidean distance  Gaussian data  Beyond Gaussian? Exponential family distributions  Bregman divergences

8. Squared Euclidean distance is a Bregman divergence

9. Relative entropy (i.e., KL-divergence) is another Bregman divergence

27. Recall Bregman Diverences

32. Now, how about generalizing soft clustering Algorithms using Bregman divergences?

33. (natural parameter)

53. Add a bit of unit-variance Gaussian noise to each point

54. Now remove the original model…

59. Remember the exponential family?

62. Remember Bregman Divergences?

76. Discussion