1. Bregman Information Bottleneck NIPS’03, Whistler December 2003 Koby Crammer Hebrew University of Jerusalem Noam Slonim Princeton University

2. Motivation Extend the IB for a broad family of representations Relation to the Exponential family Hello, world Multinomial distribution Vectors

3. Outline Rate-Distortion Formulation Bregman Divergences Bregman IB Statistical Interpretation Summary

4. Information Bottleneck XTY X [p(y=1|X) … p(y=n|X)] [p(y=1|T) … p(y=n|T)] T

5. Input Variables Distortion Rate-Distortion Formulation

6. Bolzman Distribution: Markov + Bayes Marginal Self-Consistent Equations

7. Bregman Divergences f (u,f(u)) (v,f(v)) (v, f(u)+f’(u)(v-u)) B f (v||u) = f(v) - (f(u)+f’(u)(v-u))B f (v||u) = f:S R

8. Functional Bregman Function Input Variables Distortion Bregman IB: Rate-Distortion Formulation

9. Bolzman Distribution: Prototypes: convex combination of input vectors Marginal Self-Consistent Equations

10. Special Cases Information Bottleneck:  Bregman function : f(x)=x log(x) – x  Domain: Simplex  Divergence: Kullback-Leibler Soft K-means  Bregman function: f(x)=(1/2) x 2  Domain: Reals n  Divergence: Euclidian Distance  [Still, Bialek, Bottou, NIPS 2003]

11. Bregman IB Information Bottleneck Bregman Clustering Rate-Distortion Exponential Family

12. Expectation parameters: Examples (single dimension):  Normal  Poisson

13. Expectation parameters:  Properties :  Exponential Family and Bregman Divergences

14. Illustration

15. Expectation parameters:  Properties :   Exponential Family and Bregman Divergences

16. Distortion: Data vectors and prototypes: expectation parameters Question: For what exponential distribution we have ? Answer: Poisson Back to Distributional Clustering

17. Product of Poisson Distributions Illustration a a b a a a b a a a.8.2 ab 60 40 ab Pr Multinomial Distribution

18. Back to Distributional Clustering Information Bottleneck:  Distributional clustering of Poison distributions (Soft) k-means:  (Soft) Clustering of Normal distributions

19. Distortion Input:  Observations Output  Parameters of Distribution IB functional: EM [Elidan & Fridman, before] Maximum Likelihood Perspective

20. Posterior: Partition Function: Weighted  -norm of the Likelihood  → ∞, most likely cluster governs  →0, clusters collapse into a single prototype Back to Self Consistent Equations

21. Summary Bregman Information Bottleneck  Clustering/Compression for many representations and divergences Statistical Interpretation  Clustering of distributions from the exponential family  EM like formulation Current Work:  Algorithms  Characterize distortion measures which also yield Bolzman distributions  General distortion measures