Scalable Training of Mixture Models via Coresets Daniel Feldman Matthew Faulkner Andreas Krause MIT

Fitting Mixtures to Massive Data Importance Sample EM, generally expensiveWeighted EM, fast!

Coresets for Mixture Models *

Naïve Uniform Sampling 4

5 Small cluster is missed Sample a set U of m points uniformly  High variance

Sampling Distribution Sampling distribution Bias sampling towards small clusters

Importance Weights Weights Sampling distribution

Creating a Sampling Distribution Iteratively find representative points 8

Creating a Sampling Distribution Sample a small set uniformly at random 9 Iteratively find representative points

Creating a Sampling Distribution Remove half the blue points nearest the samples Sample a small set uniformly at random 10 Iteratively find representative points

Creating a Sampling Distribution Remove half the blue points nearest the samples Sample a small set uniformly at random 11 Iteratively find representative points

Creating a Sampling Distribution Remove half the blue points nearest the samples Sample a small set uniformly at random 12 Iteratively find representative points

Creating a Sampling Distribution Remove half the blue points nearest the samples Sample a small set uniformly at random 13 Iteratively find representative points

Creating a Sampling Distribution Remove half the blue points nearest the samples Sample a small set uniformly at random 14 Iteratively find representative points

Creating a Sampling Distribution Remove half the blue points nearest the samples Sample a small set uniformly at random 15 Iteratively find representative points

Creating a Sampling Distribution Remove half the blue points nearest the samples Sample a small set uniformly at random 16 Small clusters are represented Iteratively find representative points

Creating a Sampling Distribution Partition data via a Voronoi diagram centered at points 17

Creating a Sampling Distribution Sampling distribution 18 Points in sparse cells get more mass and points far from centers

Importance Weights Sampling distribution 19 Points in sparse cells get more mass and points far from centers Weights

20 Importance Sample

21 Coresets via Adaptive Sampling

A General Coreset Framework Contributions for Mixture Models:

A Geometric Perspective Gaussian level sets can be expressed purely geometrically: 23 affine subspace

Geometric Reduction Lifts geometric coreset tools to mixture models Soft-min

Semi-Spherical Gaussian Mixtures 25

Extensions and Generalizations 26 Level Sets

Composition of Coresets Merge [c.f. Har-Peled, Mazumdar 04] 27

Composition of Coresets Compress Merge [Har-Peled, Mazumdar 04] 28

Coresets on Streams Compress Merge [Har-Peled, Mazumdar 04] 29

Coresets on Streams Compress Merge [Har-Peled, Mazumdar 04] 30

Coresets on Streams Compress Merge [Har-Peled, Mazumdar 04] 31 Error grows linearly with number of compressions

Coresets on Streams Error grows with height of tree

33 Coresets in Parallel

Handwritten Digits Obtain 100-dimensional features from 28x28 pixel images via PCA. Fit GMM with k=10 components. 34 MNIST data: 60,000 training, 10,000 testing

35 Neural Tetrode Recordings Waveforms of neural activity at four co-located electrodes in a live rat hippocampus. 4 x 38 samples = 152 dimensions. T. Siapas et al, Caltech

36 Community Seismic Network Detect and monitor earthquakes using smart phones, USB sensors, and cloud computing. CSN Sensors Worldwide

Learning User Acceleration dimensional acceleration feature vectors Bad Good

38 Seismic Anomaly Detection Bad Good GMM used for anomaly detection

Conclusions Lift geometric coreset tools to the statistical realm - New complexity result for GMM level sets Parallel (MapReduce) and Streaming implementations Strong empirical performance, enables learning on mobile devices GMMs admit coresets of size independent of n - Extensions for other mixture models 39