1. Clustering Data Streams
Liadan O’Callaghan
Stanford University
Joint work with Sudipto Guha, Adam Meyerson, Nina Mishra, Rajeev Motwani

2. What Are Data Streams? Set x1, …, xn of points
Must be examined in this order in 1 pass
Can only store small number – can’t go back
Order may not be random
Memory xi
Data stream algorithms are evaluated by NUMBER OF “PASSES” or “LINEAR SCANS” over dataset (as well as by other usual measures). Our algorithm makes one pass.
Model proposed by Munro and Patterson (1980) who studied space requirements of selection and sorting; made formal by Henzinger, Raghavan, and Rajagopalan.
Data Stream

3. What Is Clustering? Given a set of points, divide into groups so that
Two items from the same group are similar
Two items from different groups are different
There are many different ways of defining clustering quality and similarity
Some clustering definitions include the idea of minimizing the all-pairs distances between points in the same cluster, minimizing the max radius or max diameter.

4. The k-Median Problem
Given n points in a metric space M, choose k medians from M
“Assign” each of the n points to the closest median
Goal: Minimize the sum of assignment distances
I will talk about continuous version, but, for discrete version, only lose some factors of 2.
NP-Hard

5. Clustering Heuristics
Fast, small space, usually with good results
…But no guarantees about quality!
For streams: e. g., BIRCH
Not for streams: e.g., CLARA(NS), K-means, HAC
Not all of these algorithms are designed for k-median

6. Approximation Algorithms
Guaranteed to give nearly optimal answer
…but require random access to entire dataset and may use lots of time and space
E. g., [Jain,Vazirani],[Charikar, Guha], [Indyk], [Arya et al.]
Jain/Vazirani: n^3 time, n^2 space, factor 4; Charikar/Guha: n^2logn time, n^2 space, factor 6.

7. Our Goals We want algorithms that Operate on data streams
Perform well in practice
And provide approximation guarantees

8. Outline of the Rest of the Talk
Restricting the search space if the optimal clusters are not “unreasonably” small
New (non-streaming) clustering algorithm with good average-case behavior (Meyerson’s algorithm)
New (non-streaming) k-median algorithm with good approximation ratio (whp)
Finally… Divide-and-conquer so we can handle a stream

9. Warning! In the next few slides: We will not talk about streams
We will assume random access to the whole data set
We will explore k-median and a variant called facility location

10. I. Restricting Search Space
Say we are solving k-median on some data set S in a metric space M. In essence we are searching all of M for a good set of k medians
k=4 M
Analysis of Adam Meyerson’s algorithm relies on related ideas, and we will return to this idea later
: Points in S
Any point in M
could be a median

11. I. Restricting Search Space
Consider the optimal solution (k members of M that best cluster data set S)
Assume each optimal cluster is “not tiny”
k=4 M
Analysis of Adam Meyerson’s algorithm relies on related ideas, and we will return to this idea later
Optimal Clusters

12. I. Restricting Search Space
Then an W(k logk)-point sample should have a few points from each optimal cluster
Solutions restricted to use medians from such a sample should still be good
k=4 M
Analysis of Adam Meyerson’s algorithm relies on related ideas, and we will return to this idea later
Sample points: can be chosen as medians
Other points: cannot be medians

13. Facility Location Problem
Medians are from original point set
Lagrange relaxation of k-median
No k is given, but pay f for each median
Cost function is
Sum of assignment distances
Plus (# medians) × f

14. k-Median vs. Facility Location
We’ve been discussing
continuous k-median
For a while, we’ll discuss
facility location
k = 2
Facility cost = 1
Cost is (3x1)=8 1 2
Cost is =11 2 3 4

15. II. Meyerson’s Algorithm
A facility location algorithm
Let f denote facility cost
Assumption: consider points in random order
First point becomes a median
If x = ith point, d = distance from x to closest existing median:
“open” x as a median with prob. d/f
else assign x to nearest median

16. II. Meyerson’s Algorithm
Let f = 10 9
assigned
(prob = .6) 4
“opened” (prob .9)

17. III. Local Search Algorithm
Our k-median algorithm will be based on local search, i.e.:
Start with initial solution (medians + assignment function)
Iteratively make local improvements to solution
After some number of iterations, your solution is provably good

18. III. Local Search Algorithm
Find initial solution (Meyerson)
Iterative local improvement (Charikar-Guha): Check each point, “opening,” “closing,” or reassigning so as to lower total cost
If #medians  k, adjust facility cost and repeat step 2.
At the end: k medians, approx. optimal

19. III. Local Search Algorithm
Point set,
Integer k
Initial
Solution
Iterative
Improvement
Steps
# medians?
= k
 k
Done
Adjust f

20. III. Local Search Algorithm
Too many medians!
Raise f and go back to step 2
Point
Set S
k=2
1. Initial
Solution
2. Iterative
Improvement
Success!
2. Iterative
Improvement

21. III. Local Search Algorithm
Instead of considering all points as feasible facilities, take a sample at the beginning, and only let sample points be medians
Fewer potential medians to search through
Solution converges faster
…And should still be good

22. III. Local Search Algorithm
The only points
Allowed to become
Medians are sample
points
Point
Set S
k=2
1. Initial
Solution
3. Iterative
Improvement
Success!
2. Choose Feasible Medians

23. III. Local Search Algorithm
Advantages of this algorithm:
Initial solution is
Fast: O(n(initial #medians))
Expected O(1)-approximation: so only need O(1) instead of O(logn) iterative improvement steps
Fast iterative improvements: O(nk logk)

24. IV. Divide & Conquer
Local search algorithm, as described, does not handle streams
Requires data to be randomly ordered
( Random access to whole data set)
Needs >1 pass to compute local changes
We want an algorithm with guarantees that also runs on streams

25. IV. Divide & Conquer Can we get the best of both worlds?
Cluster one dataset segment at a time
Store weighted medians for each segment
Cluster the medians k k k k k k

26. Does This Idea Work?
Given k-median approx. algorithm A (e.g., local search) and a dataset X
Prove that applying A by divide & conquer is ok.
I.e., show the divide & conquer solution has cost within a constant factor of the cost of A(X)
For example, A could be the local search algorithm just described.
If so, we have the advantages of one-scan, small memory (for data stream use) and of quality guarantees.

27. Original n Points…
(k = 2)

28. First Data Segment We produce weighted intermediate medians using A
(so they’re approx. optimal for this segment)

29. …and, again, weighted medians
Second Data Segment
…and, again, weighted medians
Segment is small enough that we can CLUSTER it in avail. Memory.

30. Re-cluster For Final Medians
(Final median weights omitted,
for simplicity of display)

31. How Close To Optimal?
OPT

32. Our Cost  3 x Cost of A Aloneb c a
b  Cost of A;
c  b + Cost of A;
So a  b + c  3 x Cost of A

33. Full Stream-Clustering Algo.k k k k k k

34. Many Thanks To
Umesh Dayal, Aris Gionis, Meichun Hsu, Piotr Indyk, Dan Oblinger, Bin Zhang