By Graham Cormode and Marios Hadjieleftheriou Presented by Ankur Agrawal ( )

 Many data generation processes produce huge numbers of pieces of data, each of which is simple in isolation, but which taken together lead to a complex whole.  Examples ◦ Sequence of queries posed to an Internet search engine. ◦ Collection of transactions across all branches of a supermarket chain.  Data can arrive at enormous rates - hundreds of gigabytes per day or higher. Finding the Frequent Items in Streams of Data2

 Storing and indexing such large amount of data is costly.  Important to process the data as it happens.  Provide up to the minute analysis and statistics.  Algorithms take only a single pass over their input.  Compute various functions using resources that are sublinear in the size of input. Finding the Frequent Items in Streams of Data3

Problems range from simple to very complex.  Given a stream of transactions, finding the mean and standard deviation of the bill totals. ◦ Requires only few sufficient statistics to be stored.  Determining whether a search query has already appeared in the stream of queries. ◦ Requires a large amount of information to be stored.  Algorithms must ◦ Respond quickly to new information. ◦ Use very less amount of resources in comparison to total quantity of data. Finding the Frequent Items in Streams of Data4

 Given a stream of items, the problem is simply to find those items which occur most frequently.  Formalized as finding all items whose frequency exceeds a specified fraction of the total number of items.  Variations arise when the items are given weights, and further when these weights can also be negative. Finding the Frequent Items in Streams of Data5

 The problem is important both in itself and as a subroutine in more advanced computations.  For example, ◦ It can help in routing decisions, for in-network caching etc (if items represent packets on the Internet). ◦ Can help in finding popular terms if items represent queries made to an Internet search engine. ◦ Mining frequent itemsets inherently builds on this problem as a basic building block.  Algorithms for the problem have been applied by large corporations: AT&T and Google. Finding the Frequent Items in Streams of Data6

 Given a stream S of n items t 1 …t n, the exact ɸ-frequent items comprise the set {i | f i >ɸn}, where f i is the frequency of item i.  Solving the exact frequent items problem requires Ω(n) space.  Approximate version is defined based on tolerance for error parameterized by ɛ. Finding the Frequent Items in Streams of Data7

 Given a stream S of n items, the ɛ- approximate frequent items problem is to return a set of items F so that for all items i ε F, f i >(ɸ-ɛ)n, and there is no i∉F such that f i >ɸn. Finding the Frequent Items in Streams of Data8

 Given a stream S of n items, the frequency estimation problem is to process a stream so that, given any i, an f i * is returned satisfying f i ≤ f i * ≤ f i +ɛn. Finding the Frequent Items in Streams of Data9

 Two main classes of algorithms : ◦ Counter-based Algorithms ◦ Sketch Algorithms  Other Solutions : ◦ Quantiles : based on various notions of randomly sampling items from the input, and of summarizing the distribution of items. ◦ Less effective and have attracted less interest. Finding the Frequent Items in Streams of Data10

 Track a subset of items from the input and monitor their counts.  Decide for each new arrival whether to store or not.  Decide what count to associate with it.  Cannot handle negative weights. Finding the Frequent Items in Streams of Data11

 Problem posed by J. S. Moore in Journal of Algorithms, in Finding the Frequent Items in Streams of Data12

 Start with a counter set to zero. For each item: ◦ If counter is zero, store the item, set counter to 1. ◦ Else, if item is same as item stored, increment counter. ◦ Else, decrement counter.  If there is a majority item, it is the item stored.  Proof outline: ◦ Each decrement pairs up two different items and cancels them out. ◦ Since majority occurs > N/2 times, not all of its occurrences can be canceled out. Finding the Frequent Items in Streams of Data13

 First proposed by Misra and Gries in  Finds all items in a sequence whose frequency exceeds a 1/k fraction of the total count.  Stores k-1 (item, counter) pairs.  A generalization of the Majority algorithm. Finding the Frequent Items in Streams of Data14

 For each new item: ◦ Increment the counter if the item is already stored. ◦ If <k items stored, then store new item with counter set to 1. ◦ Otherwise decrement all the counters. ◦ If any counter equals 0, then delete the corresponding item. Finding the Frequent Items in Streams of Data15

Finding the Frequent Items in Streams of Data16

Finding the Frequent Items in Streams of Data

Finding the Frequent Items in Streams of Data18

 Time cost involves: ◦ O(1) dictionary operations per update. ◦ Cost of decrementing counts.  Can be performed in O(1) time.  Also solves the frequency estimation problem if executed with k=1/ɛ. Finding the Frequent Items in Streams of Data19

 Proposed by Manku and Motwani in  Stores tuples comprising ◦ An item ◦ A lower bound on its count ◦ A delta (△) value which records the difference between the upper bound and the lower bound. Finding the Frequent Items in Streams of Data20

 For processing i th item: ◦ Increment the lower bound if it is already stored. ◦ Else create a new tuple for the item with lower bound set to 1 and △ set to ⌊i/k⌋.  Periodically delete tuples with upper bound less than ⌊i/k⌋.  Uses O( k log(n/k) ) space in worst case.  Also solves frequency estimation problem with k=1/ɛ. Finding the Frequent Items in Streams of Data21

Finding the Frequent Items in Streams of Data22 * The CACM 2009 version of the paper has erroneously shown if block (line 8) outside for loop.

 Introduced by Metwally et al. in  Store k (item, count) pairs.  Initialize by first k distinct items and their exact counts.  If new item is not already stored, replace the item with least count and set the counter to 1 more than the least count.  Items which are stored by the algorithm early in the stream and are not removed have very accurate estimated counts. Finding the Frequent Items in Streams of Data23

Finding the Frequent Items in Streams of Data24

 The space required is O(k).  The time cost involves cost of: ◦ the dictionary operation of finding if an item is stored. ◦ finding and maintaining the item with minimum count.  Simple heap implementations can track the smallest count item in O(log k) time per update.  As in other counter based algorithms, it also solves frequency estimation problem with k=1/ɛ. Finding the Frequent Items in Streams of Data25

 The term “sketch” denotes ◦ a data structure. ◦ a linear projection of the input frequency vector.  Sketch is the product of frequency vector with a matrix.  Updates with negative values can easily be accommodated. ◦ Allows us to model the removal of items.  Primarily solve the frequency estimation problem.  Algorithms are randomized. ◦ Also take a failure probability δ so that the probability of failure is at most δ. Finding the Frequent Items in Streams of Data26

 Introduced by Charikar et al. in  Each update only affects a small subset of the sketch.  The sketch consists of a two-dimensional array C with d rows of w counters Each.  Uses two hash functions for each row: ◦ h j which maps input items onto [w]. ◦ g j which maps input items onto {−1, +1}.  Each input item i ◦ Add g j (i) to entry C[ j, h j (i)] in row j, for 1 ≤ j ≤ d. Finding the Frequent Items in Streams of Data27

Finding the Frequent Items in Streams of Data28

Finding the Frequent Items in Streams of Data29 For any row j, the value g j (i)*C[j,h j (i)] is an unbiased estimator for f i. The estimate f i * is the median of these estimates over the d rows.

 Setting d=log(4/δ) and w=O(1/ɛ 2 ) ensures that f i has error at most ɛn with probability of at least 1−δ. ◦ Requires the hash functions to be chosen randomly from a family of “fourwise independent” hash functions.  The total space used is.  The time per update is in worst case. Finding the Frequent Items in Streams of Data30

 Introduced by Cormode and Muthukrishnan in  Similarly uses a sketch consisting of 2-D array of d rows of w counters each.  Uses d hash functions h j, one for each row.  Each update is mapped onto d entries in the array, each of which is incremented.  Now frequency estimate for any item is f i * = min 1≤j≤d C[ j, h j (i)] Finding the Frequent Items in Streams of Data31

Finding the Frequent Items in Streams of Data32

 The estimate for an item in each row overestimates by less than n/w. ◦ Can be shown using Markov inequality.  Setting d=log(1/δ) and w=O(1/ɛ) ensures that f i has error at most ɛn with probability of at least 1−δ.  Similar to CountSketch with g j (i) always equal to 1.  The total space used is.  The time per update is in worst case. Finding the Frequent Items in Streams of Data33

 Sketches estimate the frequency of a single item. ◦ How to find frequent items without explicitly storing sketch for all items?  Divide-and-conquer approach limits search cost. ◦ Impose a binary tree over the domain ◦ Keep a sketch of each level of the tree ◦ Descend if a node is heavy, else stop  Correctness: all ancestors of a frequent item are also frequent.  Assumes that negative updates are possible but no negative frequencies are allowed.  Updates take \ hashing operations, and O(1) counter updates for each hash. Finding the Frequent Items in Streams of Data34

 Randomly divides the input into buckets.  Expect at most one frequent item in each bucket.  Within each bucket, the items are divided into subgroups so that the “weight” of each group indicates the identity of the frequent item.  Repeating this for all bit positions reveals the full identity of the item. Finding the Frequent Items in Streams of Data35

 Experiments performed using 10 million packets of HTTP traffic, representing 24 hours of traffic from a backbone router in a major network.  The frequency threshold ɸ varied from to 0.01 and the error guarantee ɛ set to ɸ.  Compare the efficiency of the algorithms with respect to: ◦ Update throughput, measured in number of updates per millisecond. ◦ Space consumed, measured in bytes. ◦ Recall, measured in the total number of true heavy hitters reported over the number of true heavy hitters given by an exact algorithm. ◦ Precision, measured in total number of true heavy hitters reported over the total number of answers reported. Finding the Frequent Items in Streams of Data36

Finding the Frequent Items in Streams of Data37

Finding the Frequent Items in Streams of Data38

 Overall, the SpaceSaving algorithm appears conclusively better than other counter-based algorithms.  SSH yields very good estimates, typically achieving 100% recall and precision, consumes very small space, and is fairly fast to update.  SSL is the fastest algorithm with all the good characteristics of SSH but consumes twice as much space on average. Finding the Frequent Items in Streams of Data39

 No clear winner among the sketch algorithms.  CMH has small size and high update throughput, but is only accurate for highly skewed distributions.  CGT consumes a lot of space but it is the fastest sketch and is very accurate in all cases, with high precision and good frequency estimation accuracy.  CS has low space consumption and is very accurate in most cases, but has slow update rate and exhibits some random behavior. Finding the Frequent Items in Streams of Data40

Finding the Frequent Items in Streams of Data41