Adaptive Frequency Counting over Bursty Data Streams Bill Lin, Wai-Shing Ho, Ben Kao and Chun-Kit Chui Form CIDM07

Outline. Introduction Lossy Counting, Data Aging, Problem Definition System Architecture and Feedback Mechanism Algorithm Experiments Conclusion

Introduction. Large volumes of data. Real-time processing. Bursty traffic. Data aging.

Lossy Counting. Lossy Counting (LC) algorithm, the first one-pass algorithm for counting approximately the set of frequent itemsets over a stream of transactions. Given a user-specified error bound ε, LC processes incoming data bucket-by-bucket and updates a summary structure D. D contains a set of entries of the form, where X is an itemset, is an approximate support count of X, and δ(X) is the maximum possible error in.

Lossy Counting. (cont) The summary structure D has the following two properties: 1. For each itemset X, if X is not in D, then σ(X) < εN, where N is the total number of transactions processed. 2. For each entry in D, D is updated by the following rules: 1. If D does not contain an entry for X, we create an entry ifσ i (X) +εN 1 >εN Otherwise, X has an entry in D and the approximate support count is incremented by σ i (X). 3. If the updated entry satisfies, we delete the entry from D.

Data Aging. In many data stream applications, recent data are more important than older ones. denote the support count of X in bucket T i. denote the time-weighted support count of X obtain from the first k buckets of the stream.

Problem Definition. Given a data stream T with bursty transaction arrivals, a user- specified support threshold and a decay factor α, the problem is to report all itemsets X and to estimate their support counts such that (i) all frequent itemsets in T are reported, and (ii) a maximum possible error is reported for each estimated support count. Due to the large volume of transactions, we also require that transactions be loaded into memory and processed only once.

Architecture.

Architecture. (cont) The buffer manager monitors the occupancy of the buffer slots and submits statistical information to the speed regulator to control the mining speed. The mining module processes each bucket of buffered data an summarizes the mining result in its internal summary structure. The mining module submits feedback statistics to the speed regulator after each bucket is processed. The speed regulator receives statistical information from the buffer manager to estimate the data arrival rate. It also receives feedback information from the mining module to determine the mining speed. Base on this information, the speed regulator determines a target processing speed and sends a speed control signal to the mining module.

FeedBack Mechanism The core component of AFC is the feedback mechanism, which is implemented in the speed regulator. Estimates a target processing time, denoted by p i, which is the amount of time within which the mining module should complete the processing of the next bucket of transactions (T i ).

Estimating target Processing Time. The objective of the feedback mechanism is to try to maintain the buffer occupancy to a fraction f of the Q buffer slots. For example, if q i (denote the number of buffer slots that are occupied just before bucket Ti is processed.) is larger than the target buffer occupancy fQ, AFC should set p i to a small value so that transactions are mined at a higher rate to bring the occupancy down towards fQ. Hence, the number of buckets that have arrived during this period is 1 + q i -q i-1. Therefore, the bucket arrival rate is buckets per unit time.

Estimating target Processing Time. (cont) During this period of time, we have k bucket arrivals, and the occupancy changes from q i to fQ. Hence, the number of buckets processed is q i + k- fQ. If we assume that buckets arrive at a constant rate during this period, we have Since the system has to process q i + k- fQ buckets within this period of time, the target processing time of T i should be …………(1)

Estimating target Processing Time. (cont) This estimation is redone after each bucket is processed. A smaller value of k leads to a smaller restoration time (t restore ) and thus a higher processing speed is required at the mining module.

Speed Control. In AFC, the algorithm determines a suitable error threshold ε i for processing bucket T i in order that the target processing time P i is achieved. That is the sum of all the support counts in T i of the itemsets that are retained in the summary structure D i after the bucket T i is processed. We can discover the relationship between an ε i and this sum C i.

Speed Control. (cont) The larger ε i (i.e., the error threshold used when processing T i ) is, the fewer itemsets will be kept in D i, leading to a smaller C i. We now describe a two-step approach for determining ε i. 1) Step 1. Estimate a target value of C i from p i ……(2)

Speed Control. (cont) 2) Step 2. Estimate ε i from C i AFC determines a target processing time (p i ) based on Equation (1). It then calculates a target value of C i by Equation (2). Using the C i- 1 ε i-1 curve, AFC determines the value of ε i given the estimated value of C i. This error threshold is then applied when the mining module processes bucket T i.

Algorithm. AFC provides the following accuracy guarantees: 1. All itemsets whose true time-weighted support counts exceed are returned. 2. No itemset whose true time-weighted suppor count is less than is returned. 3. If no buckets are dropped in the processing of the data stream, then the difference between the reported support count of an itemset X and the true time-weighted support count of X is at most

Algorithm. (cont)

Experiment.

Experiment. (cont)

Conclusion.