Load Shedding Techniques for Data Stream Systems Brian Babcock Mayur Datar Rajeev Motwani Stanford University

Differences from Previous Talk Our focus: Aggregation queries No quality of service specifications – Instead, focus on accuracy of query answers Compensate for dropped data by scaling answers Random drops only (no semantic drops)

Problem Setting   ΣΣ   Σ S1S1 S2S2 R Sliding Window Aggregate Queries (SUM and COUNT) Operator Sharing Filters, UDFs, and Joins w/ Relations Q1Q1 Q2Q2 Q3Q3

Inputs to the Problem   ΣΣ   Σ S1S1 S2S2 R Std Dev σ Mean μ Q1Q1 Q2Q2 Q3Q3 Stream Rate r Processing Time t Selectivity s

Load Shedding via Random Drops Stream Rate r 11 22 Σ3Σ3 S (t 1, s 1 ) (t 2, s 2 ) Load = rt 1 + rs 1 t 2 + rs 1 s 2 t 3 (t 3, s 3 ) Sampling Rate p (time, selectivity) Load = rt 1 + p(rs 1 t 2 + rs 1 s 2 t 3 ) Scale answer by 1/p Need Load ≤ 1

Problem Statement Relative error is metric of choice: |Estimate - Actual| Actual Goal: Minimize the maximum relative error across queries, subject to Load ≤ 1 – Want low error with high probability

Relating Load Shedding and Error Relative error for query i Sampling rate for query i Query-dependent constant Equation derived from Hoeffding bounds Constant C i depends on: – Variance of aggregated attribute – Sliding window size

Calculate Ratio of Sampling Rates Minimize maximum relative error → Equal relative error across queries Express all sampling rates in terms of common variable λ

Placing Load Shedders   Σ Σ Target.8λ Target.6λ Sampling Rate.8λ Sampling Rate.75 =.6λ /.8λ

Conclusion Load shedding helps cope with bursts Minimizing relative error is natural objective for aggregate queries Algorithm for load shedding: – Relate target sampling rates for all queries – Place random drop operators based on target sampling rates – Adjust sampling rates to achieve desired load

Thanks for listening! Questions?

Choosing Target Sampling Rates Sampling rate for query Variance of aggregated attribute Sliding window size Relative Error

Measuring Inaccuracy 11 22 Σ3Σ3 Sampling Rate p 1 Sampling Rate p 2 Scale answer by 1/(p 1 p 2 ) Tuple w/ value x: x / (p 1 p 2 ) 0 with pr. p 1 p 2 with pr. 1-p 1 p 2 Key point: Product of sampling rates determines quality of approximate answer