1 Incremental Aggregation on Multiple Continuous Queries Chun Jin Carnegie Mellon University 09/28/2006 ISMIS, Bari Italy

2 Intelligence monitoring Fraud detection Onset epidemic patterns Network intrusion detection GeoSpacial changes Transactions Senor network readings Network traffic data Stream Processing

3 Problem Aggregate queries Continuous evaluation Multiple concurrent queries

4 Solutions Incremental aggregation Incremental multiple aggregate query optimization (incremental sharing)

5 Roadmap System overview Query examples Incremental Aggregation Incremental sharing Evaluation

6 Query Network Query Coordinator System Catalog Common Computation Identifier (CCI) Network Operation Manager (NOM) Code Assembler Sharing Optimizer (SO) Projection Manager (PM) System Architecture New Query Insertion: 1.Index query network 2.Identify common computation 3.Select optimal sharing path 4.Expand query network Query Network Execution: 1.Code assembly 2.Incremental aggregation 3.Periodical execution Engine Generator Oracle

7 SAB hospitalvdateCOUNT(*)SUM(fee)AVERAGE(fee) SA dis_cathospitalvdateCOUNT(*)SUM(fee)AVERAGE(fee) SELECT dis_cat, hospital, vdate, COUNT(*), AVERAGE(fee) FROM Med GROUP BY CAT(disease) AS dis_cat, hospital, DAY(visit_time) AS vdate (a) Query A SELECT hospital, vdate, AVERAGE(fee) FROM Med GROUP BY hospital, DAY(visit_time) AS vdate (b) Query B Query Examples SHSH SNSN AHAH ANAN

8 Roadmap System overview Query examples Incremental Aggregation Incremental sharing Evaluation

9 Aggregate Function Types Distributive: aggregate function itself. Sum, count. Algebraic: a finite set of aggregate functions. Average. Holistic: no such finite set. Quantiles. Incremental Aggregation

10 Holistic Aggregation Revisiting the entire history. Usage: –For holistic aggregates. –For post-non-incrementally-evaluated aggregates. –Baseline to incremental aggregation. Incremental Aggregation

11 GIDCOUNT(*) AS COUNTA SUM(fee) AS SUMA AVERAGE(fee) AS AVGA GIDCOUNT(*) AS COUNTA SUM(fee) AS SUMA AVERAGE(fee) AS AVGA 0: PreUpdate State 1: Aggregate A N t1: A H t2: A N SHSH SNSN 2: Merge Groups t2.COUNTA = t1.COUNTA + t2.COUNTA t2.SUMA = t1.SUMA + t2.SUMA 3: Compute Algebraic Aggregate 4: Drop Duplicates 5: Insert New Results Algorithm Incremental Aggregation

12 Complexity 1.Aggregate S N. T 1 = O(|S N |) 2.Merge groups in A H to A N. T curr2 = O(|A H | + |A N |), T hash2 = O(|A H | + |A N |), T prefetch2 = O(|A N |) 3.Compute algebraic aggregates in A N. T 3 = O(|A N |) 4.Drop duplicates. T curr4 = O(|A N |*|A N H |) = O(|A N | 2 ), T hash4 = O(|A H |+|A N |), T prefetch4 = O(|A N |) 5.Insert new results. T 5 = O(|A N |) Incremental Aggregation

13 Implementation System catalog: –AggreRules –AggreBasics Incremental aggregation instantiation Incremental Aggregation

14 System Catalog Incremental Aggregation FunctionCategoryIncremental Aggregation Rule Vertical Expansion Rule AVERAGEASUMX/COUNTW SUMDSUMX(H)+SUMX(N)SUM(SUMX) MEDIANHNULL COUNTDCOUNTW(H)+COUNTW(N)SUM(COUNTW) FunctionBasicsBasic ID AVERAGECOUNT(W)COUNTW AVERAGESUM(X)SUMX SUMSUM(X)SUMX COUNTCOUNT(W)COUNTW AggreBasics AggreRules

15 AggreRules: AggreBasics: AVERAGE: SUM(X): SUMX AVERAGE: COUNT(W): COUNTW New Query A: AVERAGE(fee) GroupColumns: SUM(fee):SUMA COUNT(*):COUNTA AVERAGE(fee): AVGA AVERAGE fee SUM(X) SUMX COUNT(W) COUNTW SUM(fee) SUMX COUNT(*) COUNTW parse retrieve rules substitute insert columns substitute SUM(fee) SUMX SUMA COUNT(*) COUNTW COUNTA AVERAGE(fee) AVGA Name Mapping: Instantiation Incremental Aggregation

16 Roadmap System overview Query examples Incremental Aggregation Incremental sharing Evaluation

17 Incremental Multiple Query Optimization (Incremental Sharing) Index existing query plan information R. Given a new query Q, identify the sharable computations from R. Select the optimal sharing path. Expand R to compute Q. Incremental Sharing

18 Expanding Query Network Limited sharing on holistic aggregates Sharing on distributive/algebraic aggregates through vertical expansion Incremental Sharing

19 BIDRest ID COUNT(*) AS COUNTA SUM(fee) AS SUMA AVERAGE(fee) AS AVGA AHAH 1: Further Aggregate: COUNTB=SUM(COUNTA) SUMB=SUM(SUMA) GROUP BY BID 2: BIDCOUNT(*) AS COUNTB SUM(fee) AS SUMB AVERAGE(fee) AS AVGB BHBH 1: Further Aggregate COUNTB=SUM(COUNTA) SUMB=SUM(SUMA) GROUP BY BID A B Vertical Expansion Incremental Sharing Vertical Expansion

20 BI D Rest ID COUNT(*) AS COUNTA SUM(fee ) AS SUMA … ANAN A B BI D Rest ID … AHAH BI D COUNT(*) AS COUNTB SUM(fee ) AS SUMB AVERAGE(f ee) AS AVGB BHBH 2: Merge Groups t2.COUNTA = t1.COUNTA + t2.COUNTA t2.SUMA = t1.SUMA + t2.SUMA 1: Further Aggregate COUNTB=SUM(COUNTA) SUMB=SUM(SUMA) GROUP BY BID Vertical Expansion 3: Compute Algebraic Aggregate BI D COUNT(*) AS COUNTB SUM(fee ) AS SUMB AVERAGE(f ee) AS AVGB BNBN 4: Drop Duplicates 5: Insert New Results

21 Vertical Expansion Complexity T V curr = O(|A N | 2 + |B H |) T V hash = O(|A N | + |B H |) T V prefetch = O(|A N |) Incremental Sharing

22 OriginalDirectParentNodeNameGroupID OriginalExprCanonicalColumnNameNodeName OriginalGroupExprCanonicalGroupExprID GroupID GroupTopology GroupExprSet GroupExprIndex GroupColumns Incremental Sharing System Catalog

23 Select Optimal Sharing Path Select least-size node for sharing Incremental Sharing

24 Rerouting SB SA A B SB A SB B Animation Evolution Incremental Sharing

25 Roadmap System overview Query examples Incremental Aggregation Incremental sharing Evaluation

26 Evaluation Databases: –Synthesized FedWire money transfers –Anonymized Medical patient admission records Queries: –Seed queries –Generate sharable queries from seeds –A wild range of queries (aggregates in this paper) Simulation: –Historical data ( on Fed, and on Med) –Chunks of new data (4000 per chunk) Evaluation

27 Incremental Aggregation Fed (350 queries) Med (450 queries) Incremental Aggregation Non Incremental Aggregation Total execution time in seconds Evaluation

28 Number of FED queries Execution Time (s) (a) Fed Evaluation

29 Number of MED queries Execution Time (s) (a) Med Evaluation

30 Conclusion Multiple aggregates over streams Solutions: –Incremental aggregation –Incremental MQO (incremental sharing) –Built atop DBMSs for direct practical utility Big performance improvement Future work: –A broad range of queries –Built atop DSMSs.

31 Acknowledgement Work with Professor Jaime Carbonell. Part of ARGUS by CMU and Dynamix. Team: Phil Hayes, Santosh Ananthraman, Bob Frederking, Eugene Fink, Dwight Dietrich, Ganesh Mani, Johny Mathew. Thanks to Professor Chris Olston for helpful discussion.

32 Incremental Size: |S N | NonVE ITT VE ITT Non-VE IBT VE IBT IBT: Incremental-Batch Execution Time (s) ITT: Average Individual-Tuple Execution Time (s) FED Query Pair 1 (a) Pair 1 Evaluation