Aurora: a new model and architecture for data stream management Daniel J. Abadi 1, Don Carney 2, Ugur Cetintemel 2, Mitch Cherniack 1, Christian Convey 2, Sangdon Lee 2, Michael Stonebraker 3, Nesime Tatbul 2, Stan Zdonik 2 1 Department of Computer Science, Brandeis University 2 Department of Computer Science, Brown University 3 Department of EECS and Laboratory of Computer Science, M.I.T. Presenter: Saurin Kadakia
ABOUT ME MS CS STUDENT GRADUATING IN DEC 08 INTERESTED IN DATABASES AND WEB TECHNOLOGY
WHAT ARE MONITORING APPLICATIONS?? MONITORING APPLICATIONS ARE APPLICATIONS THAT MONITOR CONTINUOUS STREAMS OF DATA. EXAMPLES?? EXAMPLES?? MILITARY APPLICATIONS MILITARY APPLICATIONS FINANCIAL ANALYSIS APPLICATIONS FINANCIAL ANALYSIS APPLICATIONS TRACKING APPLICATIONS TRACKING APPLICATIONS
TRADITIONAL DBMS ASSUMPTIONS HUMAN ACTIVE, DBMS PASSIVE MODEL ONLY CURRENT VALUE IMPORTANT TRIGGERS/ASSERTIONS ARE SECONDARY QUERIES MUST HAVE EXACT ANSWERS NO REAL TIME SERVICE REQUIREMENTS
REALITY FOR MONITORING APPLICATIONS DBMS ACTIVE, HUMAN PASSIVE MODEL HISTORY OF VALUES REQUIRED TRIGGER ORIENTED APPLICATIONS APPROXIMATE ANSWERS TO QUERIES REAL TIME REQUIREMENTS
SYSTEM MODEL External data source User application Operator boxes data flow Continuous & ad hoc queries Historical Storage Aurora System QoS spec Query spec Application administrator
QUERY MODEL Traditional Traditional Structured Query Language Structured Query Language Declarative query on static data Declarative query on static data Aurora Aurora Data flow model for data stream Data flow model for data stream Application manager will construct queries using GUI Application manager will construct queries using GUI Stream Query Algebra Stream Query Algebra Queries are processed by SQuAl operators on the data stream Queries are processed by SQuAl operators on the data stream Some of the operators are filter, map, union, aggregate, join bsort, resample. Some of the operators are filter, map, union, aggregate, join bsort, resample.
AURORA QUERY MODEL b1b2b3 b4 b5b6 b7 b8b9 app QoS spec continuous query view ad-hoc query Connection point data input
AURORA QoS GRAPH TYPES
OPTIMIZATION FilterBSort Filter Union Aggregate Join Aggregate Map Hold pull data Continuous query FilterMap Ad hoc query Join Static storage
OPTIMIZATION Dynamic continuous query optimization Inserting projections Combining boxes Reordering boxes
AURORA RUNTIME ARCHITECTURE Buffer manager Storage Manager Persistent Store Q1Q1 Q2Q2 QmQm Q1Q1 Q2Q2 QnQn Scheduler Load Shedder QoS Monitor Catalog Box Processors σμσμ Router inputsoutputs
SUMMARY Solution approach itself Rethink about everything for the requirements Query model Data flow style query specification Optimization Dynamic runtime optimization QoS specification based resource management
QUESTIONS???