Network Computing Laboratory HiFi Systems: Network-Centric Query Processing for the Physical World Michael J. Franklin, Shawn R. Jeffrey, et al UC Berkeley TelegraphCQ Team 2 nd CIDR Conf. 2005

Korea Advanced Institute of Science and Technology Table of Contents One line Comment Motivating Scenario HiFi System with CSAVA processing stage Internal Architecture of HiFi Node Critiques New Idea -1,2

Korea Advanced Institute of Science and Technology One line Comment It’s a preliminary work describing the group’s vision to distribute their TelegraphCQ system to a hierarchical network

Korea Advanced Institute of Science and Technology Motivating Scenario – Supply Chain Management “Smart Shelves” continuously monitor item addition and removal. Info is sent back through the supply chain.

Korea Advanced Institute of Science and Technology Hi Fan-In system Ursa-Minor (TinyDB-based) Ursa-Major (TelegraphCQ w/Archiving) Mid-tier Stargate Mid-tier Processing Node

Korea Advanced Institute of Science and Technology Characteristics of HiFi Systems High Fan-In, globally-distributed architecture Large data volumes generated at edges Filtering and cleaning must be done there Successive aggregation as you move inwards Summaries/anomalies continually, details later Strong temporal focus Strong spatial/geographic focus Streaming data and stored data Integration within and across enterprises

Korea Advanced Institute of Science and Technology A View on this example Filtering, Cleaning, Alerts Monitoring, Time-series Data mining (recent history) Archiving (provenance and schema evolution) Geographic Scope local global Several Readers Regional Centers Central Office

Korea Advanced Institute of Science and Technology Headquarters Regional Centers Warehouse Warehouse Doors Receptor High fan-in system levels with associated CSAVA processing stages RFID Clean Remove Anomalies Smooth Interpolate for lost/garbled readings Arbitrate Remove duplicates Validate Correlate with business rules Analyze Tactical decision support

Korea Advanced Institute of Science and Technology Internal Architecture of a HiFi node Metadata Repository Data Stream Processor Cache Manager Data Listener Resource Manager Query Dispatcher Local View Manager Query Placement Service Query Listener Control Manager Data Disseminator Query Planner DSP Manager Archive Manager Logical Query Planner Physical Query Planner HiFi Glue Data Flow Query Flow Control Flow

Korea Advanced Institute of Science and Technology Critiques Strong Point They classify and formulate five distinct data processing stage They develop the prototype system (in VLDB 05) Weak Point Designing MDR is critical but no initial effort is done No new system requirement Solutions are not technically deep

Korea Advanced Institute of Science and Technology New Idea - 1 Data SourceCQ engineWeb Server SP Accel Clients Filtered out By-passing Buffering

Korea Advanced Institute of Science and Technology New Idea – related to SPAccel Designing front-end component (Cache??) Filtering out unwanted input data By-passing data matching query predicates Buffering data for windowed queries (views) or distributed queries Buffering Query Results

Korea Advanced Institute of Science and Technology Issues expected Cache replacement mechanism How to index cached elements What to cache? How much?

Korea Advanced Institute of Science and Technology New Idea -2 processing stream data for OLAP queries OLTPOLAP Users Clerk, IT professionalKnowledge worker Function Day to day operationsdecision support DB design application-orientedsubject-oriented Data current, up-to-datehistorical, summarized detailed, flat relational multidimensional isolatedintegrated, consolidated Usage repetitivead-hoc Access read/write, lots of scans index/hash on prim. key Unit of work short, simple transactioncomplex query #Records accessed tensmillions #Usersthousandshundreds DB size100MB-GB100GB-TB Metrictransaction throughput query throughput/response

Korea Advanced Institute of Science and Technology A Sample Data Cube sum USA Canada Mexico CountryCountry Date Product CD video camera 1Q2Q3Q4Q

Korea Advanced Institute of Science and Technology New Idea - 2 Stream data in terms of OLAP domain OLAP queries are Inherently multidimensional Spans a long time Need data from multiple sources Processing OLAP queries are Memory intensive Computation intensive

Korea Advanced Institute of Science and Technology Naïve Solution Pre-computing popular computation path

Korea Advanced Institute of Science and Technology Supplementary Silde Cleaning CREATE VIEW cleaned_rfid_stream AS ( SELECT receptor_id, tag_id FROM rfid_stream rs WHERE read_strength >= strength_T) Smoothing CREATE VIEW smoothed_rfid_stream AS ( SELECT receptor_id, tag_id FROM cleaned_rfid_stream GROUP BY receptor_id, tag_id HAVING count(*) >= count_T)