1. 1 Fjording The Stream An Architecture for Queries over Streaming Sensor Data Samuel Madden, Michael Franklin UC Berkeley

2. Feb 28, 2002 Fjording ICDE2 Introduction Telegraph Sensor Query Processing Architecture Telegraph Sensor Query Processing Architecture Fjords Fjords Enable push and pull in query plans Enable push and pull in query plans Operators for streaming data Operators for streaming data Sensor proxy Sensor proxy Sensor - Query Mediator Sensor - Query Mediator Web Sensors Us er Proxy

3. Feb 28, 2002 Fjording ICDE3 Roadmap Background Background Sensors Sensors Requirements Requirements Traffic Scenario Traffic Scenario Fjords Fjords Continuous Queries Continuous Queries Stream-sensitive operators Stream-sensitive operators Sensor Proxies Sensor Proxies Querying a Sensor Network Querying a Sensor Network Results Results Graphs Graphs Related Work Related Work

4. Feb 28, 2002 Fjording ICDE4 Roadmap Background Background Sensors Sensors Requirements Requirements Traffic Scenario Traffic Scenario Fjords Fjords Continuous Queries Continuous Queries Stream-sensitive operators Stream-sensitive operators Sensor Proxies Sensor Proxies Querying a Sensor Network Querying a Sensor Network Results Results Graphs Graphs Related Work Related Work

5. Feb 28, 2002 Fjording ICDE5 Sensor Challenges Battery Powered Battery Powered 2AA Cells (2800 mAh; 1.01x10 4 J), Coin Cell (100mAh) 2AA Cells (2800 mAh; 1.01x10 4 J), Coin Cell (100mAh) Communication Dominates Power Cost Communication Dominates Power Cost Can be exhausted - tens to hundreds of MBs of data / sensor Can be exhausted - tens to hundreds of MBs of data / sensor Wireless Wireless High loss rates (20% @ 5meters, typical) High loss rates (20% @ 5meters, typical) Low bandwith (10kbps) Low bandwith (10kbps) Near Real Time Near Real Time Streaming Data Streaming Data Pushed at (user defined) regular intervals Pushed at (user defined) regular intervals TinyOS “Mote”

6. Feb 28, 2002 Fjording ICDE6 Requirements of Sensor Query Processing Power Sensitivity Power Sensitivity Tolerance to unbounded streams of data Tolerance to unbounded streams of data Tolerance to push Tolerance to push Tolerance to intermittent, lossy connections Tolerance to intermittent, lossy connections Tolerance to failed sensors Tolerance to failed sensors

7. Feb 28, 2002 Fjording ICDE7 Traffic Scenario CA Department of Transportation (CalTrans) has sensors all over bay area freeways CA Department of Transportation (CalTrans) has sensors all over bay area freeways Inductive loop sensors give speed, flow, vehicle size Inductive loop sensors give speed, flow, vehicle size Motes could collect this data cheaply Motes could collect this data cheaply Many possible queries Many possible queries Commuters want to find congestion Commuters want to find congestion California Highway Patrol (CHP) wants to find accidents California Highway Patrol (CHP) wants to find accidents

8. Feb 28, 2002 Fjording ICDE8 Roadmap Background Background Sensors Sensors Requirements Requirements Traffic Scenario Traffic Scenario Fjords Fjords Continuous Queries Continuous Queries Stream-sensitive operators Stream-sensitive operators Sensor Proxies Sensor Proxies Querying a Sensor Network Querying a Sensor Network Results Results Graphs Graphs Related Work Related Work

9. Feb 28, 2002 Fjording ICDE9 Fjords Query plan implementation Query plan implementation Useful for streams and distributed environments Useful for streams and distributed environments Combine push (streaming) data and pull (static) data Combine push (streaming) data and pull (static) data E.g. traffic sensors with CHP accident reports E.g. traffic sensors with CHP accident reports

10. Feb 28, 2002 Fjording ICDE10 Push vs. Pull Problem: Need an API to combine push & pull Problem: Need an API to combine push & pull Operators (e.g. join) data-direction agnostic Operators (e.g. join) data-direction agnostic Push vs. pull implemented in queues (connectors) Push vs. pull implemented in queues (connectors) Contrast with Contrast with Iterator model Iterator model Exchange operator Exchange operator Allows arbitrary combinations of push and pull Allows arbitrary combinations of push and pull

11. Feb 28, 2002 Fjording ICDE11 Pull Example Operator Queue q;… Tuple process() { Tuple t = q.get(), outt = null; Tuple t = q.get(), outt = null; If (t != null) { If (t != null) { } else { … do something else … } } else { … do something else … } return outt; return outt;} Pull Queue Operator parent, child; Tuple get() { Tuple t = null; Tuple t = null; while (t == null) { while (t == null) { t = child.process(); t = child.process(); } return t; return t;} get( ) Pull Connection  Sca n process() Notice: Iterator semantics by making get() blocking Get() can return null Process() can return null

12. Feb 28, 2002 Fjording ICDE12 Push Example Operator Queue q;… Tuple process() { Tuple t = q.get(), outt = null; Tuple t = q.get(), outt = null; If (t != null) { If (t != null) { } else { … do something else … } } else { … do something else … } return outt; return outt;}Thread while(true) { Tuple t = op.process(); Tuple t = op.process(); if (t != null) op.outq.enqueue(t); if (t != null) op.outq.enqueue(t);} Push Queue Operator parent, child; Vector v = new Vector(); Tuple get() { if (v.size() > 0) return v.removeFirst(); else return null; } Tuple enqueue(Tuple t) { v.put(t);} Push Connection  Sca n get( )

13. Feb 28, 2002 Fjording ICDE13 Fjord Example PushPush PushPush Pu ll

14. Feb 28, 2002 Fjording ICDE14 Roadmap Background Background Sensors Sensors Requirements Requirements Traffic Scenario Traffic Scenario Fjords Fjords Continuous Queries Continuous Queries Stream-sensitive operators Stream-sensitive operators Sensor Proxies Sensor Proxies Querying a Sensor Network Querying a Sensor Network Results Results Graphs Graphs Related Work Related Work

15. Feb 28, 2002 Fjording ICDE15 Continuous Queries Given user queries over current sensor data Given user queries over current sensor data Expect that many queries will be over the same data sources (e.g. traffic sensors) Expect that many queries will be over the same data sources (e.g. traffic sensors) Queries over current data always looking at same tuples Queries over current data always looking at same tuples Those queries can share Those queries can share Current tuples Current tuples Work (e.g. selections) Work (e.g. selections) Sharing reduces messages, thereby power! Sharing reduces messages, thereby power! A new query can be “folded” into an existing query A new query can be “folded” into an existing query Use old instances of scans and selections Use old instances of scans and selections

16. Feb 28, 2002 Fjording ICDE16 Roadmap Background Background Sensors Sensors Requirements Requirements Traffic Scenario Traffic Scenario Fjords Fjords Continuous Queries Continuous Queries Stream-sensitive operators Stream-sensitive operators Sensor Proxies Sensor Proxies Querying a Sensor Network Querying a Sensor Network Results Results Graphs Graphs Related Work Related Work

17. Feb 28, 2002 Fjording ICDE17 Relational Operators And Streams Selection and Projection Apply Naturally Selection and Projection Apply Naturally No Blocking Operators No Blocking Operators Sorts and aggregates over the entire stream Sorts and aggregates over the entire stream Nested loops and sort-merge join Nested loops and sort-merge join Windowed Operators Windowed Operators Sorts, aggregates, etc. Sorts, aggregates, etc. Online, Interactive QP Techniques Online, Interactive QP Techniques In memory symmetric hash join In memory symmetric hash join Alternatives: ripple-join, Xjoin, etc. Alternatives: ripple-join, Xjoin, etc. Partial Results Partial Results

18. Feb 28, 2002 Fjording ICDE18 Roadmap Background Background Sensors Sensors Requirements Requirements Traffic Scenario Traffic Scenario Fjords Fjords Continuous Queries Continuous Queries Stream-sensitive operators Stream-sensitive operators Sensor Proxies Sensor Proxies Querying a Sensor Network Querying a Sensor Network Results Results Graphs Graphs Related Work Related Work

19. Feb 28, 2002 Fjording ICDE19 Query Processor Sensor Proxies Mediate between Sensors and Fjords Mediate between Sensors and Fjords Push operators out to sensors Push operators out to sensors Hide query processing, knowledge of multiple queries from sensors Hide query processing, knowledge of multiple queries from sensors Hide details of sensors from query processor Hide details of sensors from query processor Enable power-sensitivity Enable power-sensitivity Operators Query Parsed Queries [ sources, ops ] [ fields, filters, aggregates, rates ] [ tuples ] Registration

20. Feb 28, 2002 Fjording ICDE20 What runs where? (Multi-query) Distributed Optimization Challenge: (Multi-query) Distributed Optimization Challenge: Given set of operators, proxy must choose: Given set of operators, proxy must choose: Run on sensor or, Run on sensor or, Run on local query processor Run on local query processor Running on sensors saves power Running on sensors saves power Simple computations cheaper than messages Simple computations cheaper than messages Selection, aggregation reduce communication cost Selection, aggregation reduce communication cost Sensors have limited resources Sensors have limited resources All queries can’t run in all sensors simultaneously All queries can’t run in all sensors simultaneously Limited state precludes big joins or lots of groups Limited state precludes big joins or lots of groups Queries share operators Queries share operators Operators vary in selectivity Operators vary in selectivity

21. Feb 28, 2002 Fjording ICDE21 Roadmap Background Background Sensors Sensors Requirements Requirements Traffic Scenario Traffic Scenario Fjords Fjords Continuous Queries Continuous Queries Stream-sensitive operators Stream-sensitive operators Sensor Proxies Sensor Proxies Querying a Sensor Network Querying a Sensor Network Results Results Graphs Graphs Related Work Related Work

22. Feb 28, 2002 Fjording ICDE22 Building a Query How to translate a declarative query into a Fjord? How to translate a declarative query into a Fjord? Just like traditional query processing, except: Just like traditional query processing, except: Branches originating in sensors connected by push connectors Branches originating in sensors connected by push connectors Sensor proxy handles scans, selections over sensors Sensor proxy handles scans, selections over sensors Proxy delivers tuples from sensors Proxy delivers tuples from sensors Proxy pushes-down selections transparently Proxy pushes-down selections transparently Output of join is push if one or both inputs is push Output of join is push if one or both inputs is push Join carefully chosen Join carefully chosen accident = true Pull: Data Request Data Request Data Index NL Join Push Speed < 30

23. Feb 28, 2002 Fjording ICDE23 Roadmap Background Background Sensors Sensors Requirements Requirements Traffic Scenario Traffic Scenario Fjords Fjords Continuous Queries Continuous Queries Stream-sensitive operators Stream-sensitive operators Sensor Proxies Sensor Proxies Querying a Sensor Network Querying a Sensor Network Results Results Graphs Graphs Related Work Related Work

24. Feb 28, 2002 Fjording ICDE24 Query & Fjord Simple Test Query: Simple Test Query: SELECT AVG(s.speed, w) FROM sensors AS s WHERE s.loc in { userLocs } Loc  { userLocs } BHL Server Telegraph Server Averag e

25. Feb 28, 2002 Fjording ICDE25 Multiple Queries in a Fjord Loc  { userLocs } BHL Server Telegraph Server Loc  { userLocs }

26. Feb 28, 2002 Fjording ICDE26 Fjord Performance (In Telegraph)

27. Feb 28, 2002 Fjording ICDE27 Sensor Proxy for Traffic Measure benefit of pushing computation into sensors; in this case, vehicle identification. Measure benefit of pushing computation into sensors; in this case, vehicle identification. Simple “aggregation” dramatically reduces power costs Simple “aggregation” dramatically reduces power costs w w w t 0 : Vehicle CoversS u t 1 S u and S d t 2 : Vehicle Covers S d t 3 Neither Sensor

28. Feb 28, 2002 Fjording ICDE28 Pushing Aggregates Saves Power Atmel (TinyOS CPU) Simulator 100 samples / sec 5 vehicles / sec 7x power savings

29. Feb 28, 2002 Fjording ICDE29 Roadmap Background Background Sensors Sensors Requirements Requirements Traffic Scenario Traffic Scenario Fjords Fjords Continuous Queries Continuous Queries Stream-sensitive operators Stream-sensitive operators Sensor Proxies Sensor Proxies Querying a Sensor Network Querying a Sensor Network Results Results Graphs Graphs Related Work Related Work

30. Feb 28, 2002 Fjording ICDE30 Related Work Cougar Cougar Interactive & Adaptive Query Processors Interactive & Adaptive Query Processors Tukwila, Xjoin, Eddy, CONTROL Tukwila, Xjoin, Eddy, CONTROL Continuous Query Processors Continuous Query Processors NiagaraCQ, Xfilter, CACQ NiagaraCQ, Xfilter, CACQ Directed Diffusion Directed Diffusion Volcano / Exchange Operator Volcano / Exchange Operator Temporal Databases Temporal Databases

31. Feb 28, 2002 Fjording ICDE31 Conclusions & Future Work Required for sensor query processing: Required for sensor query processing: Fjords : API for combining push & pull Fjords : API for combining push & pull Sensor Proxy: Mediator between sensors and QP Sensor Proxy: Mediator between sensors and QP Streaming Operators Streaming Operators Big benefit from pushing selections, aggregates into network Big benefit from pushing selections, aggregates into network Combining multiple queries is a win Combining multiple queries is a win Extensions & Future Work: Extensions & Future Work: Multi-query optimization & adaptivity (CACQ, SIGMOD 2002) Multi-query optimization & adaptivity (CACQ, SIGMOD 2002) Push down selections & aggregates (Submitted to VLDB 2002) Push down selections & aggregates (Submitted to VLDB 2002) Sensor proxy policies Sensor proxy policies