Fjording the Stream: An Architecture for Queries over Streaming Sensor Data Samuel Madden, Michael J. Franklin University of California, Berkeley Proceedings of the 18th International Conference on Data Engineering (ICDE’02)

Outline Introduction Architecture Main characteristics Experiments Conclusions

Introduction Fjord  also fiord, long narrow inlet of the sea between high cliffs, as in Norway  “Framework in Java for Operators on Remote Data streams” Sensor infrastructure  Cooperators Berkeley Highway Lab (BHL) California Department of Transportation (CalTrans)  Location Bay Area Freeways  Objective Monitoring traffic conditions

Sensor limitations Push-based data  Waiting for queries wastes power Power  Sensors with battery 100mAh CPU: 3.5 hours TRM-1000 radio: 14MB  Tradeoff It is often worth spending many CPU cycles to conserve just a few bytes of radio traffic.

Issues in data stream systems Operators  Aware of the infinite nature of streams  Modified versions of AVERAGE, COUNT, SORT, JOIN hash-join A. Wischut, P.Apers. Dataflow query execution in a parallel main-memory environment. blocking operators (ex: average) specify a subset of the stream for them to operate over Query plan optimization  no mention Architecture

Architecture (1/2) Components  Operators has a set of input queues a set of output queues  Queues has one input operator one output operator  Sensor proxy

Architecture (2/2) Strategy  State based execution model  Rather than placing each pushing operator in its own thread Advantages  Better control over priority  Lower overhead output current state input new state

Main characteristics (1/2) Integrating streaming data with disk-based data  Example Relations between average speeds and traffic incidents  Means Using queues as data sources Combining multiple queries into a single plan  Reason Several queries need data from the same sensors. Duplication wastes bandwidth and power.  Means Using the sensor proxy

Main characteristics (2/2) Intergrating streaming data with disk- based data  Queue pull push putget transitionget input operatoroutput operator

Code snippet

Sensor proxy Functions  Adjust the sample rate of the sensors, based on user demand  Direct the sensor to aggregate samples in predefined ways  Let user queries share the same tuple data

Experiments

Traffic queries

Fjord

Performance Output queues become slower when there are more than a few thousand elements on them.

Scalability

Simulations

Speed, length of a vehicle Speed Length

Sensor parameters

Power consumption Scenario  The sensor 1.reads from it’s A-to-D input 2.transmits the sample 3.sleeps until the next sample period arrives

Power consumption Scenario  Sensors observe when a car passes over them transmit the { t 0, t 2 } or { t 1, t 3 } relay only a few samples per second

Power consumption Scenario  The sensors Only relay a count of the number of vehicles that passed in the previous second

Conclusions Addressing the low level infrastructure issues in a sensor stream query processing via  Fjord combines proxies, non-blocking operators and conventional query plans  Sensor proxies serve as intermediaries between sensors and query plan