Adaptive Sampling in Distributed Streaming Environment Ankur Jain 2/4/03

Outline Motivation for adaptive sampling Related work Overview of Approach

Motivation for Adaptive Sampling More useful in sensor networks and environmental monitoring applications The rate at which a sensor makes readings Affects the resource usage in the network Affects the resource usage in the network Presents a picture of the trend in the change in values of the sensor readings Presents a picture of the trend in the change in values of the sensor readings Naïve Solution --- Over Sampling Disadvantage -- Increased Resource cost Disadvantage -- Increased Resource cost CPU – processing unnecessary data CPU – processing unnecessary data Network Bandwidth – transmitting unnecessary data Network Bandwidth – transmitting unnecessary data Power Usage – measuring device, transmission, computation Power Usage – measuring device, transmission, computation

Related Work The Design of an Acquisitional Query Processor For Sensor Networks (Sigmod 2003) Samuel Madden, Michael J. Franklin, and Joseph M. Hellerstein Wei Hong Sampling rate adaptivity in context of power consumption and network bandwidth. Monitors network contention and reduces sampling rate if contention rises. Sampling rate does not depend on input characteristics. Sampling rate reduces/increases by a fixed constant each time. No server side mediation in adjusting sampling rate.

Related Work Adaptive Sampling Mechanisms in Sensor Networks (LCS 2003) A Djafari Marbini, L. E. Sacks,University College London (LCS 2003) Use of control mechanism to manipulate the sampling rate Sampled data is compared to a model representing the environment The error value produced as a result of comparison is used to adaptively increase/decrease the sampling rate. Does not take distributed sensor environment into consideration. No server side mediation, sampling rate not dependent on the sampling rate of other sensors and network bandwidth availability.

Overview Distributed sources sample data at an initially allotted sampling rate. Each source runs an estimator (Kalman Filter) and computes the innovation sequence (error values) over a fixed size sliding window. Each source is allotted a sampling window within which its sampling rate can oscillate. Each source can manipulate the sampling rate based on error within the sampling window. If the desired sampling is out of the sampling window, a new window is requested from the server. The server allocates new sampling window according to network contention, query characteristics and individual data source weight.

v2v2 v1v1 V n-1 VnVn CQ Evaluator User Update of critical values when estimation error is large Data Sources User registers queries Results are updated to the user continuously values from estimation Main Stream Proc. Adjust sampling rate Based on estimation error Request new sampling window Allocate new Sampling Window

The Discrete Kalman Filter (DKF) State Model Measurement Model Innovation Sequence

Sliding Window for Innovation Sequence Each source maintains a sliding window of error values computed each time a measurement is made. The weight of a value decays exponentially with time. If δ i is the fractional error at time i and n is the size of the sliding window, average error ae is : If In is the innovation matrix and z the measurement matrix in the Kalman Filter

Adjusting the Sampling Rate Objective Let the sampling rate decay gradually if good estimations are recorded for a longer period of time i.e. low error value aggregates over the sliding window Let the sampling rate decay gradually if good estimations are recorded for a longer period of time i.e. low error value aggregates over the sliding window Raise the sampling rate by a larger factor if the estimation is bad. (i.e. the errors are large) Raise the sampling rate by a larger factor if the estimation is bad. (i.e. the errors are large) Motivation – Approach biased towards sampling interesting events at a higher frequency, at the cost of lower sampling rate for uninteresting events (nodes where the Kalman Filter state model is performing better). Motivation – Approach biased towards sampling interesting events at a higher frequency, at the cost of lower sampling rate for uninteresting events (nodes where the Kalman Filter state model is performing better). If S cur is the current sampling rate, ae the aggregated error from the innovation sequence and λ the acceptable fractional error within the window, the new sampling rate S new is

Requesting new Sampling Windows If the new sampling rate is greater than the allotted sampling window Set the current sampling rate to the maximum in the current window. Set the current sampling rate to the maximum in the current window. Request new window from main server. Request new window from main server. If the sampling rate is less that the minimum allowed in the sampling window Set the sampling rate to the minimum allowed in the current window. Set the sampling rate to the minimum allowed in the current window. Request new window from the server. Request new window from the server. We do not set the sampling rate to the new sampling rate because the streaming source might have higher weight at the main server requiring it to stream above a threshold sampling rate. We do not set the sampling rate to the new sampling rate because the streaming source might have higher weight at the main server requiring it to stream above a threshold sampling rate.

If B is the maximum bandwidth (sum of all sampling rates in the distributed system), and w is the width of sampling window then initial sampling rate or all the sources (total n) can be W i is the weight is streaming source i from the query processor. Req i is the fraction of sampling width requested by source i over to the total width requested by all sources and Den i is the fraction sampling width denied Req i is the fraction of sampling width requested by source i over to the total width requested by all sources and Den i is the fraction sampling width denied Allocating new sampling windows

Allocating new sampling widths L i is the fraction of sampling rate decrease request from source i The score for each score i is Requests are served in non-increasing order of scores. If there is no available bandwidth for a request it is denied. (an active sources is not deprived of its latest allocated bandwidth)