Declarative Support for Sensor Data Cleaning Shawn Jeffery Gustavo Alonso Michael Franklin Wei Hong Jennifer Widom UC Berkeley ETH Zurich UC Berkeley Arch Rock Stanford Corporation University (Intel Research Berkeley) Presented By: Venkatesh (venky) Raghavan & Abhishek Mukherji Disclaimer: Slides adapted / taken from the talk given by S. Jeffery in Pervasive ‘06

Current Approach Application Raw, dirty data Data Cleaning Application Data Cleaning Sensor devices Each application implements its own data cleaning Multiple accesses to a shared resource Each application implements its own data cleaning Multiple accesses to a shared resource

Data Cleaning - Infrastructure Approach Application Cleaning Infrastructure Raw, dirty data Cleaned data Data cleaning built, tested, and deployed once One point of access to sensor devices Data cleaning built, tested, and deployed once One point of access to sensor devices The Cleaning Infrastructure translates raw sensor data to cleaned data; applications are unaffected by the unreliable devices over which they are deployed.

Challenges How to build an infrastructure that supports:  Many types of sensors  Multiple applications  Different environments Two facets to our solution: 1) Pipeline of sensor cleaning tasks 2) Declarative query processing

Temporal and Spatial Granules ESP (Extensible Sensor stream Processing) uses high-level abstractions:  Temporal Granules  Spatial Granules Granules  Define units of time and space inside which the data are expected to be homogeneous Exploits the fact that many applications are not interested in individual readings or devices, but with higher-level data in time and space

Temporal Granules  Sensor devices produce data at a frequent rate  Applications are concerned with data from a larger time period Environment Monitoring application – model micro- climate of redwood tree  Reading required for every 5 minutes.  Solution: windowed processing to group readings

Spatial Granules Reading from devices physically close to each other are expected to be homogeneous Spatial granules defines the unit of space in which this homogeneity is expected to hold.

Sensor Cleaning Pipeline Point Smooth Merge Arbitrate Virtualize Cleaning Data Involves  A set of logically distinct operation  Each operation targets different aspects of the data, from finest (single readings) to coarsest (multiple sensors and various sources)  Uses temporal and spatial characteristics of sensor data

Declarative Query Processing Program stages with declarative queries CQL: continuous query extension to SQL Data stream system as processing engine  Real-time cleaning SELECT S.city, AVG(temp) FROM SOME_STREAM S [RANGE ‘5 seconds’] WHERE S.state = ‘California’ GROUP BY S.city Window Clause

Step 1: Point Operates: Single value of sensor stream. Purpose: Filter individual values  Errant (dirty / faulty) RFID tags  Obvious outliers  Conversion of raw data into tuples Heat Sensors  Output data into voltages. We have to convert that raw data into temperature by looking into calibration of that sensor.

Step 1: Point P P P P P P P P P P P P Point

Step 2: Smoothing Purpose: Interpolates (inserts) lost readings  Temporal interpolation  Outlier detection Method:Window based queries P P P P P P SS P P P P P P SS Temporal Granules Point Smooth

Step 3: Merge Purpose: Spatial interpolation  Example: Within a spatial granule, by computing the average of the readings from different motes and omitting individual readings that are outside of two deviations from the mean. P P P P P P SS M P P P P P P SS M Spatial Granules Point Smooth Merge

Step 3: Merge Outlier mote Average Functioning motes

Step 4: Arbitrate Purpose: Remove  c onflicting readings  de-duplication P P P P P P SS M A P P P P P P SS M Point Smooth Merge Arbitrate

Step 5: Virtualize Purpose: Multi-source integration P P P P P P SS M A V P P P P P P SS M Point Smooth Merge Arbitrate Virtualize

RFID Scenario Point Smooth Merge Arbitrate Virtualize Application arbitrate_input rfid_data smooth_input Query 2 Query 4 Query 3 On Sensor Smooth Each domain needs to modeled

RFID Scenario Fig: Expected OutputFig: Query 2 result using raw RFID Data

Smoothing Difference in Shelf 0 and Shelf 1 is likely due to issues with antenna ports on these particular RFID readers.

Arbitration

Moving Average (Window (w) = 3 time-stamps At t+2, Shelf 0: count(r1) = 2 Shelf 1: count(r1) = 3 tt+1t+2 RFID : r1 NOTE: Window size must be larger than the longest period of dropped reading. But not too large.

Conclusion An infrastructural approach to sensor data cleaning is necessary ESP: a pipelined declarative framework for building such infrastructure Application ESP Raw, dirty data Cleaned data