Computer Engineering and Networks Laboratory Visualizing Large Sensor Network Data Sets in Space and Time with Vizzly Matthias Keller, Jan Beutel, Olga Saukh, Lothar Thiele SenseApp 2012, 22/10/2012

Swiss Federal Institute of Technology 2 Matthias Keller, SenseApp 2012, 22/10/2012 Low-power wireless sensor networks for permafrost monitoring Mobile urban air quality sensing GPS loggers Coin-sized temperature loggers

Swiss Federal Institute of Technology 3 Matthias Keller, SenseApp 2012, 22/10/2012 More deployments, new sensors 1 st deployment 2 nd deployment 3 rd deployment 4 th deployment New sensor: GPS Four extra trams 1 st OpenSense tram ~2,500 sensing channels

Swiss Federal Institute of Technology 4 Matthias Keller, SenseApp 2012, 22/10/2012 Visualizing All Signal Dynamics Interpolation of once loaded dataDesired functionality Short-term behavior is not visible!

Swiss Federal Institute of Technology 5 Matthias Keller, SenseApp 2012, 22/10/2012 Research Questions and Challenges Very large, multi-year data sets  How can we make all levels of detail accessible?  What are suited data structures for efficient data access?  How can we use precious resources, e.g., RAM, efficiently? Unrestricted sensing modalities  Event-based or periodic sampling  (Mean) sampling rates can vary between seconds and days System integration  Visualization service should fit into existing landscape

Swiss Federal Institute of Technology 6 Matthias Keller, SenseApp 2012, 22/10/2012 Overview Related Work  Vizzly  System architecture  Location-preserving temporal aggregation  Hierarchical cache  Output generation Case Study 1 2 3

Swiss Federal Institute of Technology 7 Matthias Keller, SenseApp 2012, 22/10/2012 Related Work Data interfaces of other projects  SensorScope, GlacsWeb, da-sense, Geigermap,... Data streaming platforms, APIs  Pachube/Cosm, Google Fusion Tables, Microsoft SensorMap, … Optimized database/data processing systems  RasDaMan, tsdb, sMAP, … Evaluated solutions did not support (at least one):  the anticipated data volume  the visualization of multi-year time series in one view  the visualization of all signal dynamics

Swiss Federal Institute of Technology 8 Matthias Keller, SenseApp 2012, 22/10/2012 Overview Related Work  Vizzly  System architecture  Location-preserving temporal aggregation  Hierarchical cache  Output generation Case Study 1 2 3

Swiss Federal Institute of Technology 9 Matthias Keller, SenseApp 2012, 22/10/2012 Vizzly Overview Goal: Fast browsing of multi-year data at all levels of detail Cache layer + web service + front-end library Provides time series and map data widgets Includes a management interface and health sensors Freely available as open source software

Swiss Federal Institute of Technology 10 Matthias Keller, SenseApp 2012, 22/10/2012 System Architecture Recording of structured data with time and (optional) location information Processes, stores and publishes incoming data Aggregated data is stored in data structures that maintain temporal and spatial locality Users only need a web browser for interactively browsing through large data sets

Swiss Federal Institute of Technology 11 Matthias Keller, SenseApp 2012, 22/10/2012 Client-Server Communication User interface is only loaded once Vizzly servers returns (time, value) and (location, value) tuples, respectively, when requests parameters change Easy integration into arbitrary web pages How is the returned CSV data generated?

Swiss Federal Institute of Technology 12 Matthias Keller, SenseApp 2012, 22/10/2012 Back-end Design Challenges  Users can select data on temporal and spatial criteria  Arbitrary combinations of temporal and spatial aggregation levels cannot be cached efficiently  Location-preserving temporal aggregation scheme  Ideally, all data should also be ready in the Vizzly cache  Memory cache is the fastest, but also the smallest  Continuous updating of a hierarchical cache  Whenever possible, unaggregated data should be returned  Size of unaggregated data is not known beforehand  Sampling rate estimation

Swiss Federal Institute of Technology 13 Matthias Keller, SenseApp 2012, 22/10/2012 Location-preserving Temporal Aggregation Idea: Perform temporal aggregation before caching, but postpone spatial aggregation to request processing Down-sampling of time information to target resolution: 1 2 Aggregation function is applied to samples of equal down-sampled time and equal location information: 3 Resulting tuple is put into cache

Swiss Federal Institute of Technology 14 Matthias Keller, SenseApp 2012, 22/10/2012 Spatial aggregation is only done when serving a concrete request Resulting tuples are sent to client Spatial Aggregation of Pre-Aggregated Data Data of reduced temporal resolution is loaded from cache 4 5 Location information is reduced to matching target resolution: 7 Aggregation function is applied to samples of equal (reduced) time and location information: 6

Swiss Federal Institute of Technology 15 Matthias Keller, SenseApp 2012, 22/10/2012 Back-end Design Challenges  Users can select data on temporal and spatial criteria  Arbitrary combinations of temporal and spatial aggregation levels cannot be cached efficiently  Location-preserving temporal aggregation scheme  Ideally, all data should also be ready in the Vizzly cache  Memory cache is the fastest, but also the smallest  Continuous updating of a hierarchical cache  Whenever possible, unaggregated data should be returned  Size of unaggregated data is not known beforehand  Sampling rate estimation

Swiss Federal Institute of Technology 16 Matthias Keller, SenseApp 2012, 22/10/2012 Hierarchical Cache Memory DBMS, e.g., MySQL Original data store fastest slowest Order of accesses until signal data is found

Swiss Federal Institute of Technology 17 Matthias Keller, SenseApp 2012, 22/10/2012 Idea: While time information can be linearized easily, doing so avoids searching for stored data and facilitates further down-sampling in the temporal domain Memory Cache Data without location information: Data with location information: Multiple entries for one timestamp

Swiss Federal Institute of Technology 18 Matthias Keller, SenseApp 2012, 22/10/2012 Vizzly “learns” interesting signals from requests received Learned signals are continuously polled for new data Strategies for hierarchical caches  Balancing by signal popularity  Distribution by aggregation level  … Cache Updating

Swiss Federal Institute of Technology 19 Matthias Keller, SenseApp 2012, 22/10/2012 Back-end Design Challenges  Users can select data on temporal and spatial criteria  Arbitrary combinations of temporal and spatial aggregation levels cannot be cached efficiently  Location-preserving temporal aggregation scheme  Ideally, all data should also be ready in the Vizzly cache  Memory cache is the fastest, but also the smallest  Continuous updating of a hierarchical cache  Whenever possible, unaggregated data should be returned  Size of unaggregated data is not known beforehand  Sampling rate estimation

Swiss Federal Institute of Technology 20 Matthias Keller, SenseApp 2012, 22/10/2012 Output generation for single view containing (n+m) signals: 1. For each signal, decide if unaggregated data points can be displayed (estimated mean sampling rate x time period) 2. Apply spatial filtering/aggregation when data with location information is loaded Request Processing in the Back-end

Swiss Federal Institute of Technology 21 Matthias Keller, SenseApp 2012, 22/10/2012 Overview Related Work  Vizzly  System architecture  Location-preserving temporal aggregation  Hierarchical cache  Output generation Case Study 1 2 3

Swiss Federal Institute of Technology 22 Matthias Keller, SenseApp 2012, 22/10/2012 Data Fetch Performance GSN: Unaggregated input  2,100 sensing channels, ~550 million data points MySQL DB: 1 st aggregation level, 4 minutes resolution  300 million aggregated data points, ~15 GB data Memory: 2 nd aggregation level, 16 minutes resolution  100 million aggregated data points, ~800 MB data 99 th percentiles:  Live agg.: 4 msec  Memory: 5 msec  MySQL: 690 msec  GSN: 6.9 sec

Swiss Federal Institute of Technology 23 Matthias Keller, SenseApp 2012, 22/10/2012 Conclusions New middleware for the interactive browsing of large sensor network data sets Provided time series and map data widgets can be easily integrated into existing web pages Data fetch are dramatically reduced by caching Vizzly has proven its usefulness in more than 1 year of operation in a production environment Vizzly is available as open-source software Demo: Project home: