MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Big Events Hans-Arno Jacobsen Middleware Systems Research Group MSRG.org

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Big Event Data

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Traditional Big Data Domain vs. Rest of Universe There are other emerging domains with needs similar to Big Data – Smart grids – Smart cities … H.-A. Jacobsen My first message: There are other relevant Big Data domains – beware!

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Smart Grids for Taming The Energy Problem H.-A. Jacobsen

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Relevance of Smart Grids Increasing penetration of variable renewable energy sources like wind and solar et al. Paradigm shift from demand-following supply to supply-following demand Need for new large-scale information system infrastructure to control demand H.-A. Jacobsen

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Distributed Generation, Flexible Loads and Energy Storage Come in big numbers Show unique behavior (users, weather, equipment, …) Have to be monitored and controlled H.-A. Jacobsen  Big event data challenge

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Solar Photovoltaic Power Generation H.-A. Jacobsen High frequency measurements required Several metrics of interest, many spatially distributed measurement points Source: National Oceanic & Atmospheric Administration (U.S.) ~2.3 TB per year and 1k panels

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Use of PEVs as Grid Resource H.-A. Jacobsen High frequency measurements required Important for SG applications: Continuous update of trip destination and energy level at destination Source: Auto21 Project, University of Winnipeg ~ 0.5 TB per year and 1k vehicles

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Electric Power Consumption H.-A. Jacobsen Very high frequency measurements required (e.g., for inferring device on/off events, grid stability, etc.) Several metrics of interest (household electricity meters, single devices, etc.) Source: UCI Machine Learning Repository ~ 27.5 PB per year and 1k homes

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Traditional Big Data Domain vs. Rest of Universe H.-A. Jacobsen My second message: Detecting events in real- time in the sea of Big Data is just as important.

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Towards Big Events Many non-traditional scenarios that require filtering of Big Events at large scales … scenarios that require filtering & storage of events at large scales Filtering & storage of “event streams” Filtering & storage of “event showers” H.-A. Jacobsen

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Event Showers vs. Event Streams Event Showers Partially ordered sets of events No single event schema Events vary in shape and size from one to the next Processing of many event expressions Tends to require support for aggregation Broader model & paradigm (dissemination, matching, coordination) Event Stream Processing Linearly ordered event sequences Schema-based, single schema per stream Stream tuples follow schema More single-expression processing-based Aggregation is a key requirement Focused on processing queries/expressions over event streams H.-A. Jacobsen

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Conclusions Big Events are Big Data in motion Processing Big Data in real-time to detect events of interest is important as well There are other emerging application domains; let us watch out for them H.-A. Jacobsen My final message: Big Data Benchmarking efforts should take this into account.

MIDDLEWARE SYSTEMS RESEARCH GROUP MSRG.ORG Acknowledgements C. Goebel for help with smart grid slides H.-A. Jacobsen