John Plummer Technical Specialist Data Platform Microsoft Ltd StreamInsight Complex Event Processing (CEP) Platform

3 What is CEP? 3 Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven Applications Query ParadigmAd-hoc queries or requestsContinuous standing queries LatencySeconds, hours, daysMilliseconds or less Data RateHundreds of events/secTens of thousands of events/sec or more request response Event output stream input stream

4 Example CEP Scenarios 4 Data Stream Stream Data Store & Archive Event Processing Engine Data Stream Asset Specs & Parameters Power, Utilities: Energy consumption Outages Smart grids 100,000 events/sec Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Web Analytics: Click-stream data Online customer behavior Page layout 100,000 events /sec Manufacturing: Sensor on plant floor React through device controllers Aggregated data 10,000 events/sec Threshold queries Event correlation from multiple sources Pattern queries Threshold queries Event correlation from multiple sources Pattern queries Lookup Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds Financial Services: Stock & news feeds Algorithmic trading Patterns over time Super-low latency 100,000 events /sec

5 “Sweet Spot” for Complex Event Processing Relational Database Applications Financial Trading Applications Aggregate Data Rate (Events/sec) higher Months Days Hours Minutes Seconds 100 ms < 1 ms Operational Analytics Applications (e.g., Logistics) Manufacturing Applications Monitoring Applications CEP Target Scenarios Data Warehousing Applications Web Analytics Applications

6 Complex Event Processing Pressure to handle large volumes of complex data in real time The size and frequency of data make it challenging to store for data mining and analysis Growing need to monitor, analyze and act on the data in motion Process large volume of events across multiple data streams in less than a second Manage your business through historical data mining and continuous insights. Built in support for different types of event handling and rich query semantics

7 APPLICATION Input Adapters Output Adapters StreamInsight

8 Events expose different temporal characteristics Point in time events Interval events with fixed duration Interval events with initially unknown duration (Edge) Rich payloads capture all properties of an event t1 t4 t3 t2 t5 Time  Payload/ value  a b c d e Events

9 Event Types Events in Microsoft’s CEP platform use the.NET type system Events are structured and can have multiple fields Fields are typed using the.NET framework types CEP engine provisioned timestamp fields capture all the different temporal event characteristics Event sources populate time stamp fields Timestamps/Met adata Long pumpID String Type String Location Double flow Double pressure ………………

10 Event Streams & Adapters A stream is a possibly infinite sequence of events Insertions of new events Changes to event durations Examples: time series sensor data, log entries Stream characteristics: Event/data arrival patterns Steady rate with end-of-stream indication Examples: files, tables Intermittent, random, or in bursts Example: retail scanners, web, weather telemetry Order of arrival of events does not match the order of their application timestamps Input & Output Design Time for dealing with Event Streams 10

11 Core CEP Query Engine Hosts “standing queries” Operators consume and produce streams Queries are composable Query results are computed incrementally 11 Query instance management: Submit, start, stop Runtime statistics Takeaway: CEP engine does the heavy lifting for you when processing temporal event data CEP Engine Output Adapters Input Adapters Event Standing Queries Event

12 Developer Experience with the CEP Platform CEP development environment Builds on top of.NET Microsoft Visual Studio as IDE Applications written in C# Queries written in LINQ Supports various deployment models for CEP applications Embedded applications Server applications 12 LINQ: var queryFilter = from c in TestEventStream where c.Field1 > 1 select c; LINQ: var queryFilter = from c in TestEventStream where c.Field1 > 1 select c;

13 StreamInsight Demo…

14 StreamInsight – More Information StreamInsight - complex-event.aspx complex-event.aspx Team Blog - Forum - US/streaminsight/threadshttp://social.msdn.microsoft.com/Forums/en- US/streaminsight/threads

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.