2 Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries LatencySeconds, hours, daysMilliseconds or less Data RateHundreds of events/secTens of thousands of events/sec or more request response Event output stream input stream

Relational Database Applications Financial trading Applications Aggregate Data Rate (Events/sec.) Latency ~1million Months Days hours Minutes Seconds 100 ms < 1ms Operational Analytics Applications, e.g., Logistics, etc. Manufacturing Applications Monitoring Applications CEP Target Scenarios Data Warehousing Applications Web Analytics Applications 3

4 Data Stream Stream Data Store & Archive Event Processing Engine Data Stream Asset Specs & Parameters Power, Utilities: Energy consumption Outages Smart grids 100,000 events/sec Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Web Analytics: Click-stream data Online customer behavior Page layout 100,000 events /sec Manufacturing: Sensor on plant floor React through device controllers Aggregated data 10,000 events/sec Threshold queries Event correlation from multiple sources Pattern queries Threshold queries Event correlation from multiple sources Pattern queries Lookup Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds Financial Services: Stock & news feeds Algorithmic trading Patterns over time Super-low latency 100,000 events /sec

Industry trends Data acquisition costs are negligible Raw storage costs are small and continue to decrease Processing costs are non-negligible Data loading costs continue to be significant Manage business via KPI-triggered actions Mine historical data Devise new KPIs Monitor KPIs Record raw data (history) CEP advantage Process data incrementally, i.e., while it is in flight Avoid loading while still doing the processing you want Seamless querying for monitoring, managing and mining 5

6 Data Sources, Operations, Assets, Feeds, Sensors, Devices Monitor & Record Monitor & Record Operational Data Store & Archive CEP Engine f(x) g(y) CEP Engine f(x) f'(x) g(y) h(x,y) History Deploy Results f'(x) h(x,y) Manage & Benefit Manage & Benefit Mine & Design Mine & Design Input Data Streams Output Data Streams

7 CEP Engine Output Adapters Input Adapters Event Standing Queries Event sources Event targets Devices, Sensors Web servers Event stores & Databases Stock tickers & News feeds Event C_IDC_NAM E C_ZI P Event stores & Databases Pagers & Monitoring devices KPI Dashboards, SharePoint UI Trading stations Event CEP Application at Runtime Static reference data IDE.NET C# LINQ CEP Application Development

t1 t4 t3 t2 t5 Time  Payload/ value  a b c d e

Timestamps /Metadata Long pumpID String Type String Location Double flow Double pressure ………………

10

LINQ Example – GROUP&APPLY, WINDOW: from e3 in MyStream3 group e3 by e3.i into SubStream from win in SubStream.HoppingWindow( FiveMinutes,ThreeSeconds) select new { i = SubStream.Key, a = win.Avg(e => e.f) }; LINQ Example – GROUP&APPLY, WINDOW: from e3 in MyStream3 group e3 by e3.i into SubStream from win in SubStream.HoppingWindow( FiveMinutes,ThreeSeconds) select new { i = SubStream.Key, a = win.Avg(e => e.f) }; LINQ Example – JOIN, PROJECT, FILTER: from e1 in MyStream1 join e2 in MyStream2 on e1.ID equals e2.ID where e1.f2 == “foo” select new { e1.f1, e2.f4 }; LINQ Example – JOIN, PROJECT, FILTER: from e1 in MyStream1 join e2 in MyStream2 on e1.ID equals e2.ID where e1.f2 == “foo” select new { e1.f1, e2.f4 }; Join Filter Project Grouping Window Project & Aggregate

demo

var outputStream = from win in input.TumblingWindow(TimeSpan.FromSeconds(10)) select new { v = win.TimeWeightedAverage(e => e.dVal) }; var outputStream = from win in input.TumblingWindow(TimeSpan.FromSeconds(10)) select new { v = win.TimeWeightedAverage(e => e.dVal) };

public class TimeWeightedAverage : CepTimeSensitiveAggregate { public override double GenerateOutput( IEnumerable > events, WindowDescriptor wd) { double avg = 0; foreach (IntervalEvent e in events) { avg += e.Payload * (e.EndTime - e.StartTime).Ticks; } return avg / (wd.EndTime - wd.StartTime).Ticks; } public class TimeWeightedAverage : CepTimeSensitiveAggregate { public override double GenerateOutput( IEnumerable > events, WindowDescriptor wd) { double avg = 0; foreach (IntervalEvent e in events) { avg += e.Payload * (e.EndTime - e.StartTime).Ticks; } return avg / (wd.EndTime - wd.StartTime).Ticks; }

Data Sources Analytics & Queries QT1 CEP Engine Output Adapters Input Adapters Event Standing Queries Event AT2 AT1 Q1 Q1’ AT3 Q1’’

Query Template Query Template Output Adapter Join Proj. Type Input Adapter Input Adapter Type Input Adapter Input Adapter Type

Data Sources Aggregation & Correlation 21 CEP CEP for lightweight processing and filtering CEP for aggregation and correlation of in-flight events CEP for complex analytics including historical data Event processing engines are deployed at multiple places on different scales At the edge – close to the data source In the mid-tier – consolidate related data sources, In the data center – historical archive, mining, large scale correlation. Devices Sensors Web servers Feeds CEP Complex Analytics & Mining

Programming/ Deployment Implicit ServerExplicit ServerIObservable Abstracts from CEP server, adapters and binding Full control over CEP server, adapters and binding Use reactive framework interfaces In-processSupported Separate processes N/ASupported Easy to use No query template re-use Full control Query template re-use Easiest to use Data sources need IObservable interface

demo

StreamInsight Output Adapters Input Adapters Asset Class TickerRegionCountryExchangeSUM Volume SUM Bid SUM Ask StockMSFTNAUSNASDAQ100 StockIBMNAUSNASDAQ200 Deriv.IBMEMEACHSWIX10 StreamInsight Output Adapters Input Adapters StreamInsight Output Adapters Input Adapters TickerExchangeSUM Volume MSFTNASDAQ100 ExchangeSUM Bid SUM Ask NASDAQ100 NASDAQ200

25 CEP Engine Output Adapters Input Adapters Event Standing Queries Event sources Event targets Event C_IDC_NAM E C_ZI P Event Static reference data CEP Application Development Development experience with.NET, C#, LINQ and Visual Studio 2008 CEP platform from Microsoft to build event- driven applications Event-driven applications are fundamentally different from traditional database applications: queries are continuous, consume and produce streams, and compute results incrementally Flexible adapter SDK with high performance to connect to different event sources and sinks The CEP platform does the heavy lifting for you to deal with temporal characteristics of event stream data

CEP Engine Reference data Custom CEP Application Scenario 1: Custom CEP Application Dev Scenario 2: Embed CEP in Application ISV Application with CEP Engine CEP Engine Reference data Scenario 4: Operational Intelligence w/ CEP Madison ETL Pipeline with CEP engine CEP Engine CEP KPIs KPI mining Scenario 3: CEP Enabled Device Device with Embedded CEP Engine CEP Engine.NET, C#, LINQ

time Aug 2009 August CTP TAP program August CTP TAP program TechEd Announcement May Release November CTP TAP continues November CTP TAP continues Nov 2009

question & answer

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.