5. 5 Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries LatencySeconds, hours, daysMilliseconds or less Data RateHundreds of events/secTens of thousands of events/sec or more request response Event output stream input stream

6. Relational Database Applications Financial trading Applications 6 Aggregate Data Rate (Events/sec.) Latency 010100100010000100000~1million Months Days hours Minutes Seconds 100 ms < 1ms Operational Analytics Applications, e.g., Logistics, etc. Manufacturing Applications Monitoring Applications CEP Target Scenarios Data Warehousing Applications Web Analytics Applications

7. 7 Data Stream Stream Data Store & Archive Event Processing Engine Data Stream Asset Specs & Parameters Power, Utilities: Energy consumption Outages Smart grids 100,000 events/sec Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance Web Analytics: Click-stream data Online customer behavior Page layout 100,000 events /sec Manufacturing: Sensor on plant floor React through device controllers Aggregated data 10,000 events/sec Threshold queries Event correlation from multiple sources Pattern queries Threshold queries Event correlation from multiple sources Pattern queries Lookup Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds Financial Services: Stock & news feeds Algorithmic trading Patterns over time Super-low latency 100,000 events /sec

8. Industry trends Data acquisition costs are negligible Raw storage costs are small and continue to decrease Processing costs are non-negligible Data loading costs continue to be significant Manage business via KPI-triggered actions Mine historical data Devise new KPIs Monitor KPIs Record raw data (history) 8 CEP advantage Process data incrementally, i.e., while it is in flight Avoid loading while still doing the processing you want Seamless querying for monitoring, managing and mining

9. 9 Data Sources, Operations, Assets, Feeds, Sensors, Devices Monitor & Record Monitor & Record Operational Data Store & Archive CEP Engine f(x) g(y) CEP Engine f(x) f'(x) g(y) h(x,y) History Deploy Results f'(x) h(x,y) Manage & Benefit Manage & Benefit Mine & Design Mine & Design Input Data Streams Output Data Streams

10. Data Sources Aggregation & Correlation 10 CEP CEP for lightweight processing and filtering CEP for aggregation and correlation of in- flight events CEP for complex analytics including historical data Event processing engines are deployed at multiple places on different scales At the edge – close to the data source In the mid-tier – consolidate related data sources, In the data center – historical archive, mining, large scale correlation. Devices Sensors Web servers Feeds CEP Complex Analytics & Mining

12. t1 t4 t3 t2 t5 Time  Payload/ value  a b c d e

13. Timestamps/ Metadata Long pumpID String Type String Location Double flow Double pressure ………………

14. 14

15. 15

16. 16 Query instance management: Submit, start, stop Runtime statistics Takeaway: CEP engine does the heavy lifting for you when processing temporal event data CEP Engine Output Adapters Input Adapters Event Standing Queries Event

19. LINQ Example – GROUP&APPLY, WINDOW: from e3 in MyStream3 group e3 by e3.i into SubStreams from s4 in SubStreams from e4 in s4.SlidingWindow(FiveMinutes,ThreeSeconds) select new { pl = new MyNewPayload(e4.i, e4.f)}; LINQ Example – GROUP&APPLY, WINDOW: from e3 in MyStream3 group e3 by e3.i into SubStreams from s4 in SubStreams from e4 in s4.SlidingWindow(FiveMinutes,ThreeSeconds) select new { pl = new MyNewPayload(e4.i, e4.f)}; LINQ Example – JOIN, PROJECT, FILTER: from e1 in MyStream1 join e2 in MyStream2 e1.ID equals e2.ID where e1.f2 = “foo” select new { e1.f1, e2.f4 }; LINQ Example – JOIN, PROJECT, FILTER: from e1 in MyStream1 join e2 in MyStream2 e1.ID equals e2.ID where e1.f2 = “foo” select new { e1.f1, e2.f4 }; Join Filter Project Grouping Window

21. 21 LINQ: var queryFilter = from c in TestEventStream where c.Field1 > 1 select c; LINQ: var queryFilter = from c in TestEventStream where c.Field1 > 1 select c;

23. Data Sources Analytics & Queries QT1 CEP Engine Output Adapters Input Adapters Event Standing Queries Event AT2 AT1 Q1 Q1’ AT3 Q1’’

24. Query Template Query Template Output Adapter Join Proj. Type Input Adapter Input Adapter Type Input Adapter Input Adapter Type

25. 25 CEP Engine Output Adapters Input Adapters Event Standing Queries Event sources Event targets Event C_I D C_NAM E C_ZI P Event Static reference data CEP Application Development Development experience with.NET, C#, LINQ and Visual Studio 2008 CEP platform from Microsoft to build event- driven applications Event-driven applications are fundamentally different from traditional database applications: queries are continuous, consume and produce streams, and compute results incrementally Flexible adapter SDK with high performance to connect to different event sources and sinks The CEP platform does the heavy lifting for you to deal with temporal characteristics of event stream data

26. CEP Engine Reference data Custom CEP Application Scenario 1: Custom CEP Application Dev Scenario 2: Embed CEP in Application ISV Application with CEP Engine CEP Engine Reference data Scenario 4: Operational Intelligence w/ CEP Madison ETL Pipeline with CEP engine CEP Engine CEP KPIs KPI mining Scenario 3: CEP Enabled Device Device with Embedded CEP Engine CEP Engine.NET, C#, LINQ

27. time July 2009 CTP2 TAP program CTP2 TAP program 2010 General Availability CTP3 TAP continues CTP3 TAP continues Late 2009

28. © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.