Advanced Microsoft SQL Server 2008 R2 StreamInsight PDC09-SVR08 Advanced Microsoft SQL Server 2008 R2 StreamInsight Beysim Sezgin (Architect) Roman Schindlauer (Program Manager) Microsoft Corporation © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

SQL Server Connections Agenda Use Cases & Challenges Formulating Declarative Queries Windows in Time Implementing Adapters Architecture Extensibility Event Flow Debugging Updates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL

SQL Server Connections What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Updates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL

SQL Server Connections What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database Applications Event-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries request response Event output stream input stream Updates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL

SQL Server Connections What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database Applications Event-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries Latency Seconds, hours, days Milliseconds or less request response Event output stream input stream Updates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL

SQL Server Connections What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database Applications Event-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries Latency Seconds, hours, days Milliseconds or less Data Rate Hundreds of events/sec > Tens of thousands of events/sec request response Event output stream input stream Updates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL

StreamInsight Platform SQL Server Connections StreamInsight Platform .NET C# LINQ StreamInsight Application Development StreamInsight Application at Runtime Event sources Event targets Input Adapters Output Adapters StreamInsight Engine Devices, Sensors Pagers & Monitoring devices Standing Queries KPI Dashboards, SharePoint UI 1 Web servers Query Logic Query Logic 4 2 3 Trading stations Event stores & Databases Query Logic Stock ticker, news feeds Event stores & Databases Updates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL

NBC Sunday Night Football: live streaming through Silverlight Rich client experience, multiple camera angles Needed: track, monitor, analyze user behavior, based on silverlight Media analytics

Use Case: NBC Sunday Night Football 1 Telemetry Receiver 4 StreamInsight Listener Adapter GeoTag and group by region SQL Adapter PerfCounter Adapter 2 Count total events Count session starts Count active sessions 3

Complex Aggregations/ Use Case: Data Center Power Consumption Visualize Process Information Complex Aggregations/ Correlations Central time series archive ETW Input Adapter Query Query 1 2 Power Meter Input Adapter Query 3

SQL Server Connections Challenges How do I … detect interesting patterns? reason about temporal semantics? correlate data? aggregate data? avoid writing custom imperative code? create a runtime environment for continuous and event-driven processing? As a developer, I need a platform! Updates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL

SQL Server Connections Query Expressiveness Selection of events (filter) Calculations on the payload (project) Correlation of streams (join) Stream partitioning (group and apply) Aggregation (sum, count, …) over event windows Ranking over event windows (topK) Updates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL

Query Expressiveness Projection var result = from e in inputStream select new { id = e.id, W = (double)e.intW / 10 };

Query Expressiveness Projection Filter var result = from e in inputStream where e.id > 3 select new { id = e.id, W = (double)e.intW / 10 };

Query Expressiveness Projection Filter Correlation (Join) var result = from eLeft in inputStream1 join eRight in inputStream2 on eLeft.id equals eRight.id select new { id = eLeft.id, diff = eLeft.W - eRight.w };

Query Expressiveness Projection Filter Correlation (Join) Aggregation over windows var result = from win in inputStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { avg = win.Avg(e => e.W) };

Query Expressiveness Projection Filter Correlation (Join) Aggregation over windows Group and Aggregate var result = from e in inputStream group e by e.id into eachGroup from win in eachGroup.TumblingWindow( TimeSpan.FromSeconds(10)) select new { eachGroup.Key, avg = win.Avg(e => e.W) };

Time Windows Tumbling Window Time

Time Windows Hopping Window Time

demo LINQ Queries 11/24/2018 1:10 AM © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Adapters Import data into StreamInsight platform Adapter characteristics: Pull vs Push Out-of-order vs. In-Order Events Adapter API Observable model wraps adapter framework

Adapter Factory public class MyFactory : ITypedInputAdapterFactory<MyConfig> { public InputAdapterBase Create<Payload>(MyConfig conf, EventShape ev) return new MyAdapter(conf); } public void Dispose() { ... }

Adapter API public class MyAdapter : TypedPointInputAdapter<MyPayload> { public MyAdapter(MyConfig conf) { ... } public override void Start() public override void Resume() }

Stream Liveliness ? ? Time All processing is done according to application time, not event arrival time Events can be enqueued with out-of- order timestamps How does the engine know when to flush its output – how long does it need to keep windows open? CTI: Current Time Increment Asserts that nothing up to its timestamp will be inserted Latency and liveliness of results is driven by CTIs Adapters can declare an AdvanceTimePolicy Issue CTIs with given frequency and time delay Bigger delay: longer grace period, slower output Smaller delay: lively output, less out-of-order allowed

Development Steps 1 2 3 4 .Net Development LINQ StreamInsight IL User App Engine 4 Adapter Runtime query Adapter

StreamInsight Architecture Compilation Runtime Parser Algebrizer Streams Events Compiler Expression Service Stream Manager Event Manager Metadata Type System Execution Operators Scheduler Stream OS Resource Governor Synch. Primitives Buffer Manager CLR Hosting

Brand Transformation Presentation Query Compilation Multiple Syntactic surfaces over a single algebra, e.g., LINQ, CQL, etc. Compile time type checking and type safe code generation for minimal runtime impact. Leverage CLR Code generator, JIT support Type System Tools and Libraries (LINQ Expressions, IDE, etc.) Multiple Deployment Environment with different form factors, e.g. SmartPhone, PDA, PCs. Compilation Parser Algebrizer Compiler Expression Service Metadata Type System

Runtime – Events, Streams, Scheduler Brand Transformation Presentation Runtime – Events, Streams, Scheduler JIT code generation for field references, expression evaluation Event manager is implemented as a combination of managed and native code in order to minimize overhead and ensure predictable performance. Runtime Streams Events Stream Manager Event Manager Execution Operators Scheduler Cooperative Scheduler (less context switch, better SLA) Operators are affinitized to a Scheduler. Periodic checks and migration for load balancing DataFlow Architecture: Reduced coupling and pipeline parallelism

Runtime – Execution Operators Efficient implementation of operators that perform incremental evaluation as each event is processed. GroupAndApply Operator: Enables parallelism for both scale-up (multi-core) and scale-out(cluster). Groups are dynamically instantiated and torn down based upon the data. Large numbers of groups can be simultaneously active. Machine2 Machine1 ~ MAP Apply Union X,Y,Z ABC Group A,B,C BBB AA CC XX ZZ YYY XYZ ~ REDUCE

SQL Server Connections Extensibility Need for domain-specific extensions Integrate with functionality available in existing libraries User-defined operators, functions, aggregates Code written in .NET, deployed as .NET assembly Window-based semantics for UDOs, UDAs Receive a set of events Produce a value / set of events Updates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL

SQL Server Connections Supportability Diagnostic interface Retrieve statistical information at runtime Part of the object model API Manage through Powershell Debugger tool Retrieve live diagnostics Record and replay queries Analyze event processing along operators Updates will be available at http://www.devconnections.com/updates/LasVegas_Fall09/SQL

demo Debugger Tool 11/24/2018 1:10 AM © 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Conclusion CEP Platform & API Event-triggered, fast Computation API for Adapters, Queries, Applications Declarative LINQ Flexible Adapter API Extensible Supportability

Please Evaluate! Sign in to microsoftpdc.com Follow the evaluation link for the session PDC will donate $1 for each person who completes an evaluation to the local Boys and Girls Club

11/24/2018 1:10 AM © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11/24/2018 1:10 AM © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.