SQL Server 2008 R2 StreamInsight Complex Event Processing Event Stream Processing

Who Am I? SQL Server MVP SQL Server Consultant Joint author on Wrox Professional SSIS book and Specialise in Moving (twitter)

Agenda Use Cases & Challenges Formulating Declarative Queries Windows in Time Event Flow Debugging Demos

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries request response Event output stream input stream What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries LatencySeconds, hours, daysMilliseconds or less request response Event output stream input stream What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database ApplicationsEvent-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries LatencySeconds, hours, daysMilliseconds or less Data RateHundreds of events/sec> Tens of thousands of events/sec request response Event output stream input stream What is CEP?

Who might need CEP Fraud Detection Real-Time Trade Risking Algorithmic Trading/Betting Meter throughputs – Oil, Gas, Water, Electricity – Use to drive alarms, alerts etc

Query Logic Event sources Event targets Devices, Sensors Web servers Event stores & Databases Stock ticker, news feeds Event stores & Databases Pagers & Monitoring devices KPI Dashboards, SharePoint UI Trading stations StreamInsight Application at Runtime.NET C# LINQ StreamInsight Application Development Input Adapters Output Adapters StreamInsight Engine Query Logic StreamInsight Platform

Query Expressiveness Selection of events (filter) Calculations on the payload (project) Correlation of streams (join) Stream partitioning (group and apply) Aggregation (sum, count, …) over event windows Ranking over event windows (topK)

Query Expressiveness var result = from e in inputStream select new { id = e.id, W = (double)e.intW / 10 }; var result = from e in inputStream select new { id = e.id, W = (double)e.intW / 10 }; Projection

Query Expressiveness var result = from e in inputStream where e.id > 3 select new { id = e.id, W = (double)e.intW / 10 }; var result = from e in inputStream where e.id > 3 select new { id = e.id, W = (double)e.intW / 10 }; Projection Filter

Query Expressiveness var result = from eLeft in inputStream1 join eRight in inputStream2 on eLeft.id equals eRight.id select new { id = eLeft.id, diff = eLeft.W - eRight.w }; var result = from eLeft in inputStream1 join eRight in inputStream2 on eLeft.id equals eRight.id select new { id = eLeft.id, diff = eLeft.W - eRight.w }; Projection Filter Correlation (Join)

Query Expressiveness var result = from win in inputStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { avg = win.Avg(e => e.W) }; var result = from win in inputStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { avg = win.Avg(e => e.W) }; Projection Filter Correlation (Join) Aggregation over windows

Projection Filter Correlation (Join) Aggregation over windows Group and Aggregate Query Expressiveness var result = from e in inputStream group e by e.id into eachGroup from win in eachGroup.TumblingWindow( TimeSpan.FromSeconds(10)) select new { eachGroup.Key, avg = win.Avg(e => e.W) }; var result = from e in inputStream group e by e.id into eachGroup from win in eachGroup.TumblingWindow( TimeSpan.FromSeconds(10)) select new { eachGroup.Key, avg = win.Avg(e => e.W) };

Windowing Hopping – Tumbling Snapshot Count

Time Windows Hopping Window

Time Windows Tumbling Window

Snapshot Windows

Debugger Has its own debugger You cannot live without it (believe me) Set breakpoints Filter A wonderful tool

DEMOS Tour Trace Reader Live Trace File Reader Query Demo (Joining) Debugger Tool (How it will save you)