1. Advanced Microsoft SQL Server 2008 R2 StreamInsight
PDC09-SVR08
Advanced Microsoft SQL Server 2008 R2 StreamInsight
Beysim Sezgin (Architect)
Roman Schindlauer (Program Manager)
Microsoft Corporation
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. SQL Server Connections
Agenda
Use Cases & Challenges
Formulating Declarative Queries
Windows in Time
Implementing Adapters
Architecture
Extensibility
Event Flow Debugging
Updates will be available at

3. SQL Server Connections
What is CEP?
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Updates will be available at

4. SQL Server Connections
What is CEP?
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications
Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
request
response
Event
output stream
input stream
Updates will be available at

5. SQL Server Connections
What is CEP?
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications
Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency
Seconds, hours, days
Milliseconds or less
request
response
Event
output stream
input stream
Updates will be available at

6. SQL Server Connections
What is CEP?
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications
Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency
Seconds, hours, days
Milliseconds or less
Data Rate
Hundreds of events/sec
> Tens of thousands of events/sec
request
response
Event
output stream
input stream
Updates will be available at

7. StreamInsight Platform
SQL Server Connections
StreamInsight Platform
.NET C#
LINQ
StreamInsight Application Development
StreamInsight Application at Runtime
Event sources
Event targets
Input
Adapters
Output
Adapters
StreamInsight Engine
Devices, Sensors
Pagers &
Monitoring devices
Standing Queries
KPI Dashboards, SharePoint UI 1
Web servers
Query Logic
Query Logic 4 2 3
Trading stations
Event stores & Databases
Query Logic
Stock ticker, news feeds
Event stores & Databases
Updates will be available at

8. NBC Sunday Night Football: live streaming through Silverlight
Rich client experience, multiple camera angles
Needed: track, monitor, analyze user behavior, based on silverlight Media analytics

9. Use Case: NBC Sunday Night Football1
Telemetry Receiver 4
StreamInsight
Listener Adapter
GeoTag and group by region
SQL Adapter
PerfCounter Adapter 2
Count total events
Count session starts
Count active sessions 3

10. Complex Aggregations/
Use Case: Data Center
Power Consumption
Visualize
Process Information
Complex Aggregations/
Correlations
Central
time series
archive
ETW
Input Adapter
Query
Query 1 2
Power Meter
Input Adapter
Query 3

11. SQL Server Connections
Challenges How do I …
detect interesting patterns?
reason about temporal semantics?
correlate data?
aggregate data?
avoid writing custom imperative code?
create a runtime environment for continuous and event-driven processing?
As a developer, I need a platform!
Updates will be available at

12. SQL Server Connections
Query Expressiveness
Selection of events (filter)
Calculations on the payload (project)
Correlation of streams (join)
Stream partitioning (group and apply)
Aggregation (sum, count, …) over event windows
Ranking over event windows (topK)
Updates will be available at

13. Query Expressiveness Projection var result = from e in inputStream
select new { id = e.id,
W = (double)e.intW / 10 };

14. Query Expressiveness Projection Filter
var result = from e in inputStream
where e.id > 3
select new { id = e.id,
W = (double)e.intW / 10 };

15. Query Expressiveness Projection Filter Correlation (Join)
var result = from eLeft in inputStream1
join eRight in inputStream2
on eLeft.id equals eRight.id
select new { id = eLeft.id,
diff = eLeft.W - eRight.w };

16. Query Expressiveness Projection Filter Correlation (Join)
Aggregation over windows
var result = from win in inputStream.TumblingWindow(
TimeSpan.FromSeconds(10))
select new { avg = win.Avg(e => e.W) };

17. Query Expressiveness Projection Filter Correlation (Join)
Aggregation over windows
Group and Aggregate
var result = from e in inputStream
group e by e.id into eachGroup
from win in eachGroup.TumblingWindow(
TimeSpan.FromSeconds(10))
select new { eachGroup.Key,
avg = win.Avg(e => e.W) };

18. Time Windows
Tumbling Window
Time

19. Time Windows
Hopping Window
Time

20. demo LINQ Queries 11/24/2018 1:10 AM
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Adapters Import data into StreamInsight platform
Adapter characteristics:
Pull vs Push
Out-of-order vs. In-Order Events
Adapter API
Observable model
wraps adapter framework

22. Adapter Factory
public class MyFactory : ITypedInputAdapterFactory<MyConfig> {
public InputAdapterBase
Create<Payload>(MyConfig conf, EventShape ev)
return new MyAdapter(conf); }
public void Dispose()
{ ... }

23. Adapter API
public class MyAdapter : TypedPointInputAdapter<MyPayload> {
public MyAdapter(MyConfig conf)
{ ... }
public override void Start()
public override void Resume() }

24. Stream Liveliness ? ?
Time
All processing is done according to application time, not event arrival time
Events can be enqueued with out-of- order timestamps
How does the engine know when to flush its output – how long does it need to keep windows open?
CTI: Current Time Increment Asserts that nothing up to its timestamp will be inserted
Latency and liveliness of results is driven by CTIs
Adapters can declare an AdvanceTimePolicy
Issue CTIs with given frequency and time delay
Bigger delay: longer grace period, slower output
Smaller delay: lively output, less out-of-order allowed

25. Development Steps 1 2 3 4 .Net Development LINQ StreamInsight IL
User App
Engine 4
Adapter
Runtime query
Adapter

26. StreamInsight Architecture
Compilation
Runtime
Parser
Algebrizer
Streams
Events
Compiler
Expression Service
Stream Manager
Event Manager
Metadata
Type System
Execution Operators
Scheduler
Stream OS
Resource Governor
Synch. Primitives
Buffer Manager
CLR Hosting

27. Brand Transformation Presentation
Query Compilation
Multiple Syntactic surfaces over a single algebra, e.g., LINQ, CQL, etc.
Compile time type checking and type safe code generation for minimal runtime impact.
Leverage CLR
Code generator, JIT support
Type System
Tools and Libraries (LINQ Expressions, IDE, etc.)
Multiple Deployment Environment with different form factors, e.g. SmartPhone, PDA, PCs.
Compilation
Parser
Algebrizer
Compiler
Expression Service
Metadata
Type System

28. Runtime – Events, Streams, Scheduler
Brand Transformation Presentation
Runtime – Events, Streams, Scheduler
JIT code generation for field references, expression evaluation
Event manager is implemented as a combination of managed and native code in order to minimize overhead and ensure predictable performance.
Runtime
Streams
Events
Stream Manager
Event Manager
Execution Operators
Scheduler
Cooperative Scheduler (less context switch, better SLA)
Operators are affinitized to a Scheduler. Periodic checks and migration for load balancing
DataFlow Architecture: Reduced coupling and pipeline parallelism

29. Runtime – Execution Operators
Efficient implementation of operators that perform incremental evaluation as each event is processed.
GroupAndApply Operator:
Enables parallelism for both scale-up (multi-core) and scale-out(cluster).
Groups are dynamically instantiated and torn down based upon the data. Large numbers of groups can be simultaneously active.
Machine2
Machine1
~ MAP
Apply
Union X,Y,Z
ABC
Group A,B,C
BBB AA CC XX ZZ
YYY
XYZ
~ REDUCE

30. SQL Server Connections
Extensibility
Need for domain-specific extensions
Integrate with functionality available in existing libraries
User-defined operators, functions, aggregates
Code written in .NET, deployed as .NET assembly
Window-based semantics for UDOs, UDAs
Receive a set of events
Produce a value / set of events
Updates will be available at

31. SQL Server Connections
Supportability
Diagnostic interface
Retrieve statistical information at runtime
Part of the object model API
Manage through Powershell
Debugger tool
Retrieve live diagnostics
Record and replay queries
Analyze event processing along operators
Updates will be available at

32. demo Debugger Tool 11/24/2018 1:10 AM
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

33. Conclusion CEP Platform & API Event-triggered, fast Computation
API for Adapters, Queries, Applications
Declarative LINQ
Flexible Adapter API
Extensible
Supportability

34. Please Evaluate! Sign in to microsoftpdc.com
Follow the evaluation link for the session
PDC will donate $1 for each person who completes an evaluation to the local Boys and Girls Club

35. 11/24/2018 1:10 AM
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

36. 11/24/2018 1:10 AM
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.