1. An Extensible Test Framework for Microsoft StreamInsight Alex Raizman Asvin Ananthanarayan Anton Kirilov Badrish Chandramouli Mohamed Ali

2. Agenda Introduction to Microsoft StreamInsight Testing Challenges End to End test framework – Intent based testing (Declarative testing) – Event generation – Functional Verification Case study – Top K Conclusions

3. What is Stream Processing? Stream Processing is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. RELATIONALSTREAMING Query Paradigm Ad-hoc queries or requests Continuous standing queries LatencySeconds, hours, daysMilliseconds or less Data RateHundreds of events/secTens of thousands of events/sec or more Temporal semantics OptionalIntegral to streaming queries query report Event alert

4. Events and Event Streams An event is a collection of fields Streaming engine provisioned timestamp fields capture all the different temporal event characteristics Event sources populate timestamp fields A stream is a sequence of events – Possibly infinite Stream characteristics: – Event/data arrival patterns (steady, bursty) – Out of order events: Order of arrival of events does not match the order of their application timestamps – Events with varying lifetimes 4

5. Top 2 rows ORDER BY C1 DESC Top 2 rows order by C1 DESC on a time window of size 10 Relational A join B on A.C1 = B.C1 Streaming A join B on A.C1 = B.C1 Relational Semantics Versus Streaming Semantics C1C2 12 23 45 810 C1C2 17 1215 410 812 A.C1A.C2B.C2 127 4510 8 12 SEC1C2 203012 4023 5045 60810 SEC1C2 152517 30401215 3538410 4570812 SEA.C1A.C2B.C2 2025127 506081012 A A Join B B A B C1C2 12 45 810 1520 23 4050 SEC1C2 1512 2845 712810 183023 25354050 915 20 SEC1C2 1101520 1108 201520 1020810 20304050 203023 C1C2 4050 1520 Out of order event Predicate matches but events do not overlap in time

6. StreamInsight Query Examples Example – GROUP&APPLY, WINDOW: from e3 in MyStream3 group e3 by e3.i into SubStream from win in SubStream.HoppingWindow( TenSeconds) order by win.f).Take(10); Example – GROUP&APPLY, WINDOW: from e3 in MyStream3 group e3 by e3.i into SubStream from win in SubStream.HoppingWindow( TenSeconds) order by win.f).Take(10); Example – JOIN, PROJECT, FILTER: from e1 in MyStream1 join e2 in MyStream2 on e1.ID equals e2.ID where e1.f2 == “foo” select new { e1.f1, e2.f4 }; Example – JOIN, PROJECT, FILTER: from e1 in MyStream1 join e2 in MyStream2 on e1.ID equals e2.ID where e1.f2 == “foo” select new { e1.f1, e2.f4 };

7. Testing Challenges Enable reuse of tests for different product languages – Multiple entry points to the query engine Event generation – Temporal characteristic. – More than payload generation Automated functional validation. – We have become very good at generating tests, but programmatic verification is still a challenge

8. Test Framework Output stream Property Validator Output stream Property Validator Intent Transformer Intent tree (declarative representation) Concrete tree (declarative representation) StreamInsight Query Language Transformer StreamInsight Query Language Transformer Stream Event Generator Stream Insight SQL-equivalent Query Language Transformer SQL-equivalent Query Language Transformer SQL Server SQL Server Output Stream Canonical Results Verifier Output Stream Canonical Results Verifier XML Query SQL constructs Stream events LINQ Query... Test reuse and Intent based testing Event generation Automated functional validation

9. Intent based testing Top K events on a window with an descending rank expression – Must have an operator as input to Top K – Must have a rank expression. Builds an intent tree. Window Rank Expression TopInput Value = K

10. Intent tree to Concrete tree Building out the rest of the tree. – Finalize tree by binding to stream – Extend tree with another operator – Run model to generate subtree Repository of event types Expression builder Concrete tree = test Test suite = Multiple concrete trees Value = 10 HoppingWindow Import.tulong_integral Rank Expression TopImport Window size = 10 seconds Hop Size = 10 seconds Event Stream REPOSITORY EventType = integral Stream Data Count MaxDuration Cti Rate Event Order Window Rank Expression TopInput Value = K

11. Stream Event Generator Two components – Temporal generation – Payload generation Supported temporal facets – event count – out of order – overlap – maximum duration of event – boundary cases – repository of pre defined patterns Payload generation – leveraged existing payload generators Tester specifies as many facets or as little as they want. Output stream Property Validator Output stream Property Validator Intent Transformer Intent tree (declarative representation) Concrete tree (declarative representation) StreamInsight Query Language Transformer StreamInsight Query Language Transformer Stream Event Generator Stream Insight SQL-equivalent Query Language Transformer SQL-equivalent Query Language Transformer SQL Server SQL Server Output Stream Canonical Results Verifier Output Stream Canonical Results Verifier XML Query SQL constructs Stream events LINQ Query...

12. Concrete tree to XML transform generates xml language for the test Concrete tree to LINQ transform generates C# code for the same test Query Language Transformer

13. Test Framework Output stream Property Validator Output stream Property Validator Intent Transformer Intent tree (declarative representation) Concrete tree (declarative representation) StreamInsight Query Language Transformer StreamInsight Query Language Transformer Stream Event Generator Stream Insight SQL-equivalent Query Language Transformer SQL-equivalent Query Language Transformer SQL Server SQL Server Output Stream Canonical Results Verifier Output Stream Canonical Results Verifier XML Query SQL constructs Stream events LINQ Query... Automated functional validation

14. Functional validation using SQL Value = 10 HoppingWindow Import.tulong_integal Rank Expression TopImport Window size = 10 seconds Hop Size = 10 seconds Event Stream REPOSITORY EventType = integral Stream Data Count MaxDuration Cti Rate Event Order

15. Functional validation using SQL SQL input table is populated with the event stream from the generator Equivalent SQL Queries are run over the input table Output of the StreamInsight query is also piped into SQL Server using output adapter Compares SQL output with output from StreamInsight for equality. If comparison fails  BUG!

16. Conclusions Intent based testing allows testers to think about test scenarios leading to richer, more interesting tests. A relational database can work as a test oracle for validating streaming queries. Most of the bugs were due to interesting temporal aspects of the event stream. Not everything can / should be tested using intent based techniques. – Negative tests, boundary cases

17. QUESTIONS ?

18. For More Information StreamInsight main page & download : http://www.microsoft.com/sqlserver/2008/en/us/R2- complex-event.aspx http://www.microsoft.com/sqlserver/2008/en/us/R2- complex-event.aspx StreamInsight blog: http://blogs.msdn.com/streaminsight/ http://blogs.msdn.com/streaminsight/ StreamInsight MSDN documentation: http://msdn.microsoft.com/en- us/library/ee362541(SQL.105).aspx http://msdn.microsoft.com/en- us/library/ee362541(SQL.105).aspx StreamInsight E-clinics on Microsoft e-learning https://www.microsoftelearning.com/eLearning https://www.microsoftelearning.com/eLearning