LINQ, Take Two Realizing the LINQ to Everything Dream Bart J.F. De Smet Senior Software Development Engineer Microsoft Corporation

Censored 7 years ago Little recent innovation Where’s the cloud?

Why? What? How? Monads

IEnumerable IQueryable IEnumerable IQueryable new[] { 42 } SelectMany IEnumerable SelectMany ( this IEnumerable source, Func > selector) Could there be more? Also see for “Project MinLINQ”

from _ in name from s in _.ToUpper() select s from _ in name from s in _.ToUpper() select s name.SelectMany( _ => _.ToUpper(), s => s) name.SelectMany( _ => _.ToUpper(), s => s) Compiler Null-propagating dot string s = name?.ToUpper(); Null-propagating dot string s = name?.ToUpper(); Syntactic sugar One single library function suffices

demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team Essential LINQ

var src = new Source (); var res = from p in src where p.Price > 100m group p by p.Category; foreach (var p in res) Console.WriteLine(p); Implements IQueryable Implements IQueryable Compiles fine Does it really have to be a runtime check?

var res = from p in src where p.Price > 100m group p by p.Category; var res = src.Where(p => p.Price > 100m).GroupBy(p => p.Category); Syntactic sugar Can be instance methods Source Filtered Projected WhereSelect No GroupBy “edge”

var res = from tweet in twitter where tweet. About From Location == “Bart” From From About Location == “LINQ” About where tweet. About Query “learns” class Twitter { public TwitterByFrom Where(Func filter); // Other filter methods } Recipe Method overloads Type state machine Operator overloads Recipe Method overloads Type state machine Operator overloads “Has a” type From operator overloading class TwitterByFrom { public TwitterByAboutFrom Where(Func filter); // Other filter methods // Fields with current filters } select tweet; Custom syntax trees class TweetAboutFromLoc { public FromString From; public AboutString About; public LocString Location; } class FromString { FilterFrom operator ==( FromString f, string s) } class TweetAboutFromLoc { public FromString From; public AboutString About; public LocString Location; } class FromString { FilterFrom operator ==( FromString f, string s) } class TweetAboutLoc { public AboutString About; public LocString Location; } class AboutString { FilterAbout operator ==( AboutString f, string s) } class TweetAboutLoc { public AboutString About; public LocString Location; } class AboutString { FilterAbout operator ==( AboutString f, string s) }

demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team Query Providers Revisited

* * Source domain Rule Set Patterns with rewrite actions Rule Set Patterns with rewrite actions Target domain BURS Tree Rewriting Engine E.g. LINQ Expression TreesE.g. T-SQL Expression Trees Const(2) Const(3) Mul Inc Currently built at runtime, e.g. once per LINQ provider that gets loaded

x 1 $ Const(x) Leaf node ? : int1  Final states 1  1 $ Any source leaf node with an int constant value can be reduced using rule 1. Final states have a cost (incremental, see further).

x + ab 1 $ Const(x) 2 $ Add(a,b) Leaf node ? : int1  Wildcard a : int2 * + 3  Final states 1  1 $ 3  2 $ Wildcard b doesn’t get its own entry, since it has the same type as wildcard a. This reduces space in the BURS tables. Wildcard b doesn’t get its own entry, since it has the same type as wildcard a. This reduces space in the BURS tables. 2D table for state transition of the + operator. Wildcard matches any subtree that’s assignable to int.

x + ab * ab 1 $ Const(x) 2 $ Add(a,b) 3 $ Mul(a,b) Leaf node ? : int1  Wildcard a : int2 * + 3  * 2 * 4  Final states 1  1 $ 3  2 $ 4  3 $ What’s the meaning of costs? The cost of 3$ is incremental and has to be added to the cost of all the wildcard matches. In this case, a match of the Mul -rule (4  ) costs 3$ + cost a + cost b. What’s the meaning of costs? The cost of 3$ is incremental and has to be added to the cost of all the wildcard matches. In this case, a match of the Mul -rule (4  ) costs 3$ + cost a + cost b.

x + ab * ab + c* ab 1 $ Const(x) 2 $ Add(a,b) 3 $ Mul(a,b) 4 $ AddMul(a,b,c) Leaf node ? : int1  Wildcard a : int2 * + 3  4  5  * 2 * 4  Final states 1  1 $ 3  2 $ 4  3 $ 5  4 $ A pattern forest is kept to map subtrees on already existent states. Here (a * b) matches state 4.

x + ab * ab + c* ab + a1 1 $ Const(x) 2 $ Add(a,b) 3 $ Mul(a,b) 4 $ AddMul(a,b,c) 1 $ Inc(a) Leaf node ? : int1  16 Wildcard a : int2 *  7  4  5  * 2 * 4  Final states 1  1 $ 3  2 $ 4  3 $ 5  4 $ 7  1 $

23 * 1 + Leaf node ? : int1  16 Wildcard a : int2 *  7  4  5  * 2 * 4  Final states 1  1 $ 3  2 $ 4  3 $ 5  4 $ 7  1 $ 5 $ 6 $ Constant leaf node 1 doesn’t contribute cost (part of pattern).

demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team Bottom Up Rewriting

Environment MoveNext Got next? Application OnNext Have next! IEnumerable IEnumerator IEnumerable IEnumerator IObservable IObserver IObservable IObserver Interactive Reactive

Observable Subscribe Observer

Event Stream Processing Interfaces namespace System { public interface IObservable { IDisposable Subscribe(IObserver observer); } public interface IObserver { void OnNext(T value); void OnError(Exception error); void OnCompleted(); }

Fixed (MSIL) Translatable (Expression trees) ToQueryable ToObservable ToEnumerable AsQueryable AsEnumerable AsQbservable AsObservable Pull (interactive) Push (reactive) Concurrency (IScheduler) LINQ to *.* What? Where? How? Worker pools Message loops Threads Distributed Duality Homo-iconic IQbservable LINQ to WMI Events ToQbservable IQueryable LINQ to SQL IEnumerable LINQ to Objects IObservable LINQ to Events

IQbservable - The Dual of IQueryable public interface IQbservable : IObservable { Expression Expression { get; } Type ElementType { get; } IQbservableProvider Provider { get; } } public interface IQbservableProvider { IQbservable CreateQuery (Expression expression); } Extended role for some operators No Execute method Homoiconic naming IQbservable IObservable Homoiconic naming IQbservable IObservable

demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team LINQ to Twitter

Baked in notion of “where”? foreach (var p in res.RemoteOn(new SqlScheduler(“server”))) // Process product Decoupled “what” from “where” Entry-point for the schema Custom schedulers could be very rich (e.g. server farm) var ctx = new NorthwindDataContext(); var res = from product in ctx.Products where product.Price > 100m select product.Name;

Rx.NET RxJS stocks.Where(function (t) { return t.Symbol == “MSFT”; }).Select(function (t) { return t.Quote; }) JSON serializer Observable data source Retargeting to AJAX from ticker in stocks where ticker.Symbol == “MSFT” select ticker.Quote

Query Server Alice F9 F10 chat talk var chat = svc.GetStream (“chat”); var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”); talk.Subscribe(s => ShowMessage(s)); F5 chat Where Select = = msg “Bob” From msg Text Translate chat Where Select = = msg “Bob” From msg Text

Query Server Alice chat talk var chat = svc.GetStream (“chat”); var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”); talk.Subscribe(s => ShowMessage(s)); Bob F9 chat F10 var chat = svc.GetStream(“chat”); Rx.Observable.FromDOMEvent(txt, “Changed”).Select(function (e) { return txt.Text; }).Throttle(.5 /* 500 ms */).Subscribe(function (s) { chat.OnNext(new Message(“Bob”, s)); }); F5 chat Where Select = = msg “Bob” From msg Text Translate chat

Query Server Alice chat talk Bob chat Hello Alice! F5 var chat = svc.GetStream(“chat”); Rx.Observable.FromDOMEvent(txt, “Changed”).Select(function (e) { return txt.Text; }).Throttle(.5 /* 500 ms */).Subscribe(function (s) { chat.OnNext(new Message(“Bob”, s)); }); Bob says: Hello Alice! Bonjour Alice! var chat = svc.GetStream (“chat”); var talk = from msg in chat where msg.From == “Bob” select Translate(msg.Text, “fr”); talk.Subscribe(s => ShowMessage(s)); F5 Bob says: Bonjour Alice! Where Select = = msg “Bob” From msg Text Translate “Bonjour Alice!” From: “Bob” Text: “Hello Alice!” From: “Bob” Text: “Hello Alice!” From: “Bob” Text: “Hello Alice!” From: “Bob” Text: “Hello Alice!” Translate “Bonjour Alice!” “Hello Alice!”

demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team Expression Tree Serialization

announcing Now with support for hosting expression tree based queries (enumerable and observable) StreamInsight v2.1

Model[ Decisions[Reals, SA, VZ], Goals[ Minimize[20 * SA + 15 * VZ] ], Constraints[ C1 -> 0.3 * SA * VZ >= 2000, C2 -> 0.4 * SA * VZ >= 1500, C3 -> 0.2 * SA * VZ >= 500, C4 -> SA <= 9000, C5 -> VZ <= 6000, C6 -> SA >= 0, C7 -> VZ >= 0 ] from m in ctx.CreateModel(new { vz = default(double), sa = default(double) }) where 0.3 * m.sa * m.vz >= 2000 && 0.4 * m.sa * m.vz >= 1500 && 0.2 * m.sa * m.vz >= 500 where 0 <= m.sa && m.sa <= 9000 && 0 <= m.vz && m.vz <= 6000 orderby 20 * m.sa + 15 * m.vz select m To compute call Solve Symbolic, non-persisted Cost / barrel Max # barrels Max # barrels Min # barrels Min # barrels Refinement %

from costSA in GetPriceMonitor("SA") from costVZ in GetPriceMonitor("VZ") from m in ctx.CreateModel(new { vz = default(double), sa = default(double) }) where 0.3 * m.sa * m.vz >= 2000 && 0.4 * m.sa * m.vz >= 1500 && 0.2 * m.sa * m.vz >= 500 where 0 <= m.sa && m.sa <= 9000 && 0 <= m.vz && m.vz <= 6000 orderby costSA * m.sa + costVZ * m.vz select m Observable data sources SelectMany Parameters to decision Subscribe here!

demo Bart J.F. De Smet Senior Software Development Engineer Cloud Programmability Team LINQ to Constraints

DEV413 – Curing your Event Processing Blues using Reactive Extensions (Rx) Find Me Later The Week At The Ask The Experts

Connect. Share. Discuss. Learning Microsoft Certification & Training Resources TechNet Resources for IT Professionals Resources for Developers

Evaluations Submit your evals online