1 LinQ Introduction

Outline Goals of LinQ Anatomy of a LinQ query More expression examples LinQ to Objects LinQ to XML LinQ to SQL 2

Goals of LinQ Integrate data queries into.Net languages Before C# 3.0 you could use delegates, extension methods, anonyous methods and Visitor Pattern to make something similar to LinQ. But the syntax is messy and key points, eg. selection criteria, are not easy to read. 3

Goals of LinQ Provide a standardized way to query data Challenges: Different data types Different data representations (xml, sql, objects) Data organization Hierarchical – xml (and object) Relational – sql Before LinQ you should use different api’s for accessing databases, objects and xml LinQ provides one single way (nearly) to access it all LinQ works on collections that implements IEnumerable (the.Net language must therefore support generics) 4

Anatomy of a LinQ query An example: 5 string[] characters = { "Donald", "Mickey", "Goofy", "Minnie", "Daisy", "Scrooge" }; IEnumerable query = from c in characters where c.StartsWith("M")!=true orderby c descending select c; foreach(string s in query) Console.WriteLine(s); string[] characters = { "Donald", "Mickey", "Goofy", "Minnie", "Daisy", "Scrooge" }; IEnumerable query = from c in characters where c.StartsWith("M")!=true orderby c descending select c; foreach(string s in query) Console.WriteLine(s); The collection is here a simple string array But the same query will run for more complex objects, SQL, XML etc.

Demo With objects After demo, note the following: Intellisense Static type checking 6

Query Expressions Types of expressions Filtering e.g. Where Projections e.g. Select Joining e.g. Join Partitioning e.g Skip and Take Ordering e.g OrderBy Aggregation e.g. Count and Sum 7

Providers The same expresions works on different kinds of data This is done by accessing a provider A LinQ provider is a gateway to a querable type. There are several builtin providers Objects SQL XML Active Directory PLINQ (Parallel processing) Etc. And many custom providers too: LinQ to Amazon LinQ to Twitter Etc. 8

How are people in Aalborg ? Twitter demo Get LinqToTwitter here: 9 var twitterCtx = new TwitterContext();.. var queryResults = from search in twitterCtx.Search where search.Type == SearchType.Search && search.Attitude == Attitude.Positive && search.GeoCode == " , ,25km" select search; foreach (SearchEntry entry in srch.Results) Console.WriteLine(entry.Text); var twitterCtx = new TwitterContext();.. var queryResults = from search in twitterCtx.Search where search.Type == SearchType.Search && search.Attitude == Attitude.Positive && search.GeoCode == " , ,25km" select search; foreach (SearchEntry entry in srch.Results) Console.WriteLine(entry.Text);

Deferred Execution Normally the query is not executed before the result is needed This is when only lazy operators (where, orderby...) are used. When busy operators are used, the query is executed immediately (count, average) 10 var adults = from p in personList where p.Age > 18 orderby p.Age select (p.FirstName + " " + p.LastName); Console.WriteLine(adults.Count()); personList.Add(new Person { FirstName = “Ib", LastName = “Madsen", Age = 35 }); foreach (var p in adults) { Console.WriteLine(p.ToString()); } var adults = from p in personList where p.Age > 18 orderby p.Age select (p.FirstName + " " + p.LastName); Console.WriteLine(adults.Count()); personList.Add(new Person { FirstName = “Ib", LastName = “Madsen", Age = 35 }); foreach (var p in adults) { Console.WriteLine(p.ToString()); }

Composed Queries A composed query is a query that uses another query. In behind LinQ will make new query that is optimized for the given data store (objects, sql, xml...) 11 var adults = from p in personList where p.Age > 18 orderby p.Age select (p.FirstName + " " + p.LastName);..... var query = from p in adults where p.StartsWith("B") select p; foreach (var p in query) Console.WriteLine(p); var adults = from p in personList where p.Age > 18 orderby p.Age select (p.FirstName + " " + p.LastName);..... var query = from p in adults where p.StartsWith("B") select p; foreach (var p in query) Console.WriteLine(p);

Encapsulate Query It is not possible directly to return an anonymous type And it wouldn’t be nice either Therefore it not is possible to return a query if is declared as var The nice way here is to declare the query as an IEnumerable and return that. If the query is a join or a projection etc. then make a class that maps the output from ‘select’ and return a collection of objects of that class 12

Collections of objects We already seen how to access a collection of objects this is called LinQ to objects LinQ to objects is a good alternative to foreach and other iterations 13

Custom providers And we have seen use of a custom provider to access a webservice. 14 If it is a plain webservice that returns a collection, then we could have accessed that with LinQ to objects

In the next part, we will see how to access XML and SQLServer. 15

LinQ to XML Uses the System.Xml.Linq namespace Is somewhat different from other xml api’s The XElement class is the key class. When instanizing a XElement you can generate the whole document in the constructor 16 XElement doc = new XElement("Inventory", new XElement("Car", new XAttribute("ID","1000"), new XElement("Color", "Red"), new XElement("Make", "Ford")) ); XElement doc = new XElement("Inventory", new XElement("Car", new XAttribute("ID","1000"), new XElement("Color", "Red"), new XElement("Make", "Ford")) ); Red Ford

Use LinQ to generate XML 17 XElement personDoc = new XElement("People", from c in personList orderby c.LastName select new XElement("Person", new XAttribute("Age", c.Age), new XElement("FirstName", c.FirstName), new XElement("LastName", c.LastName) )); XElement personDoc = new XElement("People", from c in personList orderby c.LastName select new XElement("Person", new XAttribute("Age", c.Age), new XElement("FirstName", c.FirstName), new XElement("LastName", c.LastName) )); Caroline Bendtsen Bjarne Hansen...

Use LinQ to search in XML 18 XElement doc = MakeXElementFromList(); var query = from p in doc.Elements("Person") where Convert.ToInt32(p.Attribute("Age").Value) < 40 select p; foreach (var p in query) Console.WriteLine(p.Value); XElement doc = MakeXElementFromList(); var query = from p in doc.Elements("Person") where Convert.ToInt32(p.Attribute("Age").Value) < 40 select p; foreach (var p in query) Console.WriteLine(p.Value);

LinQ to SQL LinQ accesses the sql db through a datacontext class The class can be created with a wizard in VisualStudio, where you select which tables to access Only SQLServer is supported from Microsoft But dbms vendors like Oracle are also providing support for LinQ. (haven’t tested it myself). An (better?) alternative is to use the Entity Framework 19

Use the wizard to create the DataContext class Add a new item to the project using the ”LINQ to SQL classes” template. Select the database and the tables that shall be available 20

Select the tables Note that wizard knows the carnalities 21

Using LinQ to SQL Select orders from a certain customer 22 NorhwindDataContext dc = new NorhwindDataContext(); var orders = from o in dc.Orders where o.CustomerID == "ALFKI" orderby o.OrderDate select o.OrderID; foreach (var o in orders) Console.WriteLine(o); NorhwindDataContext dc = new NorhwindDataContext(); var orders = from o in dc.Orders where o.CustomerID == "ALFKI" orderby o.OrderDate select o.OrderID; foreach (var o in orders) Console.WriteLine(o); SELECT [t0].[OrderID] FROM [dbo].[Orders] AS [t0] WHERE [t0].[CustomerID] ORDER BY [t0].[OrderDate] SELECT [t0].[OrderID] FROM [dbo].[Orders] AS [t0] WHERE [t0].[CustomerID] ORDER BY [t0].[OrderDate]

A join Write products that the customer has brought 23 var orders = from o in dc.Order_Details join p in dc.Products on o.ProductID equals p.ProductID where o.Order.CustomerID == "ALFKI" orderby o.Order.OrderDate select new { product = p.ProductName, orderDate = o.Order.OrderDate }; foreach (var p in orders) Console.WriteLine("{0:dd-MM-yyyy}: {1}", (DateTime)p.orderDate, p.product); var orders = from o in dc.Order_Details join p in dc.Products on o.ProductID equals p.ProductID where o.Order.CustomerID == "ALFKI" orderby o.Order.OrderDate select new { product = p.ProductName, orderDate = o.Order.OrderDate }; foreach (var p in orders) Console.WriteLine("{0:dd-MM-yyyy}: {1}", (DateTime)p.orderDate, p.product); {SELECT [t1].[ProductName] AS [product], [t2].[OrderDate] AS [orderDate] FROM [dbo].[Order Details] AS [t0] INNER JOIN [dbo].[Products] AS [t1] ON [t0].[ProductID] = [t1].[ProductID] INNER JOIN [dbo].[Orders] AS [t2] ON [t2].[OrderID] = [t0].[OrderID] WHERE [t2].[CustomerID] ORDER BY [t2].[OrderDate] } {SELECT [t1].[ProductName] AS [product], [t2].[OrderDate] AS [orderDate] FROM [dbo].[Order Details] AS [t0] INNER JOIN [dbo].[Products] AS [t1] ON [t0].[ProductID] = [t1].[ProductID] INNER JOIN [dbo].[Orders] AS [t2] ON [t2].[OrderID] = [t0].[OrderID] WHERE [t2].[CustomerID] ORDER BY [t2].[OrderDate] }

Join Join is similar to Inner Join in SQL That means that it is the intersection between two sequences The inner sequence is a keyed collection, that makes it a lot faster than a subquery (or traversing in a nested loop) Note that it uses Equals instead of == (remember the difference?) 24

Group Group transforms a sequence into a sequence of groups that contains a subsequence 25 var products = from p in dc.Products group p by p.Category.CategoryName; foreach (var group in products) { Console.WriteLine("\nCategory: {0}", group.Key); foreach (var product in group) Console.WriteLine(product.ProductName); var products = from p in dc.Products group p by p.Category.CategoryName; foreach (var group in products) { Console.WriteLine("\nCategory: {0}", group.Key); foreach (var product in group) Console.WriteLine(product.ProductName);

Use objects as keys It is possible to use objects as keys. It can be of an anonymous type or of a defined class 26 var products = from p in dc.Products group p by new { cname = p.Category.CategoryName, cid = p.Category.CategoryID } into productCategories orderby productCategories.Key.cid select productCategories; foreach (var group in products) { Console.WriteLine("\nID: {0} Category: {1}", group.Key.cid,group.Key.cname); foreach(var product in group) Console.WriteLine(product.ProductName); } var products = from p in dc.Products group p by new { cname = p.Category.CategoryName, cid = p.Category.CategoryID } into productCategories orderby productCategories.Key.cid select productCategories; foreach (var group in products) { Console.WriteLine("\nID: {0} Category: {1}", group.Key.cid,group.Key.cname); foreach(var product in group) Console.WriteLine(product.ProductName); }

Group into 27 Group will end the query. Use ‘Into’ to continue the query var products = from p in dc.Products group p by p.Category.CategoryName into productCategories orderby productCategories.Key select productCategories; foreach (var group in products) { Console.WriteLine("\nCategory: {0}", group.Key); foreach (var product in group) Console.WriteLine(product.ProductName); var products = from p in dc.Products group p by p.Category.CategoryName into productCategories orderby productCategories.Key select productCategories; foreach (var group in products) { Console.WriteLine("\nCategory: {0}", group.Key); foreach (var product in group) Console.WriteLine(product.ProductName);

Grouping and projecting Make a projection on the content of the group 28 var products = from p in dc.Products group p by p.Category into productCategories where productCategories.Count() < 8 orderby productCategories.Key.CategoryName select new { cname = productCategories.Key.CategoryName, count = productCategories.Count() }; foreach (var group in products) { Console.WriteLine("Category: {0}, Count: {1}", group.cname,group.count); } var products = from p in dc.Products group p by p.Category into productCategories where productCategories.Count() < 8 orderby productCategories.Key.CategoryName select new { cname = productCategories.Key.CategoryName, count = productCategories.Count() }; foreach (var group in products) { Console.WriteLine("Category: {0}, Count: {1}", group.cname,group.count); }

Nested queries Nested queries are similar to nested SELECT in SQL. But be careful: O(n 2 ) 29 var products = from p in dc.Products where p.CategoryID == (from c in dc.Categories where c.CategoryName == "Seafood" select c).First().CategoryID select p; foreach (var p in products) { Console.WriteLine("Category: {0}, Product: {1}", p.Category.CategoryName, p.ProductName); } var products = from p in dc.Products where p.CategoryID == (from c in dc.Categories where c.CategoryName == "Seafood" select c).First().CategoryID select p; foreach (var p in products) { Console.WriteLine("Category: {0}, Product: {1}", p.Category.CategoryName, p.ProductName); }

Let keyword Reuse expressions 30 var products = from p in dc.Products group p by p.Category into productCategories let upperName= productCategories.Key.CategoryName.ToUpper() where productCategories.Count() < 8 orderby upperName select new { cname = upperName, count = productCategories.Count() }; foreach (var group in products) { Console.WriteLine("Category: {0}, Count: {1}", group.cname,group.count); } var products = from p in dc.Products group p by p.Category into productCategories let upperName= productCategories.Key.CategoryName.ToUpper() where productCategories.Count() < 8 orderby upperName select new { cname = upperName, count = productCategories.Count() }; foreach (var group in products) { Console.WriteLine("Category: {0}, Count: {1}", group.cname,group.count); }

A few words on operators The LinQ operators are implemented with extension methods (recall the.Where method) You can define your own operator by defining an extension method And you can overwrite the existing operators But be careful: The operator should still do the same sort of things as it was originally intended for. E.g. the Where operator should still be a filter operator If the operator is not an aggregator it should be lazy to support deferred execution. Use ”yield” keyword when returning items from lazy operators 31

Define a contrary version of Where The extension method must be in another namespace 32 using System; using System.Collections.Generic; using LinQExamples; namespace MaryTheContrary //Rasmus Modsat { public static class ExtensionMethods { public static IEnumerable Where( this IEnumerable sequence, Func predicate) { foreach (Person p in sequence) { if (!predicate(p)) //Contrary yield return p; } }} using System; using System.Collections.Generic; using LinQExamples; namespace MaryTheContrary //Rasmus Modsat { public static class ExtensionMethods { public static IEnumerable Where( this IEnumerable sequence, Func predicate) { foreach (Person p in sequence) { if (!predicate(p)) //Contrary yield return p; } }}

Use of the contrary ‘where’ The compiler differs between different where’s by the namespace. So you must use ‘using 33 using MaryTheContrary;.... IEnumerable nonAdults = from p in personList where p.Age > 18 orderby p.Age select p; foreach(Person p in nonAdults) Console.WriteLine(“First Name: {0} Age:{1}", p.FirstName,p.Age); using MaryTheContrary;.... IEnumerable nonAdults = from p in personList where p.Age > 18 orderby p.Age select p; foreach(Person p in nonAdults) Console.WriteLine(“First Name: {0} Age:{1}", p.FirstName,p.Age);

LINQPad Utility for interactively search with LinQ You can get it here: 34

Exercise 1 You can get the processes that runs on the machine with IEnumerable System.Diagnostics.Process.GetProcesses() The Process object has several properties, e.g. ProcessName returns the name of the process WorkingSet64 returns the allocated memory in bytes TotalProccessorTime return the CPU time that has been used TODO: Write and test a LinQ query that returns the processes that has allocated more than 50mb (1mb=10124*1024 bytes) and ordered by process name. The allocated memory should be outputted in mb. Do the same in a foreach without LinQ. 35

Exercise 2, advanced Continued from exercise 1. You will get an exception if you try to get TotalProcessorTime on a process that you do not own, e.g. a system process And it not possible to filter those processes out  ! A workaround could be like this: 36 IEnumerable processes = Process.GetProcesses(); foreach (Process p in processes){ try { Console.WriteLine("Process Name: {0} Time: {1}", p.ProcessName, p.TotalProcessorTime); } catch (Exception e){ Console.WriteLine("Process Name: {0} Time: {1}", p.ProcessName, null); }} IEnumerable processes = Process.GetProcesses(); foreach (Process p in processes){ try { Console.WriteLine("Process Name: {0} Time: {1}", p.ProcessName, p.TotalProcessorTime); } catch (Exception e){ Console.WriteLine("Process Name: {0} Time: {1}", p.ProcessName, null); }} TODO: Override “select” so it inserts null’s for values that are not available. Test it in a LinQ expression (To get it work: Remove “Using System.Linq”, and also implement “Where”)