ADO.NET 2.0: Advanced Data Access Patterns Pablo Castro DAT408 ADO.NET Technical Lead Microsoft Corporation

2 PRESENTATION ASP.NET System.Web (2.0) Compilation Configuration Handlers Hosting Mail Management Security UI (2.0) Util Presentation Framework System.Windows (3.0) Annotations Automation Controls Data Documents Ink Interop Navigation Resources Serialization Shapes Threading System.Windows.Media (3.0) 3D Animation Imagine Windows Forms System.Windows.Forms (2.0) Design Layout VisualStyles Printing Subsystem System.Printing (3.0) GDI+ System.Drawing Design Drawing2D Printing ASP.NET Data Management System.Web Caching Profile SessionState XML Data System.Xml Schema Serialization XPath Xsl Windows File System System.Storage (F) Core Audio Calendar Contacts Documents Image Media Messages Rules Sync Video XPS Documents System.Windows.Xps (3.0) System.IO.Packaging (3.0) Speech Integration System.Speech (3.0) Recognition Synthesis Language Integrated Query System.Query (F) System.Data.DLinq (F) System.Xml.XLinq (F) System.Expressions (F) Windows Workflow Foundation System.Workflow (3.0) System.Workflow.Activities (3.0) Rules System.Workflow.ComponentModel (3.0) Complier Desinger Serializaztion System.Workflow.Runtime (3.0) Hosting Messaging Windows Communications Foundation System.ServiceModel (3.0) Channels Configuration Diagnostics Integration QueueHelper System.ServiceModel.Security(3.0) Protocols Tokens Network Class Library System.Net Cache Mail (2.0) Network Information (2.0) Security (2.0) Sockets.NET Remoting System.Runtime.Remoting ASMX Web Services System.Web.Services Identity Management Microsoft.InfoCards (3.0) MSMQ System.Messaging Directory Services System.DirectoryServices COMMUNICATION FUNDAMENTALS BASE CLASS LIBRARIES System System.CodeDom System.ComponentModel System.Diagonostics System.IO (2.0) System.Resrouces System.Text System.ServiceProcess System.Threading System.Timers System.EnterpriseServices System.Transactions (2.0).NET Remoting System.Runtime.Remoting System.Text Generic (2.0) System.Reflection System.Configuration System.Text AccessControl (2.0) Cryptography (2.0) Permissions Policy Principal (2.0) Managed Add-In Framework System.Addins (F) Contact Microsoft.Build (2.0) System.Runtime CompilerServices ConstrainedExecution (2.0) InteropServices Hosting Serialization Versioning “ClickOnce” Deployment System.Deployoment (2.0) WINDOWS VISTA (2.0) (3.0)- New (2.0)- Substantially Improved - Windows Presentation Foundation (formerly codenamed “Avalon”) - Windows Communication Foundation (formerly codenamed “Indigo”) - Windows Workflow Foundation KEY ADO.NET System.Data Common Odbc OleDb OracleClient Sql SqlClient DATA

3 Agenda It’s all about moving data around… From the application to the database Moving tons of data fast From the database to the cache DataSet as a cache Maintaining a cache in sync From the cache to the application Querying the cache yes, query over DataSet :)

4 Moving Tons of Data Getting data from the server fast is easy DataReader objects do most of the work Going back to the server fast is harder Sending INSERT/UPDATE/DELETE statements separately is very slow Batching statements helps UpdateBatchSize property in the adapter SqlClient/OracleClient – DataSet-only Parameter arrays On databases/providers that support it

5 Client  Server: BCP+DML Useful when need to send lots of changes Use SqlBulkCopy in 2.0 Update sequence: First, bulk-insert all the changes into the server Use a temporary table Second, run a DML statement per change type One INSERTs, one for UPDATEs, one for DELETEs Do everything in the same transaction

6 Client  Server: BCP+DML Serious performance gains Even several times faster in some scenarios Fine print For SQL Server, the database needs to be in simple or bulk-logged recovery mode tempdb is in simple mode by default so it’s ok It works in “full” mode, but BCP is slower Talk to your DBA about the implications

7 Putting All Together For example… in a web application: Consume HTTP request as a stream Build a DataReader over the request Feed the DataReader to BCP Sends data to the server in streaming mode Kick off the DML statements Use an asynchronous ASP.NET page Execute an asynchronous ADO.NET command

8 Agenda It’s all about moving data around… From the application to the database Moving tons of data fast From the database to the cache DataSet as a cache Maintaining a cache in sync From the cache to the application Querying the cache yes, query over DataSet :)

9 Using dataset as a cache DataSet has been greatly enhanced Better scalability across the board Incremental index updates Low-overhead DataView maintenance Faster, more compact serialization We can now handle huge caches Cache granularity becomes important

10 SqlDependency For Caching SqlDependency can notify of changes Track queries and fire an event when the results would change It tracks whole result-sets SqlDependency and caching Having change notifications helps in caching No need for cache expiration policies Expire cache when the underlying data changed

11 SqlDependency + DataSet Middle-tier cache Cache data in a DataSet Track the queries used to load the DataSet using SqlDependency When change notifications come Simply get rid of the DataSet Load a new one on the next request

12 Controlling Granularity For large caches, getting rid of all the data in the cache is not an option SqlDependency doesn’t have granularity control, but…

13 Controlling Granularity Use a large DataSet Load one “segment” at a time e.g., on cache miss On change notification Use.Select to find the invalidated segment Remove it from the cache Next request will fill it up again DataTable SqlDependency SqlDependency SqlDependency SqlDependency Change SqlDependency

14 Agenda It’s all about moving data around… From the application to the database Moving tons of data fast From the database to the cache DataSet as a cache Maintaining a cache in sync From the cache to the application Querying the cache Yes, query over DataSet :)

15 Using DataSet as a Cache DataSet enhancements enable huge caches e.g. millions of rows With lots of data… It’s hard to navigate through data It’s more practical to query over the data DataSet does some query… …but it’s often not enough

16 Query Over DataSet We get requests for this every week :) What does it mean? Today, you can filter/sort over a single table Joins are the main missing piece What does it take to do it? Most base services are there in.NET 2.0 Enough to build basic scenarios, some cases will be hard to do efficiently

17 Representing a Query “Algebra tree” is a simple option Nodes represent logical query operators Free from syntax constructs Full query syntax is “nice” but not needed Easy to extend this sample with syntax support Create a compiler that produces algebraic trees

18 Algebraic Operators ProjectFilterJoinSort… Project (name, date) Join (c.id = o.customerId) SELECT c.name, o.date FROM Customers c INNER JOIN Orders o ON c.id = o.customerId WHERE o.status = ‘Pending’ Scan (Customers) Filter (status=‘…’) Scan (Orders)

19 Compiling A Query Project (name, date) Join (c.id = o.customerId) Scan (Customers) Filter (status=‘…’) Scan (Orders) Projection iterator Nested-loop join iterator Base-table scan iterator Filtered-scan iterator + Index (DataView) Indexed join iterator Indexed access

20 Executing A Query Traditional approach using “iterators” Build an iterator tree from the logical tree We’re skipping the physical plan for simplicity Iterators represent physical operators I.E. how the operator is implemented Consume from the top-level iterator

21 Operators: Projection/Filter Projection (“select”) Simply keep track of the required columns Expressions implemented with DataSet expressions Filter (“where”) Implemented with DataTable.Select Indexes created on-demand by.Select If not already there A DataView can be used as a permanent index

22 Operators: Sort Sort (“order by”) DataSet surfaces this together with filter DataTable.Select or a DataView Will also use indexes Need to collapse operators Or do one (filter/sort) manually

23 Operators: Join Join Not supported by DataSet out-of-the-box Different strategies Do a merge join if inputs already sorted Do a relationship-based sort if found a DataRelation Do a index-based join if one side of the join has an “index” (DataView) Do a nested-loop join as the last resort Optionally, create indexes on demand

24 Operators: The hard ones DataView/Select don’t take comparison operator as input Only “==“ or “!=“ So indexed range queries cannot take advantage of indexes Not many workarounds Non-indexed queries Roll your own index :(

25 Summary ADO.NET 2.0 provides tons of flexible services Easy scenarios are straightforward Hard scenarios possible with some extra code ADO.NET continues to evolve to match modern application requirements

26 Resources Blogs Team blog: Data-access MSDN site Newsgroups NNTP server: msnews.microsoft.com msnews.microsoft.com Group: microsoft.public.dotnet.framework.adonet microsoft.public.dotnet.framework.adonet

© 2005 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.