SQL Server 2005: CLR Integration & BI Aseel Mansour aseelm@microsoft.com Product Technology Specialist South Gulf

Agenda Programmability Business Intelligence Summary Database Programming Evolution New Application Frameworks .NET Framework Integration XML Support Web Services Support Data Access Encryption Support Business Intelligence ETL BI Studio Reporting Services Analysis Services Data Mining Summary

Early Database Programmability Early DB servers were non-extensible In late 80’s Stored Procs began extensible DB revolution Financial community was quick to adopt Why stored procs? Abstraction layer Application specific abstractions Business logic & rich constraint enforcement Security on application abstractions Data intensive code closer to data Stored proc languages functionally weak, proprietary, with poor dev tool support

Future Database Programmability Server-side DB programming trends Data-intensive code moved to the data Symmetric programming language support & tools Leverage programming language community progress Multiple languages & rich dev tools Abstract data types, Inheritance, encapsulation, .. Rich libraries & application frameworks

Agenda Programmability Business Intelligence Summary Database Programming Evolution New Application Frameworks .NET Framework Integration XML Support Web Services Support Data Access Encryption Support Business Intelligence ETL BI Studio Reporting Services Analysis Services Data Mining Summary

New in SQL Server 2005 Three new application frameworks: SQL Service Broker SQL Server Reporting Services New server-side programming support Stored procedures, triggers, & functions in C++, C#, or VB Server-side Abstract Data Types Industry leading programming tools

SQL Service Broker Inbound messages arrive on protocol pipe Service Queue Shared Stuff Private State Message Service Program SQL Service Broker Inbound messages arrive on protocol pipe Message is: Authenticated Dispatched to appropriate queue Service Programs: Pick up work from queue Run in different context than incoming message May run inside or outside server process May send additional messages

Agenda Programmability Business Intelligence Summary Database Programming Evolution New Application Frameworks .NET Framework Integration XML Support Web Services Support Data Access Encryption Support Business Intelligence ETL BI Studio Reporting Services Analysis Services Data Mining Summary

.NET Framework Integration Key Features Programming environment for Functions, Stored Procedures, Triggers User Defined Types, Aggregates In-Proc Data Access (ADO.NET V2) Symmetric data access Mid-tier/data tier deployment decision Security Integration of SQL and CLR security Three levels of code access security Safe, External-Access (verifiable), Unsafe Tight integration with Visual Studio Authoring, debugging, deployment, & profiling

The Developer Experience VB,C#,C++ Build Assembly: “TaxLib.dll” VS .NET Project SQL Data Definition: create assembly … create function … create procedure … create trigger … create type … Runtime hosted by SQL (in-proc) SQL Server SQL Queries: select sum(tax(sal,state)) from Emp where county = ‘King’

Authoring/Debugging/Deploying New Visual Studio project type in “Whidbey” for “Yukon” managed code Server debug integration Set breakpoints anywhere Full debugger visibility Single step support: Between languages: T-SQL, C#, VB, & C++ Between deployment tiers: E.g. Mid-tier, through SQL Server stored proc call & back to mid-tier

.NET Integration Key Theme: Choice & Control Choice of where to run logic Database, for logic that runs close to data Mid-tier, for logic that scales out Symmetric programming model Leverage skills mid-tier & server Choice of programming language C#, VB.NET, & Managed C++, for a safe, modern execution environment T-SQL continues to be supported & enhanced Remains good choice for data-intensive procedures Safe extended stored proc replacement

CLR Integration Demo

Agenda Programmability Business Intelligence Summary Database Programming Evolution New Application Frameworks .NET Framework Integration XML Support Web Services Support Data Access Encryption Support Business Intelligence ETL BI Studio Reporting Services Analysis Services Data Mining Summary

XML Support XML data type & XML index support XML views Unified XML & relational store Both SQL & XQuery supported by same industrial strength infrastructure Leverages existing SQL engine & optimizer XQuery with data modification extensions XML views Declarative Mapping Files XML schema enforcement Client access using ADO.NET & SOAP

XML Scenarios Data Exchange… XML data Point-of-Sales Supplier Platform independent transport format Loosely-coupled systems B2B, B2C, work flow, …

XML Scenarios Semi-structured storage… First Name Last Name Address Phone XML data type XML Datatype Loosely structured data Data with a dynamic schema XML Views Mixed data – structured/unstructured XML stores w/o relational support challenged

Native XML Store XML Data Type Native SQL type Use for column, variable or parameter CREATE TABLE docs (id INT PRIMARY KEY, xDoc XML NOT NULL) Store un-typed or typed XML instances Well-formed and validation checks Methods on XML data type Query(), value(), exist() & modify()

Native XML Store XML Schema Support Rich mechanism for schema definitions XML-based syntax (W3C standard) Not required for Query or Storage Benefits of typed data Guarantees shape of data Allows storage and query optimizations XML type system Store XML schemas in system meta-data

Native XML Store XML Index Create XML index on XML column CREATE XML INDEX idx_1 ON docs (xDoc) Creates indexes on tags, values & paths Speeds up queries Entire query is optimized Same award winning cost based optimizer Indexes are used as available You can create an XML index on the XML column. The example shows the simplest form. It is a variation of the regular index creation statement. You specify XML index, a name for the index, the table name and the XML column name. The XML index creates B-trees on the tags, values and paths within the XML instances in the column. This index is used when your search specifies tags or values or both to speeds up queries. A query can include accesses to both relational and XML columns. The entire query is optimized based upon algorithms used inside the query optimizer.

Native XML Store XML Query XQuery: query XML documents & data Standards-based: W3C working draft In document 123, return section heading of section 3 and later SELECT id, xDoc::query( 'for $i in /doc[@id = 123]//sec[@num >= 3] return <topic>{data($i/heading)}</topic>') FROM docs 1 @id=123 3 @id=30 1 <topic>Sort</topic> <topic>Search</topic> 3 NULL 

Native XML Store XML Data Modification Insert, update, & delete XQuery extensions XML sub-tree modification: Add or delete XML sub-trees Update values Add a new section after section 2: UPDATE docs SET xDoc::modify( ‘insert <section num=“2”> <heading>Background</heading> </section> after /doc/section[@num=1]') XQUERY in its current draft standard form does not support data modifications. Yukon introduces INSERT, DELETE and UPDATE statements on XML data type. These are not part of the standard. Our representatives are working on it with some other members in the committee. Tree-level modifications are supported. For example, you can insert or delete a subtree such as a whole section within a document. You can update values. For example, you can change section headings or section numbers. The example on this slide shows the insertion of a new section numbered 2 after the section numbered 1. It is a regular UPDATE statement on the table except for the XML column. On the XML column, the statement invokes the MODIFY() method. Within this method, an XML INSERT statement specifies the subtree to be inserted and the point of insertion.

XML Views Overview Default XML view of relational data User-defined XML views Specified using schema mapping Bulk load XML View XQuery, Updates First Name Last Name Address Phone XML data type

XML Views Features Decoupling mapping from domain-specific schemas Increased support for common database design patterns Stored Proc support Removed scoping and hierarchy restrictions Extensibility mechanism (Custom Tables) XQuery views over user-defined and default views for $i in sql:view(“nwnd.xsd”)//Customer where $i/name = “fred” return $i Removed scoping and hierarchy restrictions Previously we only allowed for 1 table to be in scope at any one time. This restricted nesting Cust/Order/Detail where a value in detail referred to the Cust table. This was troublesome for many customers. Previously we also only allowed 1 level of hierarchy per table. This restricted nesting values contained within that table. Cust/Company/CompanyAddress

XML Demo

XML Programming Platform Rich APIs, tools XMLAdapter, XMLReader, XMLWriter, Schema validation, and XQuery support XQuery builder XML Editor Support for new standards XPath 2.0 High perf XML Parsing & XSLT engines XML data type supported in ADO.NET Besides XQUERY and XML data modification, Yukon provides rich APIs and tools for data access for XML. An example is XML Adapter that allows you to access XML in XML data type columns or any other XML source such as a file. You can run an XQUERY expression on the XML data. The result is cached. Mapping schema allows you to map the result XML into tabular form and expose as XMLInfoSetDocument Updates are made in cache. You can roll them back or commit them. A very interesting tool is the XQUERY builder. It guides you through the task of constructing XQUERY expressions based on the shape of your XML instances. The XML APIs support new and emerging standards such as XPATH 2.0 They provide high performance in parsing, XSLT transformations, etc. XML is a new type in ADO.NET to fully support XML data type in client accesses.

Agenda Programmability Business Intelligence Summary Database Programming Evolution New Application Frameworks .NET Framework Integration XML Support Web Services Support Data Access Encryption Support Business Intelligence ETL BI Studio Reporting Services Analysis Services Data Mining Summary

HTTP/SOAP Features SQL & stored proc calls via HTTP/SOAP Easy, standards based connectivity from Unix platforms to SQL Server Provides native HTTP listening HTTP endpoint specifying URL, port, reqs Publish WSDL for endpoints Standard-based SOAP 1.1 and 1.2, WSDL 1.1, inlined XSD Windows and SQL authentication (SSL only) Stored Proc can return result as DataSet Compatible with mid-tier prog. model Native SOAP over HTTP allows users to execute SQL statements and invoke stored procedures. By default, this support is disabled. You have to create HTTP endpoints specifying URL, port and the requests to be supported on the endpoint. The endpoints publish the web service using (WSDL) Web-services Definition Language. Native support is standards compliant. SOAP 1.1 and 1.2, WSDL 1.1, and XSD 1.0 for inlined schema. Windows and SQL authentication are both supported. For SQL authentication, the user name and password are encoded in base64, and authenticated using secure connection. Stored procedure invocations through HTTP/SOAP can return the result as DataSet (tabular format with XML transmission). This is compatible with middle-tier programming.

SOAP Demo

Agenda Programmability Business Intelligence Summary Database Programming Evolution New Application Frameworks .NET Framework Integration XML Support Web Services Support Data Access Encryption Support Business Intelligence ETL BI Studio Reporting Services Analysis Services Data Mining Summary

Data Access API Enhancements: ADO .NET V2 Multiple active result sets (MARS) Object Persistence FX (ObjectSpaces) Query notifications – for cache invalidation Functional parity with ADO (ADOX, etc) Server cursors (SqlResultSet) Asynchronous client access Bulk update, paging, and batching XML Data type support User Defined Type (UDT) support

Encryption Support Key Secured By Associated with… Wraps the … Database entity created by CREATE KEY DDL Always stored encrypted. Secured in one of three ways Key Secured By Database entity created by CREATE CERTIFICATE DDL Always private keys stored encrypted. Secured in one of two ways User Password Certificate Another key Associated with… Wraps the … Private Key Public Key Secured By Password Master Key Secured By Secured By Service Key DPAPI ©2004 Microsoft Corporation. All rights reserved.

Encryption Support Demo

A Complete Enterprise Data Management and BI Solution SQL Server 2005 A Complete Enterprise Data Management and BI Solution Development Tools Reporting Services Management Tools Analysis Services OLAP & Data Mining Data Transformation Services ETL SQL Server Relational Engine

Data Transformation Services Breakthrough ETL capabilities Enterprise ETL platform High performance High scale Trustworthy and reliable Best in class usability Rich development environment Source control Visual debugging of control flow and data Great range of transforms out-of-the-box Highly extensible Custom tasks Custom enumerations Custom transformations Custom data sources Enterprise ETL platform High performance – competitive with the best in the industry High scale – distribute transformations across processors Trustworthy and reliable – built-in data cleansing, error data handling and restartability Best in class usability Full developer IDE Source control Visual debugging of control flow and data Configuration and deployment for develop, test and release. Great range of transforms out-of-the-box: little or no scripting required! Highly extensible – with native or managed code Custom tasks Custom enumerations Custom transformations Custom data sources

BI Development Studio A complete, integrated tool for the development of BI applications One tool, multiple technologies: Relational, OLAP, DM, DTS, Reporting, Code, Web pages… Enterprise software development environment: Integrated into Visual Studio.NET Team development, source control, versioning, developer isolation, resource independent coding Development cycle lifetime support Develop, Test, Deploy, Modify, Test… Breakthrough ease of use

Reporting Services Open, extensible enterprise reporting solution Report authoring, management, delivery Office System integration VS.NET development environment Integration with AS, DTS, management tools Developer enhancements Improved report interactivity Rich end-user reporting

Reporting Services Architecture Browser Office Custom App Report Control Data Proc. Rendering Report Processing Web UI Security Services (NT, Passport, Custom) Data Sources (SQL, OLE DB, XML/A, ODBC, Oracle, Custom) XML Web Service Interface Report Server Report Processing Data Processing Security Rendering Scheduling and Delivery Delivery Targets (E-mail, SharePoint, Custom) Output Formats (HTML, Excel, PDF, Custom) SQL Server Catalog

Reporting Services Demo

Analysis Services Unified Dimensional Model Pro-active caching Integrating relational and OLAP views Pro-active caching Bringing the best of MOLAP to ROLAP Advanced Business Intelligence KPIs, MDX scripts, translations, currency… Web services Native XML/A

Data Mining Focus Complete set of analytics Embedded data mining Most popular data mining algorithms (5 new) Enhanced creating, editing, and viewing environment Embedded data mining Embed data mining in LOB applications Complete SQL language based API Native XML/A support Integrated solution Tight integration with Relational, OLAP, DTS, Reporting technologies Comprehensive SQL Server BI platform Alliance with ISVs Focus on broadening the market Agreed on finalizing DM industry standard based on OLE DB for DM and XML/A Microsoft's goal for the data mining functionality in SQL Server 2005 is to build tools that: Are easy to use Provide a complete set of features Are easily embedded in production applications Integrate closely with the other SQL Server BI technologies, and Extend the market for data mining applications. To date, the development of such applications has been focused on the biggest problems at the biggest companies – those who can afford scarce analytical talent and high development costs that have traditionally been required to build a data mining application. Just as Microsoft's OLAP technologies have helped the OLAP market to grow, we are looking to extend data mining technologies to enterprises and corporate departments that have not been able to develop such applications in the past. Use the SQL Server 2005 Data Mining tools to explore a set of data for patterns, and then optionally perform prediction on those patterns. That's all data mining is: exploration, pattern discovery, and pattern prediction.

Complete Set of Algorithms Decision Trees Clustering Time Series Introduced in SQL Server 2000 Sequence Clustering Association Naïve Bayes All data mining tools, including Microsoft SQL Server 2005 Analysis Services, use multiple algorithms. Analysis Services, of course, is extensible; third party ISVs can develop algorithms that snap in seamlessly to the Analysis Services data mining framework. Depending on the data and the goals, different algorithms are preferred, and each algorithm can be used for multiple problems. Analytical problem Examples Microsoft algorithms Classification: Assign cases to predefined Decision Trees Naďve Bayes Nueral Nets classes such as "Good" vs "Bad“ Credit risk analysis Churn analysis Customer retention Neural Net

Summary Three New application frameworks Relational, Object, & XML Data Integration Rich, multi-lingual programming model Industry leading development tools Deep server-side storage services Integrated Web Services support Enhanced Data Access stack Real-time decision making Drive business insight at all levels of the organization Productive, integrated BI development experience End-to-end Business Intelligence capabilities out-of-the-box

© 2005 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.