Riaan Vermeulen Data Solutions Architect Eclipse Networks

Information-Centric Applications Relational Data XML Documents and Multimedia Spatial Applications Structured Unstructured Semi-Structured Structured

Key Application Challenges Growth of the types of data to be hosted Increasing complexity Search and Indexing Caching and Synch Object Mapping Rich Query Capability Data Model Tier Business Intelligence (BI) Analysis Application Integration Compliance Reporting Application Tier Large Data Sets Transactions and Security Reliability and Scale Referential Integrity Storage Tier RelationalData Documents and Multimedia Spatial XML

Strategic Challenges and Goals Dealing with relational and non-relational data platforms Growth in application complexity and duplicated functionality Compensating for unavailable services Pain Points Goals Reduce the cost of managing all types of data Simplify the development of applications which use relational and non-relational data Extend services currently available for relational data to non-relational data Provide non-relational services to relational data

Four Pillars of SQL Server Your Data, Any Place, Any Time Server Mobile and Desktop OLAP 2 FILE XML RDBMS 1 Services Query AnalysisReportingIntegrationSynch Search Cloud 1: Relational database management systems 2: Online Analytical Processing

Beyond Relational Feature Overview SQL Server 2005 SQL Server 2008 Large UDTs Sparse Columns Wide Tables/Column Set Filtered Indices HierarchyID Relational BR Support User Defined Types Full Text Indexing Documents & Multimedia Remote BLOB Store API FILESTREAM Integrated Full-Text Search (FTS) Spatial Fully supported Geometry and Geography data types and Functions XML Data Type and Functions XML Upgrades XML

Spatial Key Application Challenges Growth of the types of data to be hosted Increasing complexity Search and Indexing Caching and Synch Object Mapping Rich Query Capability Data Model Tier Business Intelligence (BI) Analysis Application Integration Compliance Reporting Application Tier Large Data Sets Transactions and Security Reliability and Scale Referential Integrity Storage Tier RelationalData Documents and Multimedia XML

BLOB storage options Low cost per GB Streaming Performance Complex application development and deployment Integration with structured data Advantages Challenges Integrated management Data-level consistency Poor data streaming support File size limitations Highest cost per GB Lower cost/GB at scale Scalability & Expandability Complex application development and deployment Separate management Enterprise-scales only Example Windows File Servers NetApp NetFiler EMC Centera Fujitsu Nearline SQL Server VARBINARY(MAX) Documents & Multimedia Use File Servers DB ApplicationApplication BLOBs Dedicated BLOB Store DB ApplicationApplication BLOBs Store BLOBs in Database DB ApplicationApplication BLOBs

SQL Server 2008 BLOBs Remote BLOB Storage FILESTREAM Storage SQL BLOB Documents & Multimedia Use File Servers DB ApplicationApplication BLOB Dedicated BLOB Store DB ApplicationApplication BLOB Store BLOBs in Database DB ApplicationApplication BLOB Store BLOBs in DB + File System ApplicationApplication BLOB DB

FILESTREAM Storage Attribute on VARBINARY(MAX) Works with integrated FTS Unstructured data stored directly in the file system (requires NTFS) Dual Programming Model TSQL (Same as SQL BLOB) Win32 Streaming APIs with T-SQL transactional semantics Data Consistency Integrated Manageability Back Up/Restore Administration Size limit is the file system volume size SQL Server Security Stack Documents & Multimedia Store BLOBs in DB + File System ApplicationApplication BLOB DB

FILESTREAM : Basics Enabled per instance When enabling for the first time, computer or instance restart may be required to hook in the file-system filter driver FILESTREAM data stored in a FILESTREAM FILEGROUPS Stored as NTFS directories, called ‘data containers’ Can be on compressed volumes Create a FILESTREAM FILEGROUP at database create time or add one later Documents and Multimedia

FILESTREAM : Basics FILESTREAM columns are defined as varbinary(max) columns that have the FILESTREAM attribute Table must have a uniqueidentifier ROWGUIDCOL column and a varbinary(max) column with the FILESTREAM attribute If full-text indexing is required, the table also needs a column to store the document file type extension Documents and Multimedia

FILESTREAM : Programming T-SQL Cannot be used to perform partial updates to FILESTREAM data. Underlying BLOB data in the file system is deleted asynchronously by Garbage Collector. Win32 API Win32 Streaming APIs with T-SQL transactional semantics. A token must be obtained before FILESTREAM files can be accessed. GET_FILESTREAM_TRANSACTION_CONTEXT(), provides the token that binds the FILESTREAM file system streaming operations with a started transaction. SqlFileStream API can be used to call Win32 streaming interfaces, such as ReadFile() and WriteFile(). Documents and Multimedia

FILESTREAM

FILESTREAM : Security FILESTREAM directories have restricted access while the database is open Users have permissions to open files based on SQL Server table/column security settings Privileged users *can* mess with the files though… When the database is closed, FILESTREAM directory access is gated by Windows security As the Demo showed, the FILESTREAM directories and files are not file locked like open databases and can be altered/deleted when not actively in use Documents and Multimedia

FILESTREAM : Data Consistency When a FILESTREAM value is updated (by creating a new file), the old file is not deleted immediately The link on the data page is updated to point to the new file After the transaction has committed, the OLD file is made available for garbage collection If the transaction rolls back, the change to the link in the data page is rolled back and the NEW file is made available for garbage collection Garbage collection is driven by tombstone tables (stored as internal tables in the default FILEGROUP) Isolation Schematics Win 32 – Only support Read-Commited Isolation Level T-SQL – Full serializable Support Documents and Multimedia

FILESTREAM : Interaction with other Features Failover clustering FILESTREAM filegroups must be on shared storage Log shipping Both databases must have FILESTREAM enabled Backup/Restore FILESTREAM data backed up as part of regular backup types FILESTREAM filegroups can be excluded from database backups Full-Text Indexing The table must have a column storing the FILESTREAM BLOB file type extension SQL Express is Supported The 4GB database size limit does not include FILESTREAM data Documents and Multimedia

FILESTREAM : Replication FILESTREAM data can be replicated When the replication topology involves instances using different versions of SQL Server, there are limitations on the size of data than can be sent to down-level instances. The replication filter options determine whether the FILESTREAM attribute is replicated or not using transactional replication. When merge replication is used, both it and FILESTREAM require a uniqueidentifier column. Care must be taken with the table schema when using merge replication so that the GUIDs are sequential (i.e., use NEWSEQUENTIALID() rather than NEWID()). Documents and Multimedia

FILESTREAM : Unsupported Features Database mirroring DBM cannot be configured if the database contains FILESTREAM This may be a MAJOR barrier to adoption in V1 Database snapshots Snapshot can be created on non-FILESTREAM FILEGROUP only SQL Server encryption FILESTREAM data cannot be encrypted using either column level encryption OR Transparent Data Encryption Documents and Multimedia

Write Performance (Remote) Documents and Multimedia

Read Performance (Remote) Documents and Multimedia

FILESTREAM : Limitations Only works with NTFS FILESTREAM files cannot be properly deleted or renamed through Win32 Operations have to be done transactionally otherwise they cause corruption "SELECT GET_FILESTREAM_TRANSACTION_CONTEXT()" FILESTREAM data cannot be stored remotely FILESTREAM columns cannot be INCLUDED columns In-place partial updates not supported New file created and old file only available for deleting after commit Documents and Multimedia

Migration SQL BLOBs to FILESTREAM Conceptual walkthrough Create one (or more) FILESTREAM (filegroup) Alter table to add FILESTREAM column For each existing row, update data in new row with empty FILESTREAM value Write LOB data (File I/O or T-SQL access) Optional: drop Varbinary(MAX) column BLOB migration utilities support (intended for SQL Server 2008 R2)

Case Study – What are we doing? Department of Education – Mpumalanga Script and Marksheet Management System Track, Scan & Store 500k+ Marksheets every year. 650 GB – 1.3 TB of Files per Anum FILESTREAM Win 32 API Write and Read Access (C# VS 2008) Partitioned over 6 LUNs Use ITFs SharePoint Search Portal

Customer Application Remote Blob Store Architecture Customer applications can transparently support different BLOB stores Each Remote Blob Store vendor responsible for delivering their own providers NetApp lib IBM lib Centera lib SQL RBS API NetApp IBM Centera SQL DB Provider API RBS Services Create Fetch GC Delete Documents & Multimedia

RBS Workflow ApplicationApplication RBS Client Library BLOB Store Provider Library BLOB Store SQL Server ClaimIDClaimDatePhotoRef43906/5/2007<Binary(20)> Write BLOB(Photo) Return Blob ID Write Blob ID to PhotoRef field 2 3 Machine Boundary

RBS Fundamentals Most useful in environments where interoperability is required No restrictions on back-end store Back-end can change with no app change Looser (link level) consistency guarantees SQL Server handles link consistency and garbage collection

Unstructured Storage In SQL08 File Stores / External Blob Stores (CAS) SQL BLOBsRemote Blob API FILESTREAM Streaming Performance Depends on external store Link Level Consistency Data Level Consistency Integrated Management Non-local Windows File Servers n/a External Blob Stores n/a Documents & Multimedia

FILESTREAM Comparison

Full Text Search Challenges Indexes stored outside SQL Server lead to manageability challenges Mixed query performance suffers from having to pull over complete full-text result set Scaling issues on big boxes Documents & Multimedia

Full Text Indexing Upgrades Full-text indexes fully integrated into SQL Server Make mixed queries perform and scale SELECT * FROM candidates WHERE CONTAINS(resume,’”SQL Server”’) AND ZipCode = ‘98052’

Spatial Data Overview Storage and retrieval of spatial data using standard SQL syntax New Spatial Data Types (geometry, geography) New Spatial Methods (intersects, buffer, etc.) New Spatial Indexes Offers full set of Open Geospatial Consortium components (OGC/SQL MM, ISO 19125) Spatial Builder Interface SSMS Visualization Integration with Virtual Earth Spatial See also: DAT03-HOL Integrating Microsoft SQL Server 2008 Spatial Support with Microsoft Virtual Earth

XML Improvements Improved XML Schema Validation Support for storing and validating Office 12 Document formats Support for lax validation Full xs:dateTime support Support for values without timezone timezone preservation Improved support for lists and union types Added support for let-clause in XQuery Added fn:upper-case()/fn:lower-case() Added support for insert into /a/b XML

Storing Relational Data RelationalData HierarchyID Store arbitrary hierarchies of data and efficiently query them Large UDTs No more 8K limit on User Defined Types Sparse Columns Optimized storage for sparsely populated columns Wide Tables Support for hundreds of thousands of sparse columns Filtered Indices Define indices over subsets of data in tables

Filtered Indexes Filtered Indexes and Statistics Indexing a portion of the data in a table Filtered/co-related statistics creation and usage Query/DML Optimization to use filtered indexes and statistics Restrictions Simple limited grammar for the predicate Only on non-clustered indexes Benefits Lower storage and maintenance costs for large number of indexes Query/DML performance benefits: IO only for qualifying rows Relational BR Support

Summary SQL Server 2008 will make it easier to create information-centric applications that require Unstructured documents XML Semi-structured information Combine the above with relational data by: Reducing the cost of managing all types of data Simplifying the development of applications which use relational and non-relational data Extending services currently available for relational data to non-relational data

Future of Beyond Relational Rich unstructured data Enable existing BR apps to store data in SQL Server Win 32 File I/O API compatibility Better integration of FILESTREAM and RBS programming models Better scalability of FILESTREAM Property Search and promotion iFTS improvements in functionality and scale/performance Deep Spatial More functionality Across BI components

Resources Whitepapers & Videos FILESTREAM/RBS Whitepapers: unstructured.aspx What’s new for XML in SQL Server 2008: xml.aspx xml.aspx iFTS: SQL Server 2008 Business Value Calculator:

Sessions On-Demand & Community Resources for IT Professionals Resources for Developers Microsoft Certification & Training Resources Resources

Please complete an evaluation

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.