1. FILESTREAM Storage Attribute
Klaus Aschenbrenner

2. About me Software Architect @ EDS, an HP company Speaker, Author
„Pro SQL Server 2008 Service Broker“

3. Agenda Challenges for using unstructured data
The Filestream Feature - An Overview
Filestream and T-SQL
OpenSqlFilestream
System.Data.SqlTypes.SqlFileStream
Filestream access patterns
Filestream transactional semantics

4. Motivation Storing large binary objects in databases is suboptimal
Large objects take buffers in database memory
Updating large objects cause database fragmentation
In file system however, "update" is delete and insert
"Before image" in an update is not deleted immediately
Storing all related data in a database adds
Transactional consistency
Integrated, point-in-time backup and restore
Single storage and query vehicle

5. Challenges using Unstructured Data
Cost of storing your unstructured data, including not only hardware costs, but also the management overhead costs
Managing retention and archival and policies
Integrating unstructured data files with associated relation data
Indexing and searching unstructured data

6. Business Scenarios 1/2
An insurance company needs to store policy and retrieve them for claims processing
An online retailer needs to store video footage of its products for display in an e-commerce site
A radio station needs to make searchable library of podcasts available for download from its website

7. Business Scenarios 2/2
A telephony system needs to store voic messages as audio streams so that they can be retrieved remotely
A library needs to convert and archive large volumes of existing paper and analog content for indexing and use in digital research tool
An online retailer needs to store video footage of its products for display in an e-commerce site

8. FILESTREAM Storage FILESTREAM is an alternative storage format
For VARBINARY(MAX) columns
Specified by FILESTREAM keyword on column in CREATE/ALTER TABLE
Metadata still stored in database
Data stored in file system
One file per non-null row
In database optimal up to 256 KB
In file system optimal over 1 MB

9. FILESTREAM Instance Setup
OS portion of FILESTREAM setup
SQL Server Configuration Manager
Three setup alternatives
Enable FILESTREAM, local access only, remote access
SQL Server configuration
sp_configure filestream_access_level
Disabled, SQL only, SQL and Win32 streaming

10. FILESTREAM Setup Database
Add filegroup for filestream storage
ALTER DATABASE ADD FILEGROUP
The filegroup is mapped to a directory
Directory must not exist
Parent directory must exist
Must be NTFS file system
NAS volumes and remote shares not supported
Caution: Files deletable from file system if you have appropriate permissions

11. FILESTREAM Table Setup
Columns must be VARBINARY(MAX)
(N)VARCHAR(MAX), XML type not supported
Table must contain UNIQUEIDENTIFIER
Marked with ROWGUID property
FILESTREAM columns can be larger than 2 GB
Limited by file size

12. FILESTREAM Programming
FILESTREAM is a normal SQL column
Can be accessed using Win32 I/O APIs and special file handle
Note: file must not be accessed with an ordinary Win32 file handle or database corruption can result
DACL'd to Administrators and SQL Server Service Account only

13. FILESTREAM Programming - TSQL
Using T-SQL directly means
SQL Server's buffer pool is used
WRITE method is unavailable
Insert and updating NULL to non-NULL must use SQL
Insert or update to empty content creates the file
Use streaming APIs to populate
Delete deletes underlying file

14. SQL Functions
Functions that work against the VARBINARY(MAX) column work with FILESTREAM
SUBSTRING - reads part of column
CHARINDEX, LEN, REPLACE
CAST, CONVERT
Full text indexed FILESTREAM columns can use CONTAINS, FREETEXT
Encryption is NOT supported on FILESTREAM data

15. FILESTREAM Programming - TSQL
Demo
FILESTREAM Programming - TSQL

16. FILESTREAM - Streaming
Using FILESTREAM with streaming API is like using multiple SQL statements
BEGIN TRANSACTION
Obtain UNC name and transaction context
Read, Write,Update file
Open is "beginning of statement"
Close is "end of statement"
COMMIT or ROLLBACK transaction

17. Streaming Requirements
To use streaming IO, you must have
Windows login (not SQL login)
Access to the table with FILESTREAM column
SMB access to the server if access is remote

18. OpenSqlFilestream
OpenSqlFilestream is a method defined in SQL Native Client driver
Useable in ODBC, OLE DB directly
Useable in .NET through DLLImport

19. OpenSqlFilestream OpenSqlFilestream returns a file handle Requires
File name
Required Access (Read, Write, ReadWrite)
AccessOptions
FILESTREAM transaction context

20. Getting Parameters
Pathname exposed through PathName method on FILESTREAM instance
PathName is case sensitive
Returns a symbolic UNC name
Returns null if FILESTREAM column is NULL
GET_FILESTREAM_TRANSACTION_CONTENT
Gets the transaction context
Transaction must already be started

21. Opening Options ASYNC NO_BUFFERING WRITE_THROUGH SEQUENTIAL_SCAN
WRITE_THROUGH is default for native APIs
SEQUENTIAL_SCAN
Fastest for reading entire file
RANDOM_ACCESS

22. Win32 APIs Supported OpenSqlFilestream – retrieves a handle
CloseHandle – frees the handle
ReadFile, WriteFile, TransmitFile, SetFilePointer, SetEndOfFile, FlushFileBuffers supported
Any other APIs return permission denied
Creating memory-mapped views unsupported

23. SqlFileStream Class
SqlFileStream is a .NET class that encapsulates OpenSqlFilestream
In System.Data.SqlTypes
Inherits from System.IO.Stream
Supports asynchronous reads and writes using .NET AsyncCallback pattern
Write-through is not default, uses client buffering

24. Demo
SqlFilestream

25. Access Pattern - SELECT
SELECT requires
BEGIN TRANSACTION
SELECT to retrieve other columns, transaction context, file name
Open SqlFileStream for Read
COMMIT TRANSACTION

26. Access Pattern - INSERT
Insert consists of
Begin transaction
SQL INSERT w/empty (not null) FILESTREAM column
OUTPUT clause to obtain transaction context, filename
SqlFilestream, access Write
Fill filestream content
Using 60k multiples for buffer size optimal
Commit Transaction

27. Access Pattern - UPDATE
UPDATE consists of
BEGIN TRANSACTION
SQL UPDATE statement,
must update NULL FILESTREAM column to empty
OUTPUT to obtain transaction content, filename
UPDATE (or fill) FILESTREAM column
COMMIT TRANSACTION

28. Partial Updates Handle returned by SqlFilestream can get existing data
ReadFile on this handle returns end of file
Use fsctl_sql_filestream_fetch_old_content to cause copy of existing file to be returned
Using this with overlapped IO can perform a partial update "in place"

29. Transactional Semantics
Transactions and Isolation Levels
Streaming – read committed only
SQL – read committed, repeatable read, serializable, snapshot
SQL – read uncommitted supported but will not see in-flight changes
Do not attempt to commit/rollback transaction when file handle is open

30. SQL and Streaming Interactions
When FILESTREAM is open for read
SQL DDL and DML statements are allowed
SQL SELECT is allowed
SQL commit and save points not allowed
When FILESTREAM is open for write
SQL DDL and DML statements not allowed
SQL SELECT is allowed (gets old data)

31. FILESTREAM & Triggers Triggers fire when
INSERT/UPDATE statement completes
File handle is closed

32. FILESTREAM Performance
SQL access uses SQL buffers
File access use less SQL buffers
But using file buffers could decrease storage allocated to SQL Server
Except for small data, use file access
For incremental updates, all data must be read in buffers

33. Load Balancing
Partitioning can be used to create a table schema that allows crude of load balancing of file stream between multiple volumes
Desirable to avoid I/O hardware limitations problems
Enable easy hardware scalability without
N simultaneous users accessing videos data files might saturate I/O subsystem

34. Demo
Load Balancing

35. LINQ to SQL & Entity Framework
These APIs support FILESTREAM column as VARBINARY(MAX)
No special support for SqlFileStream and related APIs

36. Remote BLOB Storage RBS is part of SQL Server 2008 Feature Pack
Provides blob management with transactional consistency
Blob files can reside on remote server
Content Addressable Store (CAS)
Service Storage Solution (MSN XStore)
Consists of
Stored procedures for transactional blob management
SQL Agent jobs for maintenance and cleanup
API for integrating with SQL Server and TransactionScope

37. Summary FILESTREAM is a storage choice for VARBINARY(MAX)
Set up at OS, instance, and database level
Defined as a column property
You can access FILESTREAM in
“Normal" T-SQL
Streaming APIs through file system
OpenSqlFilestream or SqlFileStream data type

38. Complete the Evaluation Form & Win!
You could win a Dell Mini Netbook – every day – just for handing in your completed form! Each session form is another chance to win!
Pick up your Evaluation Form:
Within each presentation room
At the PASS Booth near registration area
Drop off your completed Form:
Near the exit of each presentation room
Sponsored by Dell

39. for attending this session and the 2009 PASS Summit in Seattle
Thank you
for attending this session and the PASS Summit in Seattle