Using BLOB Storage In this module- What is BLOB storage? How do I access it? What is Azure Drive?

What is a BLOB Binary Large OBject

Need shared access When running in a scaled environment, data must be in a shared location. The LB is not sticky. Web Role LB Web Role Storage Web Role

BLOBs in Azure & Size Each BLOB can be up to 1TB in size. You can have as many containers and BLOBs as you want. Containers can be created/destroyed on the fly.

BLOB Addresses Pages/ Blob Blocks Account Container images videos http://<account>.blob.core.windows.net/<container>/<blobname> Account Container Blob Pages/ Blocks PIC01.JPG images Block/Page PIC02.JPG contoso Block/Page videos VID1.AVI

Containers Similar to a top level folder Can hold up to 100TB of files Can only contain BLOBs Each container has an access level: Private Default, will require the account key to access Full public read Public read only Full public read – anonymous access; ability to list contents of container; access to container data Public read only – anonymous access; only list blob – not contents of container; no access to container data

Blob Containers Multiple Containers per Account Blob Container Special $root container Blob Container A container holds a set of blobs Set access policies at the container level Associate Metadata with Container List the blobs in a container Including Blob Metadata and MD5 NO search/query. i.e. no WHERE MetadataValue = ? Blobs Throughput Effectively in Partition of 1 Target of 60MB/s per Blob Slide Objective Understand containers Speaker Notes Account can contain unlimited number of containers Root container useful when serving Silverlight and flash out of Blob storage. May need to store Cross domain access policy files in root of the domain Metadata is up to 8KB of name value pairs per container Notes http://msdn.microsoft.com/en-us/library/dd179361.aspx http://msdn.microsoft.com/en-us/library/ee395424.aspx A root container serves as a default container for your storage account. A storage account may have one root container. The root container must be explicitly created and must be named $root. A blob stored in the root container may be addressed without referencing the root container name, so that a blob can be addressed at the top level of the storage account hierarchy. For example, you can now reference a blob that resides in the root container in the following manner:

Working with Containers

Working with BLOBs

Uploading a BLOB Other Options UploadFile UploadText UploadFromStream

Downloading a BLOB Other Options DownloadText DownloadByteArray DownloadToFile

Container & BLOB Metadata Simple Dictionary. Must call SetMetadata() to persist to the store.

Snapshotting Pins a version of your BLOB that is read only.

Two Types of Blobs Under the Hood Block Blob Targeted at streaming workloads Each blob consists of a sequence of blocks Each block is identified by a Block ID Size limit 200GB per blob Optimistic Concurrency via ETags Page Blob Targeted at random read/write workloads Each blob consists of an array of pages Each page is identified by its offset from the start of the blob Size limit 1TB per blob Optimistic or Pessimistic (locking) concurrency via Leases Slide Objective Understand different blob types Speaker Notes Block blobs are comprised of blocks, each of which is identified by a block ID. You create or modify a block blob by uploading a set of blocks and committing them by their block IDs. If you are uploading a block blob that is no more than 64 MB in size, you can also upload it in its entirety with a single Put Blob operation. When you upload a block to Windows Azure using the Put Block operation, it is associated with the specified block blob, but it does not become part of the blob until you call the Put Block List operation and include the block's ID. The block remains in an uncommitted state until it is specifically committed. Writing to a block blob is thus always a two-step process. Each block can be a maximum of 4 MB in size. The maximum size for a block blob in version 2009-09- 19 is 200 GB, or up to 50,000 blocks. Page blobs are a collection of pages. A page is a range of data that is identified by its offset from the start of the blob. To create a page blob, you initialize the page blob by calling Put Blob and specifying its maximum size. To add content to or update a page blob, you call the Put Page operation to modify a page or range of pages by specifying an offset and range. All pages must align 512-byte page boundaries. Unlike writes to block blobs, writes to page blobs happen in-place and are immediately committed to the blob. The maximum size for a page blob is 1 TB. A page written to a page blob may be up to 1 TB in size but will typically be much smaller Notes http://msdn.microsoft.com/en-us/library/dd135734.aspx

Uploading a Block Blob THE BLOB Benefit blobName = “TheBlob.wmv”; PutBlock(blobName, blockId1, block1Bits); PutBlock(blobName, blockId2, block2Bits); ………… PutBlock(blobName, blockIdN, blockNBits); PutBlockList(blobName, blockId1,…,blockIdN); 10 GB Movie Block Id 1 Block Id 2 Block Id 3 Block Id N Benefit Efficient continuation and retry Parallel and out of order upload of blocks Windows Azure Storage TheBlob.wmv TheBlob.wmv

Page Blob – Random Read/Write Create MyBlob Specify Blob Size = 10 Gbytes Sparse storage - Only charged for pages with data stored in them Fixed Page Size = 512 bytes Random Access Operations PutPage[512, 2048) PutPage[0, 1024) ClearPage[512, 1536) PutPage[2048,2560) GetPageRange[0, 4096) returns valid data ranges: [0,512) , [1536,2560) GetBlob[1000, 2048) returns All 0 for first 536 bytes Next 512 bytes are data stored in [1536,2048) 512 1024 1536 2048 10 GB Address Space 2560 10 GB

Shared Access Signatures Fine grain access rights to blobs and containers Easiest way to allow an untrusted client to upload files to your blob account Sign URL with storage key – permit elevated rights Revocation Use short time periods and re-issue Use container level policy that can be deleted Slide Objective Introduce Shared Access Signatures Speaker Notes Shared Access Signatures provide access rights to containers and blobs at a more granular level than by simply setting a container’s permissions Grant users access to a specific blob or to any blob within a specified container for a specified period of time. Specify what operations a user may perform on a blob that's accessible via a Shared Access Signature. Use HTTPS to protect the signature (it is like a short dated password) Two approaches Ad-hoc Use for very short dated single use scenarios Policy based Use for longer dated revocable permission sets Always endeavour to use Least Permission set possible Notes http://msdn.microsoft.com/en-us/library/ee395415.aspx

Sample Use Attach to container Create Read Policy for 10 Minutes Create Write Policy for 10 Minutes From Steve Marx’s blog: The first statement creates the policy, which specifies read-only access for the next ten minutes. The second statement lists all the blobs in a container and creates a list of URIs that have the Shared Access Signature query string appended to them. I use this to data bind the View.aspx page. To grant write access to a specific blob, I could use the same code as above, except with Write permissions. Attach to URI to apply

Content Delivery Network (CDN) Scenario Frequently accessed blobs Accessed from around the world Windows Azure Content Delivery Network (CDN) provides high-bandwidth global blob content delivery 20+ locations globally (US, Europe, Asia, Australia and South America), and growing Same experience for users no matter how far they are from the geo-location where the storage account is hosted Cost US located CDN nodes 15c/GB + 1c/10,000 txn Rest of World 20c/GB + 1c/10,000 txn Traffic from Storage node to edge node at standard rates Slide Objectives Understand basic concept of a CDN Understand at a high level how Windows Azure CDN works Speaker Notes The Windows Azure CDN provides edge nodes around the world Data stored in CDN enabled storage accounts is retrieved from the origin storage container and cached at each edge node in a lazy load fashion Windows Azure Customers have control over how long data is cached for. Windows Azure CDN has 18 locations globally (United States, Europe, Asia, Australia and South America) and continues to expand The benefit of using a CDN is better performance and user experience for users who are farther from the source of the content stored in the Windows Azure Blob service. Windows Azure CDN provides worldwide high-bandwidth access to serve content for popular events. Notes http://blogs.msdn.com/b/windowsazure/archive/2009/11/05/introducing-the-windows-azure-content- delivery-network.aspx

Reach, Scale and Capability OVER 3 TERABITS PER SECOND OF CAPACITY IS AVAILABLE FROM OUR 20+ GLOBAL LOCATIONS WITH 99.95% AVAILABILITY. CDN SERVICE SCALES AUTOMATICALLY WITHOUT USER INTERVENTION. LIVE INSIDE MICROSOFT FOR OVER 3 YEARS – NOW AVAILABLE TO AZURE CUSTOMERS.

How does it work? http://civ5.blob.core.windows.net/demo/civ5demo.exe Public Blob Container ! Edge Node 1 http://<guid>.vo.msecnd.net/demo/civ5demo.exe

Windows Azure Drives Durable NTFS volume for Windows Azure Instances Use existing NTFS APIs to access a network attached durable drive Use System.IO from .NET Benefits Move existing apps using NTFS more easily to the cloud Durability and survival of data on instance recycle A Windows Azure Drive is a NTFS VHD Page Blob Mounts Page Blob over the network as an NTFS drive Local cache on instance for read operations All flushed and unbuffered writes to drive are made durable to the Page Blob

Windows Azure Drive Capabilities A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD) Drives can be up to 1TB A Page Blob can be mounted: On one instance at a time for read/write Using read-only snapshots to multiple instances at once An instance can dynamically mount up to 16 drives Remote Access via standard BlobUI Can’t remotely mount drive. Can upload the VHD to a Page Blob using the blob interface, and then mount it as a Drive Can download the VHD to a local file and mount locally Slide Objectives Understand Drives at a high level Speaker Notes Backed by Page blobs Allows Page blob to be accessed as a drive letter on a Compute instance Read write is limited to a single instance as a time. Data is cached for reads on local instance All write flushed operations are immediately committed Notes http://blogs.msdn.com/b/windowsazure/archive/2009/11/05/introducing-the-windows-azure-content- delivery-network.aspx

How Windows Azure Drives Works VM Drive is a formatted page blob stored in blob service Mount obtains a blob lease Mount specifies amount of local storage for cache NTFS flushed/unbuffered writes commit to blob store before returning to app NTFS reads can be served from local cache or from blob store (cache miss) Application Drive X: OS Local Cache Lease Slide Objectives Understand Drives Mounting and Caching Speaker Notes A Windows Azure drive acts as a local drive mounted on the file system and is accessible to code running in a role. The data written to a Windows Azure drive is stored in a page blob defined within the Windows Azure Blob service, and cached on the local file system. Because data written to the drive is stored in a page blob, the data is Durable. Notes http://blogs.msdn.com/b/windowsazure/archive/2009/11/05/introducing-the-windows-azure-content- delivery-network.aspx Windows Azure Blob Service DemoBlob

Cloud Drive Client Library Sample CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); //Initialize the local cache for drives mounted by this role instance CloudDrive.InitializeCache(localCacheDir, cacheSizeInMB); //Create a cloud drive (PageBlob) CloudDrive drive = account.CreateCloudDrive(pageBlobUri); drive.Create(1000 /* sizeInMB */); //Mount the network attached drive on the local file system string pathOnLocalFS = drive.Mount(cacheSizeInMB, DriveMountOptions.None); //Use NTFS APIs to Read/Write files to drive … //Snapshot drive while mounted to create backups Uri snapshotUri = drive.Snapshot(); //Unmount the drive drive.Unmount(); Slide Objectives Understand Drives API Speaker Notes In Storage Client API No equivalent REST calls A Windows Azure drive may be mounted as a writable drive, or as a read-only drive if it is created from a snapshot of a page blob. To create a read-only drive, call the Snapshot method to create a new snapshot and return the snapshot's URI, then create a new instance of the CloudDrive object from the snapshot's URI and mount the drive Before a role instance mounts a drive for the first time, it must initialize the cache by calling the InitializeCache method. When a role instance mounts a writable drive, it acquires an exclusive-write lease on the associated page blob that it retains as long as the drive is mounted. If the same role instance attempts to mount a drive with the same URI a second time, the operation is ignored and the Mount method returns the local path to the existing drive. Notes http://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storageclient.clouddrive_members.aspx http://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storageclient.clouddrive.mount.aspx

Failover with Drives Must issue NTFS Flush command to persist data Use System.IO.Stream.Flush() Read/Write Drives protected with leases 1 Minute lease expiry Maintained by Windows Azure OS Driver Unmount on RoleEntryPoint.OnStop On failure Lease will timeout after 1 minute Re-mount drive on new instance Slide Objectives Understand Drives under Failure scenarios Speaker Notes All writes must be flushed to be persisted to the underlying Page Blob Read/Write drives maintain a lease Unmount drives in OnStop method of Role In failure will need to wait for lease to expire < 1 minute before remounting Notes http://social.msdn.microsoft.com/Forums/en/windowsazure/thread/5742e360-6ea9-44b4-bd59- edf4c95d5e2a

Guidance Manage connection strings/keys in cscfg Do not share keys, wrap with a service Strategy for accounts and containers You can assign a custom domain to your storage account

Windows Azure Guest Book Using Blobs C:\WindowsAzurePlatformKit\Demos\WindowsAzureGuestBookUsingBlobs demo

Q & A ponderances?

Exploring Windows Azure Storage C:\WAPTK\Labs\ExploringWindowsAzureStorageVS2010 Do Exercise 2 hands on lab time