Windows Azure Storage Deep Dive Tech Ed North America 2010 4/1/2017 11:25 AM Required Slide SESSION CODE: COS310 Windows Azure Storage Deep Dive Jai Haridas Software Engineer Microsoft Corporation © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

AGENDA Introduction to Storage Storage Abstractions Summary Blobs Drives Tables Queues Summary Tribune Company Questions?

WINDOWS AZURE STORAGE Enables developers, apps and users to access cloud storage Scalable, durable, and available Anywhere at anytime access Only pay for what the service uses Easy to use REST and Client Libraries Blobs, tables, queues Existing NTFS APIs Drives

STORAGE ABSTRACTIONS Blobs – Provide a simple interface for storing named files along with metadata for the file Drives – Provides durable NTFS volumes for Windows Azure applications to use Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

WINDOWS AZURE STORAGE ACCOUNT User creates a globally unique storage account name Can choose geo-location to host storage account US – “North Central” and “South Central” Europe – “North” and “West” Asia – “East” and “Southeast” Can co-locate storage account with compute account Receive a 512 bit secret key when creating account You get two keys which can be used to rotate

BLOB STORAGE CONCEPTS Account Container Blob images videos PIC01.JPG http://<account>.blob.core.windows.net/<container>/<blobname> Account Container Blob PIC01.JPG images PIC02.JPG cohowinery videos VID1.AVI

BLOB FEATURES AND FUNCTIONS Programming Interfaces PutBlob Insert blob and Update blob GetBlob Get whole blob or a specific range DeleteBlob CopyBlob SnapshotBlob LeaseBlob Associate Metadata with Blob Standard HTTP metadata (Cache-Control, Content-Encoding, Content-Type, etc.) Metadata is <name, value> pairs, up to 8KB per blob

BLOB CLIENT LIBRARY EXAMPLE CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudBlobClient blobClient = new CloudBlobClient( account.BlobEndpoint, account.Credentials); // Create Container CloudBlobContainer cloudContainer = blobClient.GetContainerReference(containerName); bool hasCreated = cloudContainer.CreateIfNotExist(); // Access Blob in the Container CloudBlob cloudBlob = cloudContainer.GetBlobReference(blobName); //BlobRequestOptions has retry policy, timeout etc. BlobRequestOptions options = new BlobRequestOptions(); //Upload the local file to Blob service cloudBlob.UploadFile(uploadFileName, options); //Download to local file name cloudBlob.DownloadToFile(downloadFileName, options);

TWO TYPES OF BLOBS UNDER THE HOOD Block Blob Targeted at streaming workloads Each blob consists of a sequence of blocks 2 Phase commit: Blocks are uploaded and then separately committed Size limit 200GB per blob Page Blob Targeted at random read/write workloads Each blob consists of an array of pages Each page range write is committed on PUT Size limit 1TB per blob

BLOCK BLOB Blob Streaming Workload w/ Random Reads + Committed Writes Uploading a large blob Blob blobName = “blob.wmv”; PutBlock(blobName, blockId1, block1Bits); PutBlock(blobName, blockId2, block2Bits); ………… PutBlock(blobName, blockIdN, blockNBits); PutBlockList(blobName, blockId1, blockId2…,blockIdN); 10 GB Movie Block Id 1 Block Id 2 Block Id 3 Block Id N Benefit Update the blob via blocks any way you want Efficient continuation and retry Parallel and out of order upload of blocks Windows Azure Storage blob.wmv blob.wmv

PAGE BLOB – RANDOM READ/WRITE Create MyBlob Specify Blob Size = 10 GB Fixed Page Size = 512 bytes Random Access Operations PutPage[512, 2048) PutPage[0, 1024) ClearPage[512, 1536) PutPage[2048,2560) GetPageRange[0, 4096) returns valid data ranges: [0,512) , [1536,2560) GetBlob[1000, 2048) returns All 0 for first 536 bytes Next 512 bytes are data stored in [1536,2048) 10 GB Address Space 512 1024 1536 2048 2560 10 GB

WINDOWS AZURE CONTENT DELIVERY NETWORK Scenario Frequently accessed blobs Blobs accessed from many geographic locations Windows Azure Content Delivery Network (CDN) Cache and serve your Windows Azure Blobs from our network 18 locations globally (US, Europe, Asia, Australia and South America), and growing Benefit Better experience for users far from blob service Provide high-bandwidth content delivery, around the world, for popular events

WINDOWS CDN DETAILS Registering for CDN in Windows Azure Dev Portal Enable storage account to use CDN Get back a domain name and can specify your own custom domain name to use Windows Azure CDN URL: http://<id>.vo.msecnd.net/ Custom Domain Name for CDN: http://events.cohowinery.com/ Windows Azure Storage URL: http://account.blob.core.windows.net/ Using Windows Azure CDN for Blob Access Specify blobs HTTP Cache-Control policy as TTL Make blob’s container(s) publically accessible Reference blobs using CDN domain name The request is routed to our closest edge location On miss or TTL expiration blobs are retrieved from blob service and cached in the CDN Allows the blobs to be served from our CDN caches

SHARED ACCESS SIGNATURES (SIGNED URLS) Services want to distribute access to blobs, but do not want to distribute their secret key Can create a Shared Access Signature (SAS) using the secret key Then give out the SAS to provide time based access to blobs https://cohowinery.blob.core.windows.net/images/pic1.jpg? Valid time range st=Start time (optional) se=End time Two resource levels of access to grant c=Container | b=Blob Four types of permissions (or any combination) r=Read | w=Write | d=Delete | l=List Signed Identifier (optional) Allows time range and permissions to be stored in the blob service for the SAS Provides instant revocation of SAS st=2009-11-07T08:49Z &se=2009-11-07T09:49Z &sr=c &sp=rw &sig=3OSeIHP8haK%2fle9%2bBK3BX1DsdMM%3d &si=foo

BLOB TIPS For high throughput clients Set ServicePointManager.DefaultConnectionLimit to allow parallel connections to cloud service Default value is 2 Upload/Download multiple files in parallel ParallelOperationThreadCount in CloudBlobClient controls the parallelism for single blob uploading Parallel uploads used when size >= 32MB BlobRequestOptions timeout should be set as a factor of the blob size and your connection BW Timeout = Size in KB/ExpectedThroughputInKBps Use retry and exponential backoff for timeouts/server busy

SUMMARY OF WINDOWS AZURE BLOBS Blob Access Patterns Block Blobs – streaming + commit-based writes Page Blobs – random read/write Blob Operations Copy, Snapshot, and Lease work for both types Ways of Accessing and Serving Blob Content Content Delivery Network access Shared Access Signatures (Signed URLs) Custom Domain Names

STORAGE ABSTRACTIONS Blobs – Provide a simple interface for storing named files along with metadata for the file Drives – Provides durable NTFS volumes for Windows Azure applications to use Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

WINDOWS AZURE DRIVE Provides a durable NTFS volume for Windows Azure applications to use Use existing NTFS APIs to access a network attached durable drive Benefits Enables existing applications using NTFS to more easily migrate to the cloud Durability and survival of data on application failover or hardware failure A Windows Azure Drive is a Page Blob Mounts Page Blob over the network as an NTFS drive All flushed and unbuffered writes to drive are made durable to the Page Blob

WINDOWS AZURE DRIVE CAPABILITIES A Windows Azure Drive is a Page Blob formatted as a NTFS single volume Virtual Hard Drive (VHD) Drives can be up to 1TB A Page Blob can only be mounted by one VM at a time for read/write A VM can dynamically mount up to 16 drives Remote Access via Page Blob Can upload the VHD to a Page Blob using the blob interface, and then mount it as a Drive Can download the Drive through the Page Blob interface

HOW WINDOWS AZURE DRIVES WORKS VM Drive is a formatted page blob stored in blob service Mount obtains a blob lease Mount specifies amount of local storage for cache NTFS flushed/unbuffered writes commit to blob store before returning to app NTFS reads can be served from local cache or from blob store (cache miss) Application Drive X: OS Local Cache Lease Windows Azure Blob Service DemoBlob

CLOUD DRIVE CLIENT LIBRARY SAMPLE //Create Local Storage resource and initialize the local cache for drives CloudDrive.InitializeCache(localCacheDir, cacheSizeInMB); CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); //Create a cloud drive (PageBlob) CloudDrive drive = account.CreateCloudDrive(pageBlobUri); drive.Create(1000 /* sizeInMB */); //Mount the network attached drive on the local file system string pathOnLocalFS = drive.Mount(cacheSizeInMB, DriveMountOptions.None); //Use NTFS APIs to Read/Write files to drive … //Snapshot drive while mounted to create backups Uri snapshotUri = drive.Snapshot(); //Unmount the drive drive.Unmount();

WINDOWS AZURE DRIVE SUMMARY Provides VMs access to durable NTFS volumes Drives in the Cloud are only mountable by VMs within Cloud A Page Blob can only be mounted by one VM at a time for read/write access All NTFS non-buffered writes and flushes are made durable to the Page Blob Can backup and create versions of a drive with Snapshot drive Can mount a drive snapshot as a read-only VHD with many VMs at the same time

STORAGE ABSTRACTIONS Blobs – Provide a simple interface for storing named files along with metadata for the file Drives – Provides durable NTFS volumes for Windows Azure applications to use Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

WINDOWS AZURE TABLES Provides Massively Scalable Structured Storage Table can have billions of entities (rows) and TBs of data Familiar and Easy to use API WCF (ADO.NET) Data Services .NET classes and LINQ REST – with any platform or language

TABLE STORAGE CONCEPTS Account Table Entity Name =… Email = … customers Name =… Email = … cohowinery Photo ID =… Date =… winephotos Photo ID =… Date =…

ENTITY PROPERTIES Each entity can have up to 255 properties Mandatory Properties for every entity PartitionKey & RowKey Uniquely identifies an entity Defines the sort order Timestamp Optimistic Concurrency No fixed schema for rest of properties Each property is stored as a <name, typed value> pair No schema stored for a table Properties can be the standard .NET types String, binary, bool, DateTime, GUID, int, int64, and double

PARTITIONKEY AND PARTITIONS Every Table has a PartitionKey It is the first property (column) of your Table Used to group entities in the Table into partitions A Table Partition All entities in a Table with the same partition key value RowKey provides uniqueness within a partition PartitionKey is exposed in the programming model Allows application to control the granularity of the partitions and scalability of its tables

PURPOSE OF THE PARTITIONKEY Entity Locality Entities in the same partition will be stored together Efficient querying and cache locality Entity Group Transactions Atomically perform multiple Insert/Update/Delete over entities in same partition in a single transaction Table Scalability System monitors the usage patterns of partitions Automatically load balance partitions Each partition can be served by a different storage node Scale to meet the traffic needs of your table

PARTITIONS AND PARTITION RANGES PartitionKey (Category) RowKey (Title) Timestamp ReleaseDate Action Fast & Furious … 2009 The Bourne Ultimatum 2007 Animation Open Season 2 The Ant Bully 2006 Comedy Office Space 1999 SciFi X-Men Origins: Wolverine War Defiance 2008 PartitionKey (Category) RowKey (Title) Timestamp ReleaseDate Action Fast & Furious … 2009 The Bourne Ultimatum 2007 Animation Open Season 2 The Ant Bully 2006 Server B Table = Movies [Comedy - MaxKey) Server A [MinKey - Comedy) Server A Table = Movies PartitionKey (Category) RowKey (Title) Timestamp ReleaseDate Comedy Office Space … 1999 SciFi X-Men Origins: Wolverine 2009 War Defiance 2008

TABLE OPERATIONS Table Entities Create, Query, Delete Insert Update Merge – Partial update Replace – Update entire entity Delete Query Entity Group Transactions

TABLE SCHEMA Define the schema as a .NET class [DataServiceKey("PartitionKey", "RowKey")] public class Movie { /// Movie Category is the partition key public string PartitionKey { get; set; } /// Movie Title is the row key public string RowKey { get; set; } public DateTime Timestamp { get; set; } public int ReleaseYear { get; set; } public double Rating { get; set; } public string Language { get; set; } public bool Favorite { get; set; } }

TABLE CLIENT LIBRARY SAMPLE CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudTableClient tableClient = new CloudTableClient( account.TableEndpoint, account.Credentials); // Create Movie Table string tableName = “Movies“; tableClient.CreateTableIfNotExist(tableName); TableServiceContext context = tableClient.GetDataServiceContext(); // Add movie context.AddObject(tableName, new Movie("Action", “White Water Rapids Survival")); context.SaveChangesWithRetries(); // Query movie var q = (from movie in context.CreateQuery<Movie>(tableName) where movie.PartitionKey == "Action" && movie.Rating > 4.0 select movie).AsTableServiceQuery<Movie>(); foreach (Movie movieToUpdate in q) { movieToUpdate.Favorite = true; context.UpdateObject(movieToUpdate); } context.SaveChangesWithRetries( ); SaveChangesOptions.Batch

TABLE TIPS Partition your data appropriately Scale Queries Entity Group Transactions Avoid “Append only” write patterns based on PartitionKey values Avoid using monotonically increasing suffix with a constant prefix Example: using only the current timestamp as PartitionKey If needed, add varying prefix to PartitionKey

TABLE TIPS Avoid large table scans when performance is critical Restructure your schema if required Concatenate different keys to form appropriate index Most Optimal: PartitionKey == “SciFi” and RowKey == “Star Wars” Scans: Expect continuation tokens PartitionKey == “SciFi” and “Sphere” ≤ RowKey ≤ “Star Wars” “Action” ≤ PartitionKey ≤ “Thriller” PartitionKey == “Action” || PartitionKey == “Thriller” - currently scans entire table “Cars” ≤ RowKey ≤ “Star Wars” - scans entire table

TABLE TIPS Default .NET HTTP connections is set to 2 ServicePointManager.DefaultConnectionLimit = X; Use retries and exponential back-off SaveChangesWithRetries – add/update/delete AsTableServiceQuery – queries Handle Conflict errors on Inserts and NotFound errors on Delete from retry Prior try could have completed

STORAGE ABSTRACTIONS Blobs – Provide a simple interface for storing named files along with metadata for the file Drives – Provides durable NTFS volumes for Windows Azure applications to use Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

QUEUE STORAGE CONCEPTS Account Queue Message customer ID order ID http://… cohowinery order processing customer ID order ID http://…

LOOSELY COUPLED WORKFLOW WITH QUEUES Enables workflow between roles Load work in a queue Producer can forget about message once it is in queue Many workers consume the queue Worker Role Web Role Worker Role Azure Queue Input Queue (Work Items) Web Role Worker Role Web Role Worker Role

QUEUE’S RELIABLE DELIVERY Azure Queue Input Queue (Work Items) Web Role Worker Role Guarantee delivery/processing of messages (two-step consumption) Worker Dequeues message and it is marked as Invisible for a specified “Invisibility Time” Worker Deletes message when finished processing it If Worker role crashes, message becomes visible after the invisibility time for another Worker to process

QUEUE CLIENT LIBRARY SAMPLE CloudStorageAccount account = CloudStorageAccount.FromConfigurationSetting("CloudStorageAccount"); CloudQueueClient queueClient = new CloudQueueClient( account.QueueEndpoint, account.Credentials); //Create Queue CloudQueue queue = queueClient.GetQueueReference(queueName); queue.CreateIfNotExist(); //Add Message CloudQueueMessage message = new CloudQueueMessage(“some content"); queue.AddMessage(message); //Get Message message = queue.GetMessage(TimeSpan.FromMinutes(3) /*Invisibility timeout*/); // Process Message within the Invisibility Timeout //Delete Message queue.DeleteMessage(message);

QUEUE TIPS Messages can be up to 8KB Use blob to store large messages, and store blob ref in message A message may be processed more than once Make message processing idempotent Work should be repeatable and can be done multiple times Assume messages put into queue can be processed in any order For higher throughput Batch multiple work items into a single message or into a blob Use multiple queues Use DequeueCount to remove poison messages Enforce threshold on message’s dequeue count Monitor message count to dynamically increase/reduce workers

WINDOWS AZURE STORAGE TAKEAWAYS Fundamental data abstractions to build your applications Blobs – Files and large objects Drives – NTFS APIs for migrating applications Tables – Massively scalable structured storage Queues – Reliable delivery of messages Easy to use via the Storage Client Library Get going at the Windows Azure website http://www.microsoft.com/windowsazure/ More info on Windows Azure Storage at: http://blogs.msdn.com/windowsazurestorage/

CUSTOMER Tribune Company Tech Ed North America 2010 4/1/2017 11:25 AM Tribune Company Jerome Schulist Solutions Architect Tribune Company CUSTOMER © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

What is the Tribune Company?

What has the Tribune Company done? Tribune Company (LA Times, Chicago Tribune, WGN news, etc.) Reinventing themselves to stay relevant in a digital age Storage Needs 100 TB of content produced annually 25 years of digital historical content 137 years of additional history content to digitize

What is the value of Windows Azure storage? Scale elastically and dynamically (out of the disk business) Improve customer experience through reliability and improved performance Quicker to market Easy operational management

What did they build? 2 3 1 Image Processor (Queue) Content Ingestion Services 3 1 Image Processor (Worker Role) Content (Blob Storage)

Tech Ed North America 2010 4/1/2017 11:25 AM Track Resources Windows Azure, Microsoft SQL Azure, Windows Azure platform AppFabric: www.windowsazure.com/techedus Learn More: Visit the Windows Azure Boot Camp, Room 396 © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Resources Learning Required Slide www.microsoft.com/teched Tech Ed North America 2010 4/1/2017 11:25 AM Required Slide Resources Learning Sessions On-Demand & Community Microsoft Certification & Training Resources www.microsoft.com/teched www.microsoft.com/learning Resources for IT Professionals Resources for Developers http://microsoft.com/technet http://microsoft.com/msdn © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Complete an evaluation on CommNet and enter to win! Tech Ed North America 2010 4/1/2017 11:25 AM Required Slide Complete an evaluation on CommNet and enter to win! © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Sign up for Tech·Ed 2011 and save $500 starting June 8 – June 31st http://northamerica.msteched.com/registration   You can also register at the North America 2011 kiosk located at registration Join us in Atlanta next year

Tech Ed North America 2010 4/1/2017 11:25 AM © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Tech Ed North America 2010 4/1/2017 11:25 AM © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.