1. Windows Azure Storage – Essential Cloud Storage Services Denver Cloud Computing User Group 1.13.2009

2. Administravia New website: http://www.azureusergroup.com/ http://www.azureusergroup.com/ New location starting next month – Details to follow in invite for next meeting and on website

3. Agenda Discuss Azure Cloud Storage Services – Blobs – Tables – Queues Demos of each

4. Windows Azure Storage The goal is to allow users and applications to: – Access their data efficiently from anywhere at anytime – Store data for any length of time – Scale to store any amount of data – Be confident that the data will not be lost – Pay for only what they use/store

5. Windows Azure Storage Storage that is – Durable – Scalable (capacity and throughput) – Highly Available – Security – Performance Efficient Rich Data Abstractions – Service communication: queues, locks, … – Large user data items: blobs, blocks, … – Service state: tables, caches, … Simple and Familiar Programming Interfaces – REST Accessible and ADO.NET

6. Windows Azure Storage Account User creates a globally unique storage account name – Receive a 256 bit secret key when creating account Provides security for accessing the store – Use secret key to create a HMAC SHA256 signature for each request – Use signature to authenticate request at server

7. SETTING UP CLOUD STORAGE

8. Fundamental Data Abstractions Blobs – Provide a simple interface for storing named files along with metadata for the file Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

9. Blob Storage Concepts BlobContainerAccount sally pictures IMG001.JPG IMG002.JPG movies MOV1.AVI

10. Storage Account And Blob Containers Storage Account – An account can have many Blob Containers Container – A container is a set of blobs – Sharing policies are set at the container level Public READ or Private – Associate Metadata with Container Metadata is pairs Up to 8KB per container – List the blobs in a container BlobContainerAccount sally pictures IMG001.JPG IMG002.JPG movies MOV1.AVI

11. Blob Namespace Blob URL http://.blob.core.windows.net/ / Example: Account – sally Container – music BlobName – rock/rush/xanadu.mp3 URL: http://sally.blob.core.windows.net/music/rock/rush/xanadu.mp3 http://sally.blob.core.windows.net/music/rock/rush/xanadu.mp3

12. Blob Features And Functions Store Large Objects (up to 50 GB each) Standard REST PUT/GET Interface http://.blob.core.windows.net/ / – PutBlob Inserts a new blob or overwrites the existing blob – GetBlob Get whole blob or by starting offset, length – DeleteBlob – Support for Continuation on Upload Associate Metadata with Blob – Metadata is pairs – Set/Get with or separate from blob data bits – Up to 8KB per blob

13. Continuation On Upload Scenario Want to upload a large multi GB file into the cloud If upload fails in the middle, need an efficient way to resume the upload from where it failed

14. Uploading A Blob Via Blocks Uploading a Large Blob 10 GB Movie Windows Azure Storage Block Id 1 Block Id 2 Block Id 3 Block Id N blobName = “TheBlob.wmv”; PutBlock(blobName, blockId1, block1Bits); PutBlock(blobName, blockId2, block2Bits); ………… PutBlock(blobName, blockIdN, blockNBits); PutBlockList(blobName, blockId1,…,blockIdN); blobName = “TheBlob.wmv”; PutBlock(blobName, blockId1, block1Bits); PutBlock(blobName, blockId2, block2Bits); ………… PutBlock(blobName, blockIdN, blockNBits); PutBlockList(blobName, blockId1,…,blockIdN); TheBlob.wmv Benefit: Efficient continuation and retry Parallel and out of order upload of blocks

15. PutBlockList Example Sequence of Operations – PutBlock(BlockId1) – PutBlock(BlockId3) – PutBlock(BlockId4) – PutBlock(BlockId2) – PutBlock(BlockId4) – PutBlockList(BlockId2, BlockId3, BlockId4) Block Id 1Block Id 3Block Id 2 BlobName = ExampleBlob.wmv Block Id 4 Block Id 2Block Id 3 Block Id 4 Committed and readable version of blob Example Uploading Blocks Out of Order Same Block IDs Unused Blocks

16. Blob Storage Concepts – Adding Blocks BlockBlobContainerAccount sally pictures IMG001. JPG IMG002. JPG movies MOV1.AVI Block 1 Block 2 Block 3

17. Blob As A List Of Blocks Blob – Consists of a List of Blocks Properties of Blocks – Each Block defined by a Block ID Up to 64 Bytes, scoped by Blob Name – Blocks are immutable – A block is up to 64MB Do not have to be same size

18. BlockList Operations PutBlockList for a Blob – Provide the list of blocks to comprise the readable version of the blob – If multiple blocks are uploaded with same Block ID – Last committed block wins – Blocks not used will be garbage collected GetBlockList for a Blob – Returns the list of blocks that represent the readable (committed) version of the blob Block ID and Size of Block is returned for each block

19. Blob Storage

20. REST PutBlock PUT http://dvd.blob.core.windows.net/movies/TheBlob.wmv ?comp=block &blockid=BlockId1 &timeout=60 HTTP/1.1 Content-Length: 4194304 Content-MD5: HUXZLQLMuI/KZ5KDcJPcOA== Authorization: SharedKey dvd: F5a+dUDvef+PfMb4T8Rc2jHcwfK58KecSZY+l2naIao= x-ms-date: Mon, 27 Oct 2008 17:00:25 GMT ……… Block Data Contents ……… AccountContainer Blob Name

21. REST PutBlockList PUT http://dvd.blob.core.windows.net/movies/TheBlob.wmv ?comp=blocklist &timeout=120 HTTP/1.1 Content-Length: 161213 Authorization: SharedKey dvd: QrmowAF72IsFEs0GaNCtRU143JpkflIgRTcOdKZaYxw= x-ms-date: Mon, 27 Oct 2008 17:00:25 GMT BlockId1 BlockId2 ……………… AccountContainer Blob Name

22. REST GetBlob GET http://dvd.blob.core.windows.net/movies/TheBlob.wmv HTTP/1.1 Authorization: SharedKey dvd: RGllHMtzKMi4y/nedSk5Vn74IU6/fRMwiPsL+uYSDjY= X-ms-date: Mon, 27 Oct 2008 17:00:25 GMT Get whole blob GET http://dvd.blob.core.windows.net/movies/TheBlob.wmv ?timeout=60 HTTP/1.1 Range: bytes=1024000-2048000 Get a range of the blob

23. Blob Enumeration – Delimiter Enumerates the list of blobs in a container – Hierarchical listing: prefix + delimiter – Continuation: NextMarker + Marker Container “movies” has: Action/Rocky.avi Action/Rocky2.avi Drama/Crime/GodFather.avi Drama/Crime/GodFather2.avi Drama/LordOfRings.avi Thriller/TheBlob.wmv List top level “directories” REST Request: GET http://dvd.blob.windows.net/movies ?comp=list &delimiter=/ Results: Action Drama Horror Container “movies” has: Action/Rocky1.wmv Action/Rocky2.wmv Action/Rocky3.wmv Action/Rocky4.wmv Action/Rocky5.wmv Drama/Crime/GodFather1.wmv Drama/Crime/GodFather2.wmv Drama/Memento.wmv Horror/TheBlob.wmv

24. Blob Enumeration – Prefix Enumerates the list of blobs in a container – Hierarchical listing: prefix + delimiter – Continuation: NextMarker + Marker Container “movies” has: Action/Rocky1.avi Action/Rocky2.avi ……… Action/Rocky5.avi Drama/Crime/GodFather1.avi Drama/Crime/GodFather2.avi Drama/Gladiator.avi Horror/TheBlob.wmv List directory “Drama”: REST Request: GET http://dvd.blob.windows.net/movies ?comp=list &prefix=Drama/ &delimiter=/ Results: Drama/Crime Drama/Memento.wmv Container “movies” has: Action/Rocky1.wmv Action/Rocky2.wmv Action/Rocky3.wmv Action/Rocky4.wmv Action/Rocky5.wmv Drama/Crime/GodFather1.wmv Drama/Crime/GodFather2.wmv Drama/Memento.wmv Horror/TheBlob.wmv

25. Container “movies” has: Action/Rocky1.wmv Action/Rocky2.wmv Action/Rocky3.wmv Action/Rocky4.wmv Action/Rocky5.wmv Drama/Crime/GodFather1.wmv Drama/Crime/GodFather2.wmv Drama/Memento.wmv Horror/TheBlob.wmv Blob Enumeration – MaxResults Enumerates the list of blobs in a container – Hierarchical listing: prefix + delimiter – Continuation: NextMarker + Marker Max Results and Next Marker REST Request: GET http://dvd.blob.windows.net/movies ?comp=list &prefix=Action &maxresults=3 Results: Action/Rocky1.wmv Action/Rocky2.wmv Action/Rocky3.wmv OpaqueMarker1

26. Blob Enumeration – Continuation Enumerates the list of blobs in a container – Hierarchical listing: prefix + delimiter – Continuation: NextMarker + Marker Results: Action/Rocky4.wmv Action/Rocky5.wmv Container “movies” has: Action/Rocky1.wmv Action/Rocky2.wmv Action/Rocky3.wmv Action/Rocky4.wmv Action/Rocky5.wmv Drama/Crime/GodFather1.wmv Drama/Crime/GodFather2.wmv Drama/Memento.wmv Horror/TheBlob.wmv Using Continuation Marker REST Request: GET http://dvd.blob.windows.net/movies ?comp=list &prefix=Action &maxresults=3 &marker=OpaqueMarker1

27. Summary Of Windows Azure Blobs Easy to use REST Put/Get/Delete interface Can read from any Offset, Length of Blob Conditional Put and Get Blob Max Blob size – 50 GB using PutBlock and PutBlockList – 64 MB using PutBlob Blocks provide continuation for blob upload Put Blob/BlockList == Replace Blob for CTP – Can replace an existing blob with new blob/blocks

28. Future Windows Azure Blob Support Update Blob – Ability to replace, add, or remove blocks from a blob Append Blob – Ability to append a block to a blob Copy Blob – Ability to copy an existing blob to a new blob name

29. Fundamental Data Abstractions Blobs – Provide a simple interface for storing named files along with metadata for the file Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

30. Windows Azure Tables Provides Structured Storage – Massively Scalable Tables Billions of entities (rows) and TBs of data Automatically scales to thousands of servers as traffic grows – Highly Available Can always access your data – Durable Data is replicated at least 3 times Familiar and Easy to use Programming Interfaces – ADO.NET Data Services –.NET 3.5 SP1.NET classes and LINQ REST - with any platform or language

31. Table Data Model Table – A Storage Account can create many tables – Table name is scoped by Account Data is stored in Tables – A Table is a set of Entities (rows) – An Entity is a set of Properties (columns) Entity – Two “key” properties that together are the unique ID of the entity in the Table PartitionKey – enables scalability RowKey – uniquely identifies the entity within the partition

32. Partition Key And Partitions Every Table has a Partition Key – It is the first property (column) of your Table – Used to group entities in the Table into partitions A Table Partition – All entities in a Table with the same partition key value Partition Key is exposed in the programming model – Allows application to control the granularity of the partitions and enable scalability

33. Partition Key Document Name Row Key Version Property 3 Modification Time …..Property N Description Examples DocV1.08/2/2007…..Committed version Examples DocV2.0.19/28/2007Alice’s working version FAQ DocV1.05/2/2007Committed version FAQ DocV1.0.17/6/2007Alice’s working version FAQ DocV1.0.28/1/2007Sally’s working version Partition Example Table Partition - all entities in table with same partition key value Application controls granularity of partition Partition 1 Partition 2

34. Purpose Of The Partition Performance and Entity Locality – Entities in the same partition will be stored together Efficient querying and cache locality Table Scalability – We monitor the usage patterns of partitions – Automatically load balance partitions Each partition can be served by a different storage node Scale to meet the traffic needs of your table

35. Performance And Scalability Partition Key Document Name Row Key Version Property 3 Modification Time …..Property N Description Examples DocV1.08/2/2007…..Committed version Examples DocV2.0.19/28/2007Alice’s working version FAQ DocV1.05/2/2007Committed version FAQ DocV1.0.17/6/2007Alice’s working version FAQ DocV1.0.28/1/2007Sally’s working version Partition 1 Partition 2 Efficient retrieval of all of the versions of FAQ Doc – Since we are accessing a single partition The two partitions can be served from different servers to scale out access

36. Choosing A Partition Key Use a PartitionKey that is common in your queries – If Partition Key is part of Query Fast access to retrieve entities within a single partition – If Partition Key is not specified in a Query Then every partition has to be scanned

37. Partition Key Document Name Row Key Version Property 3 Modification Time …..Property N Description Examples DocV1.08/2/2007…..Committed version Examples DocV2.0.19/28/2007Alice’s working version FAQ DocV1.05/2/2007Committed version FAQ DocV1.0.17/6/2007Alice’s working version FAQ DocV1.0.28/1/2007Sally’s working version Query With And Without PartionKey Partition 1 Partition 2 Getting all entities with (PartitionKey ==“FAQ Doc”) is fast – Access single partition Get all docs with (ModifiedTime < 6/01/2007) is more expensive – Have to traverse all partitions

38. Choosing A Partition Key Use a PartitionKey that is common in your queries – If Partition Key is part of Query Fast access to retrieve entities within a single partition – If Partition Key is not specified in a Query Then every partition has to be scanned Spread out load across partitions – Partition Key allows you to control what goes into your Table partitions – More partitions – makes it easier to automatically balance load At the tradeoff of Entity Locality

39. Primary Key For Entity Partition Key Document Name Row Key Version Property 3 Modification Time …..Property N Description Examples DocV1.08/2/2007…..Committed version Examples DocV2.0.19/28/2007Alice’s working version FAQ DocV1.05/2/2007Committed version FAQ DocV1.0.17/6/2007Alice’s working version FAQ DocV1.0.28/1/2007Sally’s working version Partition 1 Partition 2 Primary Key for the Entity is the composite of – PartitionKey Unique ID of the partition the entity belongs to within the Table – RowKey Unique ID for the entity within the partition

40. Entities And Properties Each Entity can have up to 255 properties Mandatory Properties for every Entity in Table – Partition Key – Row Key All entities have a system maintained version No fixed schema for rest of properties – Each property is stored as a pair No schema stored for a table – 2 entities within the same table can have different properties

41. Property Types Supported Partition Key and Row Key – String (up to 64KB) Property Types – String (up to 64KB) – Binary (up to 64KB) – Bool – DateTime – GUID – Int – Int64 – Double

42. Table Programming Model Provide familiar and easy to use interfaces – Leverage your.NET expertise Table Entities are accessed as objects via ADO.NET Data Services –.NET 3.5 SP1 – LINQ – Language Integrated Query – RESTful access to table and entities Insert/Update/Delete Entities over the Table Query over Tables – Get back a list of structured Entities

43. Example Table Definition Example using ADO.NET Data Services Table Entities are represented as Class Objects [DataServiceKey("PartitionKey", "RowKey")] public class Customer { // Partition key – Customer Last name public string PartitionKey { get; set; } // Row Key – Customer First name public string RowKey { get; set; } // User defined properties here public DateTime CustomerSince { get; set; } public double Rating { get; set; } public string Occupation { get; set; } }

44. Create Customers Table Every Account has a master table called “Tables” – It is used to keep track of the tables in your account – To use a table it has to be inserted into “Tables” [DataServiceKey("TableName")] public class TableStorageTable { public string TableName { get; set; } } TableStorageTable table = new TableStorageTable("Customers"); context.AddObject("Tables", table); DataServiceResponse response = context.SaveChanges(); // serviceUri is “http://.table.core.windows.net/” DataServiceContext context = new DataServiceContext(serviceUri);

45. Table Storage

46. Create And Insert Entity Create a new Customer and Insert into Table Customer cust = new Customer( “Lee”, // Partition Key = Last Name “Geddy”, // Row Key = First Name DateTime.UtcNow, // Customer Since 2.0, // Rating “Engineer” // Occupation); context.AddObject(“Customers”, cust); DataServiceResponse response = context.SaveChanges(); // Service Uri is “http://.table.core.windows.net/” DataServiceContext context = new DataServiceContext(serviceUri);

47. Query A Table LINQ DataServiceContext context = new DataServiceContext(“http://myaccount.table.core.windows.net”); var customers = from o in context.CreateQuery (“Customers”) where o.PartitionKey == “Lee” select o; foreach (Customer customer in customers) { } GET http://myaccount.table.core.windows.net/Customers? $filter= PartitionKey eq ‘Lee’

48. Update And Delete Entity context.DeleteObject(cust); DataServiceResponse response = context.SaveChanges(); cust.Occupation = “Musician”; context.UpdateObject(cust); DataServiceResponse response = context.SaveChanges(); Customer cust = ( from c in context.CreateQuery (“Customers”) where c.PartitionKey == “Lee” // Partition Key = Last Name && c.RowKey == “Geddy” // Row Key = First Name select c).FirstOrDefault();

49. Summary Of Windows Azure Tables Built to provide Massively Scalable, Highly Available and Durable Structured Storage Automatic Load Balancing and Scaling of Tables – Partition Key is exposed to the application Familiar and Easy to use LINQ and REST programming interfaces – ADO.Net Data Services Not a “relational database” – No joins, no maintenance of foreign keys, etc

50. Future Windows Azure Table Support At CTP – Single Index Query and retrieve results sorted by PartitionKey and RowKey – Single Entity Transactions Atomically Insert, Update, or Delete a Single Entity Future Support for – Secondary Indexes Query and retrieve results sorted by other properties – Entity Groups Atomic transactions across multiple entities within same partition

51. Fundamental Data Abstractions Blobs – Provide a simple interface for storing named files along with metadata for the file Tables – Provide structured storage. A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

52. Web + Worker Queue Example Cloud Storage (blob, table, queue) Web Role LB n Worker Role m

53. Windows Azure Queues Provide reliable message delivery – Simple, asynchronous work dispatch – Programming semantics ensure that a message can be processed at least once Queues are Highly Available, Durable and Performance Efficient Access is provided via REST

54. Account, Queues And Messages An Account can create many Queues – Queue Name is scoped by the Account A Queue contains Messages – No limit on number of messages stored in a Queue But a Message is stored for at most a week http://.queue.core.windows.net/ Messages – Message Size <= 8 KB – To store larger data, store data in blob/entity storage, and the blob/entity name in the message

55. Queue Programming API Queues – Create/Clear/Delete Queues – Inspect Queue Length Messages – Enqueue (QueueName, Message) – Dequeue (QueueName, Invisibility Time T) Returns the Message with a MessageID Makes the Message Invisible for Time T – Delete(QueueName, MessageID)

56. 2 2 1 1 C1C1 C1C1 C2C2 C2C2 Dequeue And Delete Messages 1 1 2 2 3 3 4 4 Producers Consumers P2P2 P2P2 P1P1 P1P1 3 3 2. Dequeue(Q, 30 sec)  msg 2 1. Dequeue(Q, 30 sec)  msg 1 1 1 2 2

57. C1C1 C 1 C2C2 C2C2 Dequeue And Delete Messages 3 4 4 Producers Consumers P2P2 P2P2 P1P1 P1P1 1 1 2 2 2. Dequeue(Q, 30 sec)  msg 2 3. C2 consumed msg 2 4. Delete(Q, msg 2) 7. Dequeue(Q, 30 sec)  msg 1 1. Dequeue(Q, 30 sec)  msg 1 5. C 1 crashed 1 1 2 2 1 1 6. msg1 visible 30 seconds after Dequeue Benefit: Insures that every message can be processed at least once 3 3

58. Queue Storage

59. Summary Of Windows Azure Queues Provide reliable message delivery – Allows Messages to be retrieved and processed at least once No limit on number of messages stored in a Queue Message size is <=8KB

60. DURABILITY, AVAILABILITY AND SCALABILITY OF WINDOWS AZURE STORAGE

61. Storage Durability All data is replicated at least 3 times – Replicas are spread out over different fault and upgrade domains in same data center or geo-distribution and geo-replication – All of Storage (Blobs, Tables and Queues) is built on this replication layer Dynamic replication to maintain a healthy number of replicas – Recover from a lost/unresponsive Drive or Node – Recover from data bit rot Data continuously scanned against bit rot

62. Availability And Scalability Efficient Failover – Data served immediately elsewhere within data center from available replicas Automatic Load Balancing of Hot Data – Monitor the usage patterns and load balance access to Blob Containers, Table Partitions and Queues – Distribute access to the hot data over the data center according to traffic Caching of Hot Blobs, Entities and Queues – Hot Blobs are cached to scale out access to them – Hot Entity and Queue data pages are cached and served from memory

63. Windows Azure Storage Summary Essential Storage for the Cloud – Durable, Scalable, Highly Available, Security, Performance Efficient Familiar and Easy to Use Programming Interfaces – REST Accessible, LINQ and ADO.NET Rich Data Abstractions – Service communication: queues, locks, … – Large user data items: blobs, blocks, … – Service state: tables, caches, …

64. Account Container Blobs Table Entities Queue Messages Azure Data Storage Concepts http://.blob.core.windows.net/ http://.table.core.windows.net/ http://.queue.core.windows.net/

65. QUESTIONS?