Design considerations for storing data in the Cloud Eric Nelson Microsoft UK http://bit.ly/ericnel http://twitter.com/ericnel http://ukazure.ning.com

Session Objectives SQL Azure vs/and Windows Azure Storage Drill into Windows Azure Storage Next session: Keith Burns drills into SQL Azure

New online community http://ukazure.ning.com/

Windows Azure Platform

Windows Azure Platform Windows Azure – Your code and data SQL Azure Database – Your relational data AppFabric* – Your messages Code Data Relational Data Firewall friendly messaging between cloud and on-premise

Azure Application = WebRole + Worker Role + Storage Your Service Web Site (ASPX, ASMX, WCF) Web Site (ASPX, ASMX, WCF) Web Role (ASPX, WCF) Queues LB Internet Worker Service Worker Role LB Blobs Tables SQL Azure

Windows Azure and SQL Azure Azure Storage SQL Azure Vision Access Relational? (today) Relational? (tomorrow) Analogy

Windows Azure and SQL Azure Azure Storage SQL Azure Vision Highly scalable, highly available store in the Cloud Access Uses WCF Data Services - REST Relational? (today) No Relational? (tomorrow) Analogy

Windows Azure and SQL Azure Azure Storage SQL Azure Vision Highly scalable, highly available store in the Cloud Scalable, highly available relational store in the Cloud Access Uses WCF Data Services - REST SqlClient + TSQL (Use Azure Web Role + WCF Data Services if REST is required) Relational? (today) No Yes – but with some limitations Relational? (tomorrow) Yes – with less and less limitations Analogy

Windows Azure and SQL Azure Azure Storage SQL Azure Vision Highly scalable, highly available store in the Cloud Scalable, highly available relational store in the Cloud Access Uses WCF Data Services - REST SqlClient + TSQL (Use Azure Web Role + WCF Data Services if REST is required) Relational? (today) No Yes – but with some limitations Relational? (tomorrow) Yes – with less and less limitations Analogy File System RDBMS – as it is 

Cost impacting Architecture

Windows Azure and SQL Azure Azure Storage SQL Azure Maximum Amount of data in a single “database” Price per GB per month Price to store 1KB per month

Windows Azure and SQL Azure Azure Storage SQL Azure Maximum Amount of data in a single “database” 100TB 10GB (50GB this summer) Price per GB per month Price to store 1KB per month

Windows Azure and SQL Azure Azure Storage SQL Azure Maximum Amount of data in a single “database” 100TB 10GB (50GB this summer) Price per GB per month £0.09 £6.06 Price to store 1KB per month

Windows Azure and SQL Azure Azure Storage SQL Azure Maximum Amount of data in a single “database” 100TB 10GB (50GB this summer) Price per GB per month £0.09 £6.06 * Price to store 1KB per month £0.00

Azure Application = WebRole + Worker Role + Storage Your Service 2 Web Site (ASPX, ASMX, WCF) Web Site (ASPX, ASMX, WCF) Web Role (ASPX, WCF) Queues LB Internet Worker Service Worker Role LB Blobs Tables 1 SQL Azure

SQL Azure

+ SQL Azure http://www.mygreatsqlazureidea.com SQL Azure Database “Next big thing” http://www.sqlazurelabs.com http://www.mygreatsqlazureidea.com

SQL Azure Application Architectures SQL Azure access from within Datacenter SQL Azure access from outside Application/ Browser Application / Tools SOAP/REST HTTP/S WCF Data Svcs/REST - EF HTTP/S Web Role Windows Azure T-SQL (TDS) T-SQL (TDS) SQL Azure SQL Azure Code Near Code Far

Demo The sys.database_usage view lists the number, type, and duration of databases on the server and the sys.bandwidth_usage view describes the bandwidth used with each database. The sys.database_usage view contains the following fields: Field Definition time The date when the usage events occurred. sku The type of database, either Business for the Business Edition or Web for the Web Edition. quantity The maximum number of databases of an SKU type that existed during that day. When a database is created, the quantity is increased immediately. When a database is dropped, the quantity is reduced the next day. The sys.bandwidth_usage view contains the following fields: The hour when the bandwidth was consumed. The rows in this view are on a per-hour basis. For example, 2009-09-19 02:00:00.000 means that the bandwidth was consumed on September 19, 2009 between 2:00 A.M. and 3:00 A.M. database_name The name of the database that used bandwidth. direction The type of bandwidth that was used. Ingress is data moving into the SQL Azure database. Egress is data moving out of the SQL Azure database. class The class of bandwidth that was used. Internal is data from within the Windows Azure platform. External is data from outside the Windows Azure platform. time_period The time period when the usage occurred is either Peak or OffPeak. The Peak time is based on the region in which the server was created. For example, if a SQL Azure server was created in the "US_Northwest" region, the Peak time is defined as being between 10:00 A.M. and 6:00 P.M. PST. The amount of bandwidth, in kilobytes (KBs), that was used.

Windows Azure Storage “NoSQL”

Azure Storage Storage Queues Blobs Account Tables http://<account>.queue.core.windows.net/<queue> Blobs Account http://<account>.blob.core.windows.net/<container> Tables http://<account>.table.core.windows.net/<table>

Queues Queues Messages “Body1” “Body2” ThumbnailJobsQ “Body3”

Blobs Containers Blobs Each Blob is a list of blocks Block operation Photo1.png Photo2.png Easy to use REST Put/Get/Delete interface Can read from any Offset, Length of Blob Conditional Put and Get Blob Max Blob size 50 GB using PutBlock and PutBlockList 64 MB using PutBlob Blocks provide continuation for blob upload Put Blob/BlockList == Replace Blob for CTP Can replace an existing blob with new blob/blocks Each Blob is a list of blocks A Block can be up to 4MB each Each block can be variable size Each block has a 64 byte ID, scoped by blob name Block operation PutBlock Puts an uncommitted block defined by the block ID for the blob Block list operations PutBlockList Provide the list of blocks to comprise the readable version of the blob Can use uncommitted blocks only GetBlockList Returns the list of committed blocks (not the block data) Block ID and size of block are returned for each block Pictures Photo3.png

Content Delivery Network “Near” London “Near” Sydney CDN Node Photo1.png Containers Pictures Photo1.png Blobs Photo3.png Photo2.png Default time 72 hours Cache-control header – in put blob set header or set properties “Near” London “Near” Sydney Photo1.png Windows Azure Blob service URL: http://ericnel.blob.core.windows.net/pictures/ Photo1.png Windows Azure CDN URL: http://<guid>.vo.msecnd.net/pictures/ Photo1.png

Tables - EAV MyAzureTable1 Entity: Shiny red Car PK, RK, TS, att1, att2, att8 Entity: Blue Bird PK, RK, TS, att1, att4 Entity: 2008 Tax Return PK, RK, TS, att1, att2, att8, att9, att23, att46, att40

Windows Azure Tables Massively Scalable Tables Highly Available Billions of entities (rows) and TBs of data Automatically scales to thousands of servers as traffic grows Highly Available Can always access your data Durable Data is replicated at least 3 times

No Referential Integrity Tables - Capabilities What tables don’t do What tables can do Not relational  Cheap  No Referential Integrity  Very Scalable  If these are important to you, use SQL Azure No Joins  Flexible  Limited Queries  Durable  No Group by  Store LOTS of stuff   No Aggregations  No Transactions  No Transactions 

Demo The Query Tables operation supports the query options defined by the ADO.NET Data Services Framework. For more information, see ADO.NET Data Services Query Options (1st link below). So here are some of the things that could help you get going: You need not URI Encode your query as it will be done by application itself. Please escape single quotes with two single quotes. For example: if your partition key is a’a then use a’’a. Query table operation in Azure storage does not support SELECT clause just yet. So if you want to query for a particular condition just specify that condition. For example if you want to query your table for certain PartitionKey values, just type (PartitionKey eq 'YourPartitionKey'). Some examples of queries: (PartitionKey eq '1') (RowKey eq 'test') (Name eq 'Amit') (PartitionKey eq '1' and RowKey gt '5'), (PartitionKey eq ‘1’ and RowKey lt ‘10’ or Name eq ‘Amit’)

Summary Important: You have SQL Azure and Windows Azure Storage to take advantage of SQL Azure has a well known API and behavior Windows Azure Storage has a new API and behavior UK Site http://bit.ly/landazure UK Community http://bit.ly/ukazure / http://ukazure.ning.com My blog http://bit.ly/ericnel