Training Workshop Windows Azure Platform

Presentation Outline (hidden slide): Technical Level: 300 Intended Audience: Developers Objectives (what do you want the audience to take away): 1. To be able to build and deploy a Windows Azure Service 2. To be able to use blobs, tables and queues effectively 3. Understand how to use the API for logging Presentation Outline (including demos): Compute Demo: Building and Deploying a Service Demo: Logging and Configuration Demo: Worker Roles Storage Demo: Using Blobs Demo: Using Queues Demo: Using Tables Advanced Table Concepts

3 Building Services using Windows Azure Name Title Organization

4 Windows Azure Windows Azure is the foundation of Microsoft’s Cloud Platform It is an “Operating System in the Cloud” and provides Essential Services for the cloud Virtualized Computation Scalable Storage Automatic Management Developer SDK

5 Role Programming Model Inherits RoleEntryPoint OnStart() Method Called by Fabric on startup, allows you to perform initialization tasks. Reports Busy status to load balancer until you return true. OnStop() Method Called when role is to be shutdown, graceful exit. Run() Method Main logic is here – can do anything, typically loop and never exit.

6 Web Role ASP.NET 3.5 SP1 – 64bit IIS7 Hostable Web Core Hosts Webforms or MVC FastCGI applications (e.g. PHP) Http(s)/TCP WebRole New web.roleConfig to support FastCGI Support same Role model semantics as Worker Role (e.g. OnStart, Run, etc.)

7 Packaging & Deployment ServicePackageServicePackage ServiceConfigurationServiceConfiguration

8 Building and Deploying a Service

9 Configuration Service Configuration Serviceconfiguration.csdef – Service Model ServiceConfiguration.cscfg – instance data RoleEnvironment.GetConfigurationSettingValue() Don’t use web.config for values you wish to change at runtime Web.config change requires re-deploy

10 Monitoring No Debugging in Cloud Instrument your application using Trace, Debug DiagnosticMonitorTraceListener Use Diagnostics API to Configure and Collect Event Logs Performance Counters Trace/Debug information (logging) IIS Logs, Failed Request Logs Crash Dumps or Arbitrary files Request data on demand or scheduled Transferred into your table and/or blob storage Everything is remotely configurable

11 Logging and Configuration

12 Using Queues to Decouple Roles Use Queue to Decouple Design for Idempotency multiple applications of the operation does not change the result Web Role Worker Role Storage Queue LB Worker Role Web Role Worker Role

13 Using the Worker Roles

14 Upgrading Your Application Two Models: VIP Swap and In-Place Upgrade VIP Swap: Uses Staging and Production environments. Allows to quickly swap environments. Production: v1  Staging: v2, after swap then Production: v2  Staging: v1. In-Place Upgrade Performs a rolling upgrade on live service. Entire service or a single role Manual or Automatic across update domains

15 Storage Blobs – Provide a simple interface for storing named files along with metadata for the file Drives – A durable NTFS volume backed by blob storage Tables – Provide structured storage; A Table is a set of entities, which contain a set of properties Queues – Provide reliable storage and delivery of messages for an application

16 Storage in the Dev Fabric Provides a local “Mock” storage Emulates storage in cloud Allows offline development Requires SQL Express 2005/2008

17 Storage in the cloud Create a Storage Account Endpoints Access Keys CDN

18 Windows Azure Storage Account User creates a globally unique storage account name Can choose geo-location to host storage account “US Anywhere”, “US North Central”, “US South Central”, Can co-locate storage account with compute account Receive a 256 bit secret key when creating account Storage Account Capacity at Commercial Availability Each storage account can store up to 100 TB Default limit of 5 storage accounts per subscription

19 Blob Containers Number of Blob Containers Can have has many Blob Containers that will fit within the storage account limit Blob Container A container holds a set of blobs Set access policies at the container level Private or Public accessible Associate Metadata with Container Metadata are pairs Up to 8KB per container List the blobs in a container Can now include Blob Metadata and MD5 (new)

20 Blob Features and Functions Store Large Objects (100s of GBs in size) Associate Metadata with Blob Metadata is pairs, Up to 8KB per blob Set/Get with or separate from blob data bits Standard REST Interface PutBlob Inserts a new blob, overwrites the existing blob GetBlob Get whole blob or a specific range DeleteBlob CopyBlob (new) SnapshotBlob (new) LeaseBlob (new)

21 Blocks or Pages Blob Storage Concepts Key concepts account, container, blob and blocks/pages BlobContainerAccount Pictures IMG001.JPG IMG002.JPG MoviesMOV1.AVI Block/Page 1 Block/Page 2 Block /Page 3

22 Two Types of Blobs Under the Hood Block Blob Targeted at streaming workloads Each blob consists of a sequence of blocks Each block is identified by a Block ID Size limit 200GB per blob Page Blob (new) Targeted at random read/write workloads Each blob consists of an array of pages Each page is identified by its offset from the start of the blob Size limit 1TB per blob

23 Block Blob Details Block can be up to 4MB each Each block can be variable size Each block has a 64 byte ID Scoped by blob name and stored with the blob Block operation PutBlock Puts an uncommitted block defined by the block ID for the blob Block List Operations PutBlockList Provide the list of blocks to comprise the readable version of the blob Can use blocks from uncommitted or committed list to update blob GetBlockList Returns the list of blocks, committed or uncommitted for a blob Block ID and Size of Block is returned for each block

24 Page Blob Details Page Blob is created with a Max Blob Size Can change the max size of the blob at anytime. Address space is broken up into fixed sized 512 byte pages for updates Page update operations – must be page aligned PutPage - limited to 4MB Overwrite range of pages starting at the specified offset ClearPage - can specify up to max size of blob Clear range of pages at the offset Reading a Page Blob GetBlob – can read from any byte offset for any valid range What parts of the Page Blob have stored pages in them GetPageRange Get valid page ranges in the blob Only charged for pages with data stored in them

25 Choosing Between Block and Page Blob Block Blob Targeted at streaming workloads Update semantics Upload a bunch of blocks. Then commit change. Concurrency: ETag Checks Page Blob Targeted at random read/write workloads Update Semantics Immediate update Concurrency: Leases

26 Using Blobs

27 Summary Of Windows Azure Blobs Blob Access Patterns Block Blobs – streaming Page Blobs – random read/write (new) New Blob Operations Copy, Snapshot, and Lease work for both types New Ways of Accessing and Serving Blob Content (see MSDN) Content Delivery Network access Custom Domain Names Root Blob Container Shared Access Signatures Additional new features ListBlob, GetBlob, Blob Properties (see MSDN docs)

28 Queues Simple asynchronous dispatch queue Create and delete queues Message: Retrieved at least once Max size 8kb Operations: put get delete

29 Queue Storage Concepts Account, queue and message MessageQueueAccount Thumbnail Jobs 128x128, 256x256, Indexing Jobs

30 Using Queues

31 Summary Of Queues Provide reliable message delivery Allows Messages to be retrieved and processed at least once No limit on number of messages stored in a Queue Message size is <=8KB

32 Tables Entities and properties (rows & columns) Tables scoped by account Designed for billions+ Scale-out using partitions Partition key & row key Operations performed on partitions Efficient queries No limit on number of partitions Use ADO.NET Data Services

33 Table Storage Concepts Account, table and entity EntityTableAccount Users Name=…hash=… PhotoIndex Tag=…id=… Tag=…,id=…

34 Entities and Properties Each Entity can have up to 255 properties Every Entity has fixed key properties Partition key Row key Timestamp No fixed schema for rest of properties 2 entities in the same table can have different properties Properties stored as pairs Each entity has a system maintained version

35 Property Types Partition key and Row key String (up to 1KB) Other properties String (up to 64KB) Binary (up to 64KB) Bool DateTime GUID Int Int64 Double

36 Partition Key And Partition Every Entity has a partition key All entities in a table with the same partition key value live in the same partition Need to choose partitioning scheme to make data access scalable

37 Partitioning Guidelines Performance Use a PartitionKey that is common in your queries Always try to specify the partition key in the query Entities with same partition key value are clustered Batch capabilities and transaction are supported within a partition only. Scalability We monitor partition traffic Automatically load balance partitions Each partition can potentially be served by a different storage node Scale to meet the traffic needs of your application More partitions – makes it easier to balance load

38 Using Tables

Client A ClientB 5 : Ch9, Jan-1, 3 1 : Ch9, Jan-2, 2 2: Ch9, Jan-2, 5 Use standard HTTP mechanisms – Etag and If-Match Get entity – get system maintained version as ETag Update Entities Locally – change rating Send Update with version check - IF-Match with Etag Success if version matches, and update version on Client-A Precondition failed (412) if version does not match Concurrent Updates 9 : Ch9, Jan-3, 6 If-Match: 1 Ch9, Jan-2, 5 If-Match: 1 Ch9, Jan-2, 4 VersionRating 1: Ch9, Jan-2, 4 1: Ch9, Jan-2, 5 Error: 412 2: Ch9, Jan-2, 5

.NET: LINQ Take(N) function Getting the Top N entities serviceUri = new Uri(" DataServiceContext context = new DataServiceContext(serviceUri); var allMessages = context.CreateQuery ("Messages"); foreach (Message message in allMessages.Take(100)) { Console.WriteLine(message.Name); } GET /Messages?$top=100

41 GET /Messages?$filter=...&$top=100 &NextPartitionKey=xxxxxxx &NextRowKey=yyyyyy Pagination – Continuation Tokens Send a request GET /Messages?$filter=...&$top=100 x-ms-continuation-NextPartitionKey: xxxxxxx x-ms-continuation-NextRowKey: yyyyyy x-ms-continuation-NextPartitionKey: xxxxxxx x-ms-continuation-NextRowKey: yyyyyy Messages Ch9, Date1, Ch9, Date2, Ch9, … Ch9,Date100, Ch9,Date101, Ch9, … 100

42 Single Table Consistency ACID transactions for single entity CUD Insert/update/delete Snapshot isolation for query within a single partition Consistent view from start time of the query No dirty (uncommitted) reads Does not block concurrent updates No snapshot isolation across partitions No snapshot isolation across different continuations of a query Batch transactions within partition only

43 Cross Table Consistency Application is responsible for maintaining consistency Example When a channel is deleted, delete all the messages for that channel Failures can occur in the middle Example - Application fails after deleting some messages Use Windows Azure Queues to help ensure completion of operation

44 1. Dequeue DelCh12. Delete Ch1 from Channels3. Delete from Messages4. Delete queue entry Messages Channels Queue Worker Cross Table Consistency Del Ch1 2 2 Delete channelDelete messages worker Front end Front End Del Ch5 Del Ch11 Ch1, Msg1 Ch1, Msg2 Ch1, Msg3 Ch2, Msg1 Ch2, Msg2 Ch3, Msg1 Ch1,… Ch2,… Del Ch1

45 1. Dequeue DelCh1 and start delete2. Fails after deleting Ch1 and Msg13. DelCh1 is visible again4. Dequeue DelCh1 again5. Repeat delete operations Messages Channels Queue Worker 1 Resuming After Failure Del Ch1 2 2 Delete channelDelete messages worker Front end Front End Del Ch5 Del Ch11 Ch1, Msg1 Ch1, Msg2 Ch1, Msg3 Ch2, Msg1 Ch2, Msg2 Ch3, Msg1 Ch1,… Ch2,… Worker2 Del Ch1 Del Ch1h

46 Table Summary Windows Azure tables are Massively Scalable Highly Available Simple familiar API Use.NET –- ADO.NET Data Services and LINQ Or use REST Leverage your.NET expertise

47 Learning Windows Azure Download the SDK You don’t need cloud access to develop! Look at the samples in the SDK Windows Azure Platform Training Kit 3 Windows Azure labs Follow the team bloggers

48

49 © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.