Design considerations for storing data in the Cloud

Slides:



Advertisements
Similar presentations
Bring your own machines, connectivity, software, etc. Complete control Complete responsibility Static capabilities Upfront capital costs for the.
Advertisements

Eric Nelson Application Architect, Microsoft |
Azure Services Platform Piotr Zierhoffer. Agenda Cloud? What is Azure? Environment Basic glossary Architecture Element description Deployment.
Innovation Group Event: Cocktails & Clouds 10 th November 2011.
Protecting your online and on premises assets "Cloud Style" Mike Martin Architect / Microsoft Azure MVP.
Windows Azure for scalable compute and storage SQL Azure for relational storage for the cloud AppFabric infrastructure to connect the cloud.
Virtual techdays INDIA │ november 2010 Windows Azure Storage – Deep Dive Gaurav Mantri │ Founder, Cerebrata Software.
Working with SQL and PL/SQL/ Session 1 / 1 of 27 SQL Server Architecture.
Platform as a Service (PaaS)
Google App Engine Danail Alexiev Technical Trainer SoftAcad.bg.
Cloud Computing Systems Lin Gu Hong Kong University of Science and Technology Sept. 21, 2011 Windows Azure—Overview.
WINDOWS AZURE STORAGE 11 de Mayo, 2011 Gisela Torres – Windows Azure MVP Aventia-Renacimiento Twitter:
Cross Platform Mobile Backend with Mobile Services James
Windows Azure SQL Database and Storage Name Title Organization.
Eric Nelson Developer Evangelist Microsoft UK | Lap around.
Training Workshop Windows Azure Platform. Presentation Outline (hidden slide): Technical Level: 200 Intended Audience: Developers Objectives (what do.
Scott Zimmerman Solutions Architect, SOA/.NET/Azure/BizTalk.
Getting Started with Windows Azure Name Title Microsoft Corporation.
Larisa kocsis priya ragupathy
Austin code camp 2010 asp.net apps with azure table storage PRESENTED BY CHANDER SHEKHAR DHALL
Windows Azure Storage – Essential Cloud Storage Services Denver Cloud Computing User Group
Jimmy Narang 1. A service in the cloud has to: Be able to handle arbitrary node failures Be available all the time Be able to scale up or down on demand.
Windows Azure Tour Benjamin Day Benjamin Day Consulting, Inc.
Virtual techdays INDIA │ august 2010 SQL Azure – Tips and Tricks Ramaprasanna Chellamuthu │ Developer Evangelist, Microsoft.
Windows Azure: Microsoft’s Cloud Platform By Shahed Chowdhuri.
T.N.C.Venkata Rangan CEO, Vishwak Solutions Your Data on Cloud.
Pradeep S Pushpendra Singh Consultants, Neudesic Technologies, Hyderabad, India.
Azure Services Platform Update James Conard Sr. Director Developer & Platform Evangelism Microsoft Corporation.
Building Mobile Phone Applications With Windows Azure Nick HarrisWindows Azure Technical Evangelist Microsoft Blog:
Mobile + Cloud: Building Mobile Applications with Windows Azure Wade Wegner Blog:
Windows Azure Fundamentals Services Storage. Table of contents Overview Cloud service basics Managing cloud services Cloud storage basics Table storage.
1 Neil Kidd MTC Architect - DPE NeilKidd Neil Kidd MTC Architect - DPE NeilKidd.
Technology Drill Down: Windows Azure Platform Eric Nelson | ISV Application Architect | Microsoft UK |
(re)-Architecting cloud applications on the windows Azure platform CLAEYS Kurt Technology Solution Professional Microsoft EMEA.
Building web applications with the Windows Azure Platform Ido Flatow | Senior Architect | Sela | This session.
 Cloud Computing technology basics Platform Evolution Advantages  Microsoft Windows Azure technology basics Windows Azure – A Lap around the platform.
Bret Stateham Owner, Net Connex blogs.netconnex.com twitter.com\bstateham.
Data storage services in the cloud Matt Milner Pluralsight.
1 Gaurav Kohli Xebia Breaking with DBMS and Dating with Relational Hbase.
Platform as a Service (PaaS)
Cloud Computing for Science
Accounting for Azure in Your Data Architecture
Platform as a Service (PaaS)
Platform as a Service (PaaS)
Cloud Data platform (Cloud Application Development & Deployment)
Couchbase Server is a NoSQL Database with a SQL-Based Query Language
Amit R Bhatia / Puneeth Nayak
Developing an app for SharePoint autohosted in Azure
Using Azure Tables In this module- Learn how to store data in tables
Utilization of Azure CDN for the large file distribution
Azure and SQL Server: Getting the best out of the cloud
03 | Data Storage Bruno Terkaly | Technical Evangelist
CloudSimplified.IO.
Google App Engine Danail Alexiev
Windows Azure 講師: 李智樺, Ruddy Lee
Using cloud storage from Windows apps
Outline Virtualization Cloud Computing Microsoft Azure Platform
Windows Azure Cloud.
Code Walkthrough of a Cloud Application Running on the Windows Azure Platform
Introduction to Building Applications with Windows Azure
Saranya Sriram Developer Evangelist | Microsoft
Serpil TOK, Zeki BAYRAM. Eastern MediterraneanUniversity Famagusta
Developing for Windows Azure
MS AZURE By Sauras Pandey.
Sayed Ibrahim Hashimi Program Manager Microsoft Corporation
Building device & cloud apps
Building global and highly-available services using Windows Azure
Running PHP on the Windows Azure Platform
Making Windows Azure Relevant to IT Professionals
New, Old or Pink - it is all good
Presentation transcript:

Design considerations for storing data in the Cloud Eric Nelson Microsoft UK http://bit.ly/ericnel http://twitter.com/ericnel http://ukazure.ning.com

Session Objectives SQL Azure vs/and Windows Azure Storage Drill into Windows Azure Storage Next session: Keith Burns drills into SQL Azure

New online community http://ukazure.ning.com/

Windows Azure Platform

Windows Azure Platform Windows Azure – Your code and data SQL Azure Database – Your relational data AppFabric* – Your messages Code Data Relational Data Firewall friendly messaging between cloud and on-premise

Azure Application = WebRole + Worker Role + Storage Your Service Web Site (ASPX, ASMX, WCF) Web Site (ASPX, ASMX, WCF) Web Role (ASPX, WCF) Queues LB Internet Worker Service Worker Role LB Blobs Tables SQL Azure

Windows Azure and SQL Azure Azure Storage SQL Azure Vision Access Relational? (today) Relational? (tomorrow) Analogy

Windows Azure and SQL Azure Azure Storage SQL Azure Vision Highly scalable, highly available store in the Cloud Access Uses WCF Data Services - REST Relational? (today) No Relational? (tomorrow) Analogy

Windows Azure and SQL Azure Azure Storage SQL Azure Vision Highly scalable, highly available store in the Cloud Scalable, highly available relational store in the Cloud Access Uses WCF Data Services - REST SqlClient + TSQL (Use Azure Web Role + WCF Data Services if REST is required) Relational? (today) No Yes – but with some limitations Relational? (tomorrow) Yes – with less and less limitations Analogy

Windows Azure and SQL Azure Azure Storage SQL Azure Vision Highly scalable, highly available store in the Cloud Scalable, highly available relational store in the Cloud Access Uses WCF Data Services - REST SqlClient + TSQL (Use Azure Web Role + WCF Data Services if REST is required) Relational? (today) No Yes – but with some limitations Relational? (tomorrow) Yes – with less and less limitations Analogy File System RDBMS – as it is 

Cost impacting Architecture

Windows Azure and SQL Azure Azure Storage SQL Azure Maximum Amount of data in a single “database” Price per GB per month Price to store 1KB per month

Windows Azure and SQL Azure Azure Storage SQL Azure Maximum Amount of data in a single “database” 100TB 10GB (50GB this summer) Price per GB per month Price to store 1KB per month

Windows Azure and SQL Azure Azure Storage SQL Azure Maximum Amount of data in a single “database” 100TB 10GB (50GB this summer) Price per GB per month £0.09 £6.06 Price to store 1KB per month

Windows Azure and SQL Azure Azure Storage SQL Azure Maximum Amount of data in a single “database” 100TB 10GB (50GB this summer) Price per GB per month £0.09 £6.06 * Price to store 1KB per month £0.00

Azure Application = WebRole + Worker Role + Storage Your Service 2 Web Site (ASPX, ASMX, WCF) Web Site (ASPX, ASMX, WCF) Web Role (ASPX, WCF) Queues LB Internet Worker Service Worker Role LB Blobs Tables 1 SQL Azure

SQL Azure

+ SQL Azure http://www.mygreatsqlazureidea.com SQL Azure Database “Next big thing” http://www.sqlazurelabs.com http://www.mygreatsqlazureidea.com

SQL Azure Application Architectures SQL Azure access from within Datacenter SQL Azure access from outside Application/ Browser Application / Tools SOAP/REST HTTP/S WCF Data Svcs/REST - EF HTTP/S Web Role Windows Azure T-SQL (TDS) T-SQL (TDS) SQL Azure SQL Azure Code Near Code Far

Demo The sys.database_usage view lists the number, type, and duration of databases on the server and the sys.bandwidth_usage view describes the bandwidth used with each database. The sys.database_usage view contains the following fields: Field Definition time The date when the usage events occurred. sku The type of database, either Business for the Business Edition or Web for the Web Edition. quantity The maximum number of databases of an SKU type that existed during that day. When a database is created, the quantity is increased immediately. When a database is dropped, the quantity is reduced the next day. The sys.bandwidth_usage view contains the following fields: The hour when the bandwidth was consumed. The rows in this view are on a per-hour basis. For example, 2009-09-19 02:00:00.000 means that the bandwidth was consumed on September 19, 2009 between 2:00 A.M. and 3:00 A.M. database_name The name of the database that used bandwidth. direction The type of bandwidth that was used. Ingress is data moving into the SQL Azure database. Egress is data moving out of the SQL Azure database. class The class of bandwidth that was used. Internal is data from within the Windows Azure platform. External is data from outside the Windows Azure platform. time_period The time period when the usage occurred is either Peak or OffPeak. The Peak time is based on the region in which the server was created. For example, if a SQL Azure server was created in the "US_Northwest" region, the Peak time is defined as being between 10:00 A.M. and 6:00 P.M. PST. The amount of bandwidth, in kilobytes (KBs), that was used.

Windows Azure Storage “NoSQL”

Azure Storage Storage Queues Blobs Account Tables http://<account>.queue.core.windows.net/<queue> Blobs Account http://<account>.blob.core.windows.net/<container> Tables http://<account>.table.core.windows.net/<table>

Queues Queues Messages “Body1” “Body2” ThumbnailJobsQ “Body3”

Blobs Containers Blobs Each Blob is a list of blocks Block operation Photo1.png Photo2.png Easy to use REST Put/Get/Delete interface Can read from any Offset, Length of Blob Conditional Put and Get Blob Max Blob size 50 GB using PutBlock and PutBlockList 64 MB using PutBlob Blocks provide continuation for blob upload Put Blob/BlockList == Replace Blob for CTP Can replace an existing blob with new blob/blocks Each Blob is a list of blocks A Block can be up to 4MB each Each block can be variable size Each block has a 64 byte ID, scoped by blob name Block operation PutBlock Puts an uncommitted block defined by the block ID for the blob Block list operations PutBlockList Provide the list of blocks to comprise the readable version of the blob Can use uncommitted blocks only GetBlockList Returns the list of committed blocks (not the block data) Block ID and size of block are returned for each block Pictures Photo3.png

Content Delivery Network “Near” London “Near” Sydney CDN Node Photo1.png Containers Pictures Photo1.png Blobs Photo3.png Photo2.png Default time 72 hours Cache-control header – in put blob set header or set properties “Near” London “Near” Sydney Photo1.png Windows Azure Blob service URL: http://ericnel.blob.core.windows.net/pictures/ Photo1.png Windows Azure CDN URL: http://<guid>.vo.msecnd.net/pictures/ Photo1.png

Tables - EAV MyAzureTable1 Entity: Shiny red Car PK, RK, TS, att1, att2, att8 Entity: Blue Bird PK, RK, TS, att1, att4 Entity: 2008 Tax Return PK, RK, TS, att1, att2, att8, att9, att23, att46, att40

Windows Azure Tables Massively Scalable Tables Highly Available Billions of entities (rows) and TBs of data Automatically scales to thousands of servers as traffic grows Highly Available Can always access your data Durable Data is replicated at least 3 times

No Referential Integrity Tables - Capabilities What tables don’t do What tables can do Not relational  Cheap  No Referential Integrity  Very Scalable  If these are important to you, use SQL Azure No Joins  Flexible  Limited Queries  Durable  No Group by  Store LOTS of stuff   No Aggregations  No Transactions  No Transactions 

Demo The Query Tables operation supports the query options defined by the ADO.NET Data Services Framework. For more information, see ADO.NET Data Services Query Options (1st link below). So here are some of the things that could help you get going: You need not URI Encode your query as it will be done by application itself. Please escape single quotes with two single quotes. For example: if your partition key is a’a then use a’’a. Query table operation in Azure storage does not support SELECT clause just yet. So if you want to query for a particular condition just specify that condition. For example if you want to query your table for certain PartitionKey values, just type (PartitionKey eq 'YourPartitionKey'). Some examples of queries: (PartitionKey eq '1') (RowKey eq 'test') (Name eq 'Amit') (PartitionKey eq '1' and RowKey gt '5'), (PartitionKey eq ‘1’ and RowKey lt ‘10’ or Name eq ‘Amit’)

Summary Important: You have SQL Azure and Windows Azure Storage to take advantage of SQL Azure has a well known API and behavior Windows Azure Storage has a new API and behavior UK Site http://bit.ly/landazure UK Community http://bit.ly/ukazure / http://ukazure.ning.com My blog http://bit.ly/ericnel