Beyond SQL Azure What’s Up with Storing Data in the Cloud?                                          Beyond SQL Azure What’s Up with Storing Data in the Cloud? Key concepts with Windows Azure focus SQL Saturday #71 02-April-2011 Boston Azure User Group http://bostonazure.org @bostonazure Bill Wilder http://blog.codingoutloud.com @codingoutloud Copyright (c) 2011, Bill Wilder – Use allowed under Creative Commons license http://creativecommons.org/licenses/by-nc-sa/3.0/

Bill Wilder has been a software professional for over 20 years Bill Wilder has been a software professional for over 20 years. In 2009 he founded the Boston Azure User Group, an in-person cloud community which gets together monthly to learn about the Windows Azure platform through prepared talks and hands-on coding. Bill is a Windows Azure MVP, an active speaker, blogger (blog.codingoutloud.com), and tweeter (@codingoutloud) on technology matters and soft skills for technologists, a member of Boston West Toastmasters, and has a day job as a .NET-focused enterprise architect. Bill Wilder

Topics The Cloud SQL Azure Beyond SQL Azure

NoSQL Federation HA Cloud Sharding Cost Transparency Scale Up vs Out Concepts Cloud Sharding NoSQL Cost Transparency The cloud continues trend from differentiated hardware towards smarter software – the software-centric data center HA Scale Up vs Out Federation Multi-Tenant

Topics The Cloud SQL Azure Beyond SQL Azure

The Horse-Drawn Carriage From http://www.flickr.com/photos/historicbeaverton/2305428647/ Was shared under http://creativecommons.org/licenses/by-nd/2.0/

What Makes a Cloud a Cloud? FLEXIBLE, METERED USAGE Some ways a “Cloud” differs from traditional data center, co-lo, or ISP COMMODITY HARDWARE The cloud continues trend from differentiated hardware towards smarter software – the software-centric data center The cloud continues trend from differentiated hardware towards smarter software – the software-centric data center HOMOGENOUS, Over-PrOVISIONED, “∞” Automation, Automation, Automation Virtual Machine-BASED Generally speaking, a cloud is a pool of resources – like a smart phone, multi-tasking computer, or server farm – Think co-lo or ISP – these are the CHANGES, the next step Self-Service

___________________ as a Service Infrastructure Platform Software ___________________ as a Service Apps, $/user, LDAP, Expertise, SLA System Software OpEx, Auto Scale Out, Geo LB, Failover, HA, OS Patching, Monitoring, Monitoring, Backup, Expertise, SLA Hardware OpEx, Networking, DB/OS Licenses, Virtualization, Automation, Geo Distribution, CDN, Geo Replication, Elasticity, Managed Facility, Expertise, SLA BYO Users  Public Cloud Rental Models SaaS BYO Apps  PaaS DDTK = Dynamic Datacenter ToolKit … Dan will go into more detail on this in a few minutes Iceberg or Pyramid? BYO VMs  IaaS

Public  Hybrid  Private Public Cloud Data Center Your Company Data Center Public Cloud Hybrid Cloud Private Cloud

Topics The Cloud SQL Azure Beyond SQL Azure

SQL Azure is SQL Server Except… only SQL Azure only Full Text Search Size not limited Old school licensing model Encrypt, compress Many more… “Mirroring” Limitations Tables require a clustered index 50 GB size limit New Capabilities HA; Rental Coming: Backups & point-in-time recovery Other new Awesomeness… Common “Just change the connection string…” http://social.technet.microsoft.com/wiki/contents/articles/inside-sql-azure.aspx “Another feature in development is the ability to take control of your backups. Currently, backups are performed in the data centers to protect your data against disk or system problems. However, there is no way currently to control your own backups to provide protection against logical errors and use a RESTORE operation to return to an earlier point in time when a backup was made. The new feature involves the ability to make your own backups of your SQL Azure databases to your own on-premises storage, and the ability to restore those backups either to an on-premises database or to a SQL Azure database. Eventually Microsoft plans to provide the ability to perform SQL Azure backups across data centers and also make log backups so that point-in-time recovery can be implemented.” http://social.technet.microsoft.com/wiki/contents/articles/inside-sql-azure.aspx Additional information on Differences: http://msdn.microsoft.com/en-us/library/ff394115.aspx

SQL Azure Automagically handles… Provisioning on demand Install, configure, optimize (azurized™ versions of) “Windows Server 2008 R2” and “SQL Server 2008” Health monitoring Patching High Availability (HA) Load Balancing (LB) Billing

What’s Left for the DBA? Schema Management Account Administration Security Database Tuning: Query Optimization Indexes “The Good Stuff” … And those “other” storage options

Storage Costs $10/GB/Month + in/out DC bandwidth SQL Azure Web Edition $9.99 per database up to 1GB per month $49.95 per database up to 5GB per month SQL Azure Business Edition $99.99 per database up to 10GB per month $199.98 per database up to 20GB per month $299.97 per database up to 30GB per month $399.96 per database up to 40GB per month $499.95 per database up to 50GB per month $10/GB/Month + in/out DC bandwidth

SQL Azure Awesomeness #1 High Availability (HA) Every byte copied across 3 SQL Azure instances in datacenter (DC) Full transactional (ACID) semantics But not same as “backup” Transparent Failover if one of the three instances fails or becomes unresponsive Load balancing (LB) provided across 3 instances for improved performance

SQL Azure Awesomeness #2 Maximum Instances Size is 50 GB “Scaling Out” with Sharding has challenges “mydatabase_part1”, “mydatabase_part2”, etc. Cache coherence, connection pool management Federation Services make Sharding easier Supports shards as single “master” database “Query Fanout” makes partitions transparent Bonus: multi-tenant easier with Filtering USE FEDERATION orders_federation(tenant_id=7) http://blogs.msdn.com/b/cbiyikoglu/archive/2011/01/18/sql-azure-federations-robust-connectivity-model-for-federated-data.aspx

Public  Hybrid  Private Public Cloud Data Center Your Company Data Center Public Cloud Hybrid Cloud Private Cloud

SQL Azure Awesomeness #3 Hybrid Environment important case! Existing enterprises will not move all at once Sensitive Data concerns Connect on-prem client to SQL Azure in cloud AppFabric Feature: Windows Azure Connect Specialized, easy-to-administer secure VPN Call from Azure Cloud backwards to on-premises systems like SQL Server, AD, LoB apps & services Bandwidth and Geography matter! http://blogs.msdn.com/b/cbiyikoglu/archive/2011/01/18/sql-azure-federations-robust-connectivity-model-for-federated-data.aspx

SQL Azure Awesomeness #4 SQL Azure table access via OData Standard-based format: http://www.odata.org Atom format via RESTful http endpoints (like RSS feed) https://odata.sqlazurelabs.com/OData.svc/v0.1/s3eqxeo3ba/bostonazure.org/Events?$filter=event_id%20gt%2014 Reporting, Data Access (mobile device), feed, etc. Enabled via portal config (as db user) Currently available through SQL Azure Labs Not a released product at this time http://blogs.msdn.com/b/cbiyikoglu/archive/2011/01/18/sql-azure-federations-robust-connectivity-model-for-federated-data.aspx

Topics The Cloud SQL Azure Beyond SQL Azure

Persistent Storage Services – Options Type of Data Traditional Azure Way Relational SQL Server SQL Azure Blob File System, SQL Server Azure Blobs File File System (Azure Drives) Azure Blobs Logs File System, SQL Server, etc. Azure Tables Non-Relational NoSQL ?

Azure Cloud Platform The rise of focused, specialized services Services are really good at one thing True for cloud platforms generally “Scalable” Persistent Storage abstractions Delivered as Services Illusion of infinite scale out Same performance with x clients as with 10x as with 100x as with 1000x as with … Billing model: pay only for what you use Friction-free access to add’l or less storage

Azure Blob Storage The place to put ANY LARGE (or small) OBJECT Private or Public Public blobs can be anonymously accessed Images, videos, CSS files Need not be binary Public blobs can be cached in Azure CDN 24 locations around the world Time-limited signed-access available “You have until tomorrow to download this video” http://msdn.microsoft.com/en-us/library/ff919703.aspx – Windows Azure CDN http://blogs.msdn.com/b/windowsazure/archive/2010/08/09/20-nodes-available-globally-for-the-windows-azure-cdn.aspx – 20 CDN nodes available across the world

Azure Blob Storage Two types of Azure Blobs “Block Blob” “Page Blob” Parallel upload scenarios Resume download scenarios Up to 200 GB “Page Blob” Random access scenarios Azure Drives Up to 1000 GB http://msdn.microsoft.com/en-us/library/ff919703.aspx – Windows Azure CDN http://blogs.msdn.com/b/windowsazure/archive/2010/08/09/20-nodes-available-globally-for-the-windows-azure-cdn.aspx – 20 CDN nodes available across the world

Azure Drive Storage Disk Drive simulation in cloud Façade over an Azure Blob Specifically, Page Blob type Size Limits same as Page Blob limits 1000 GB = 1 TB Can only be mounted by a single VM at a time for read/write access Legacy-focused, not “cloud native”

File Writing Scenario Azure app wants to read & write files Could write to disk? … Not guaranteed durable, though “free” & fast Could write to Azure Drive? … Only one compute instance at a time can mount for writing Could write to Blob Storage? … Most general solution, most cloud friendly What if you wanted to store a MP3 or Video?

Not Only SQL http://www.cloudave.com/695/nosql-is-not-sql-and-thats-a-problem/ http://notonlysql.com/

NoSQL Databases Faster, Cheaper Scales Out “Simpler” , CouchDB: JSON Document Stores Amazon Dynamo, Azure Tables: Key Value Stores Many others! Faster, Cheaper Scales Out “Simpler” http://en.wikipedia.org/wiki/NoSQL http://en.wikipedia.org/wiki/CAP_theorem http://en.wikipedia.org/wiki/NoSQL

Relational Data vs. Azure Tables Approach SQL Azure Azure Tables Normalization Normalized Denormalized (Duplication) (No duplication) (Lots of duplication) Structure Schema Flexible Transactions Distributed Limited scope Responsibility DBA/Database Developer/Code Knobs Many Few Scale Up (or Sharding) Out Cost Reasonable Less expensive http://en.wikipedia.org/wiki/NoSQL

Cost: SQL Azure=67x Table Storage $10/GB/m - SQL Azure $0.15/GB/m – Tables 10/0.15=66.66 Nominally, by SQL Azure max size SQL Azure not completely full Table Storage may denormalize 100x?

Azure Table Storage Best place for granular, semi-structured data No rigid database schema No complex joins or complex transaction Fast and easy to instantiate Strongly Consistent No performance lag Programming model is WCF Data Services All data access and data updates LINQ Not SQL, not managed with SQL Server tooling

Azure Table Storage Storage Account Table [*] Entity [*, 1 MB data] PartitionKey + RowKey Property [255x] Name/Value/Type Up to 64k

Azure Table Storage Partition Key Row Key A “logical grouping” (a shard with less baggage) PartitionKey value of up to 64 KB Row Key Identify specific row within a partition RowKey value is String of up to 64 KB Table access requires Partition Key + Row Key Not to mention (cryptographic) Access Key (for the digital signing of the http header)

All Azure Storage is Replicated Resilient in case of failure “All data replicated multiple times” – Sriram Krishnan, Programming Windows Azure, p130+ Supports scale-out Replicate specific data if it gets too hot to keep up Geographic distribution of replication Currently up to you

http://www.freefoto.com/preview/04-28-6?ffid=04-28-6&k=Pile+of+Money

Pay As You Go – Storage Storage Applies equally to: $0.15 per GB stored per month $0.01 per 10,000 storage transactions Applies equally to: Queues Tables Blobs Drives http://www.microsoft.com/windowsazure/pricing/

Pay As You Go – Data Transfer North America and Europe regions $0.10 per GB in $0.15 per GB out Asia Pacific Region $0.30 per GB in $0.45 per GB out No charge for transfer within a data center http://www.microsoft.com/windowsazure/pricing/

Pay As You Go – CDN Content Delivery Network (CDN) $0.15 per GB for data transfers from European and North American locations $0.20 per GB for data transfers from other locations $0.01 per 10,000 transactions Paying dual storage costs while items in cache http://www.microsoft.com/windowsazure/pricing/

Questions? ?

BostonAzure.org Boston Azure cloud user group Focused on Microsoft’s cloud platform Last Thursday, monthly, 6:00-8:30 PM at NERD Thu April 28th – David Makogon (awesome topic) Thu May 26th – Joel Bennett (PowerShell + Azure) Food; wifi; free; great topics; growing community Join email list: http://bostonazure.org Follow on Twitter: @bostonazure

Free 30 Day Azure/SQL Azure Pass Visit http://bit.ly/BillOnAzure Use this promo code: BillOnAzure You will be provisioned an Azure account valid for 30 days that includes for FREE: Three small compute instances Two 1 GB SQL Azure Databases 3 GB of Windows Azure Storage And more…

Azure Pass Screen Shot

Contact Me I may be able to speak at your community event Bill Wilder @codingoutloud http://blog.codingoutloud.com