Scaling Out Microsoft Azure SQL Databases Azure SQL Database supports three methods to implement sharding: Elastic Scale, Custom Sharding and Federations This session is a comparison and overview of these approaches, to be possible to choose the best one. Mihail Mateev, Solutions Architect, PM and Senior Technical Evangelist, TL @ Strypes

Say Thank you to Volunteers: They spend their FREE time to give you this event. Because they are love our COMMUNITY. Because they want YOU to learn from the BEST IN THE WORLD. If you see one of these guys in the hall – buy them a beer/wine, they deserve it.

Vitaliy Popovych

Olena Smoliak

Maksim Garnets

Yevhen Nedashkivskyi

Oksana Borysenko

3 Sponsor Sessions at 12:30 Don’t miss them, they might be providing some interesting and valuable information! Bakotech (Dell) – Room “AE” Intapp – Room “F” DevArt – Room “Lazurny”

Our Sponsors:

Scaling Out Microsoft Azure SQL Databases Welcome! Ласкаво просимо! Добро пожаловать!

About me Mihail Mateev is a Solutions Architect @ Strypes / ICT Automatisering, Community enthusiast, PASS RM for CEE and chapter lead, Microsoft Azure MVP Mihail works in various areas related to Microsoft technologies : Silverlight, WPF, WP, LightSwitch, WCF, ASP.Net MVC, MS SQL Server and Microsoft Azure

Agenda Approaches to Scaling Out Introduction to Azure Federations Custom Sharding Elastic Scale Introduction to Azure Federations Azure SQL Database Elastic Scale Recommendations for Scaling Out

DB Scaling solutions: Scale UP vs. Scale OUT Background The number of clients or sales or licenses grows over time as a linear function We have a linear function for number of DB Transactions, and amount of data The query response time grows exponentially

DB Scaling solutions: Scale UP vs. Scale OUT Scale UP improving the hardware solution of the DB faster processor more RAM faster Hard Drive Scale OUT (assume some amount of re-work) Table partitioning Master/Slave configurations Cluster Computing Sharding

DB Scaling solutions: Scale OUT Table partitioning: The large tables are split into two or more physically separate partitions The queries will end up hitting two separate tables. The tables can also be located in different DB

DB Scaling solutions: Scale OUT Master/Slave configurations We have one database which is called “Master” and multiple databases called “Slaves” The Master DB is set up for read/write access Slave databases are read-only. There is a data replication between the Master and the Slaves

DB Scaling solutions: Scale OUT Cluster Computing: Similar to the Master/Slave in the DB Cluster  we have one database node which is a DB available for read/write and multiple nodes which are in the read-only mode All data updates are received by the “Master” node and replicated into the read-only nodes The tables can also be located in different DB When Master DB is failed one of the read-only nodes becomes the new “Master” Sharding A solution in which the application operates with “Shards” which can be physically separate databases. There are queries which run inside the shard and queries which are distributed across multiple shardsets

DB Scaling solutions: Scale OUT Sharding

DB Scaling solutions: Scale OUT Sharding Example:

DB Scaling solutions: Scale OUT Sharding : Records are grouped by a particular key, known as a Sharding Key Records that are associated with a particular Sharding Key are known as a Shardlet The different databases that store the collections of Shardlets are known as Partitions

Why to Scale-Out your database Scaling out your business is important NoSQL and Scale-Out Sharding / SQL Azure Federations

Scaling out your business Social networking: the business problem 100s of million of users 10s of million of users concurrently Terabytes to petabytes of data (Structured and unstructured ) Required (eventual) data consistency across users

NoSQL and Scale-Out Is NoSQL is better at “scaling out” than RDBMS? RDBMS have ACID (Atomicity, Consistency, Isolation, Durability http://en.wikipedia.org/wiki/ACID ) and supports transactions. Scaling "out" with RDBMS is harder to implement due to these concepts. NoSQL solutions usually offer record-level atomicity, but cannot guarantee a series of operations will succeed (transaction).

SQL Azure Azure Custom Sharding You can design and develop your own mechanisms for defining shards, routing connections, and managing data distribution among them. Custom sharding typically requires significant development effort beyond application logic.

SQL Azure Custom Sharding

SQL Azure Azure Custom Sharding Horizontal Partitioning and Vertical Partitioning Horizontal partitioning, likes you use a knife to cut the table horizontally, which means split the table by rows

SQL Azure Azure Custom Sharding Horizontal Partitioning and Vertical Partitioning Vertical partitioning means split the table by columns

SQL Azure Federations utilize the horizontal partitioning to split the tables in multiple databases. make sure that all records in the tables that referred to the same ID must be in the same partition

SQL Azure Federations Provides Data Partitioning/Sharding at the Data Platform Enables applications to build elastic scale-out applications Provides non-blocking SPLIT/DROP for shards Auto-connect to right shard based on sharding keyvalue Provides SPLIT resilient query mode

SQL Azure Federations Azure Federation is a Sharding technology which allows distributing the DB transactions across multiple databases which are called Federation Members.

SQL Azure Federations The SQL Azure Federation have following essential objects: Federation object This object is responsible for the distribution schema of the Federation Members

SQL Azure Federations The SQL Azure Federation have following essential objects: Federation Key a column in the Federated Table. The value of the Federation Key determines in which Federation Member the data row is stored. 

SQL Azure Federations The SQL Azure Federation have following essential objects: Federation Member It is a physically separate SQL Azure database. it holds data related to a range of Federation Keys, configured for this Federation Member in the Federation object. The Federation Member can have two types of tables: Federated tables and Non-Federated tables.

SQL Azure Federations The SQL Azure Federation have following essential objects: Atomic Unit Atomic Unit represents all rows of data from Federated Table(s) which have a particular value of the Federation Key

SQL Azure Federations

SQL Azure Federations How does it work?: Once you connect to the SQL Azure DB you create a Federation object. This database becomes Federation Root. When you create a Federation object your first Federation Member, a separated SQL Azure DB, is also created.

SQL Azure Federations Federation “Orders_Fed” Represents the data being sharded Federation Root Database that logically houses federations, contains federation meta data Federation Key Value that determines the routing of a piece of data (defines a Federation Distribution) Atomic Unit All rows with the same federation key value: always together! Federation Member (aka Shard) A physical container for a set of federated tables for a specific key range and reference tables Federated Table Table that contains only atomic units for the member’s key range Reference Table Non-sharded table Azure DB with Federation Root Federation Directories, Federation Users, Federation Distributions, … Federation “Orders_Fed” (Federation Key: CustomerID) Member: PK [min, 100) AU PK=5 AU PK=25 AU PK=35 Member: PK [100, 488) AU PK=105 AU PK=235 AU PK=365 Connection Gateway Member: PK [488, max) AU PK=555 AU PK=2545 AU PK=3565 Sharded Application 37

SQL Azure Federations One federated database contains three types of table Federated Reference Common One federated table is created by appending FEDERATED ON to a CREATE TABLE statement. CREATE TABLE (…) FEDERATED ON (Customerid = custId)

SQL Azure Federations CREATE FEDERATION: CREATE FEDERATION WADFederation (ID int RANGE) GO ADDING A TABLE TO A FEDERATION CREATE TABLE … FEDERATED ON (Customerid = custId) ADDING DATA WITH FEDERATION DATA MIGRATION WIZARD

SQL Azure Federations CONSIDERATIONS: SQL Azure Federations requires additional efforts: administration of the federations Knowledge about the sharding architecture It is not flexible during the development cycle Code First approach with EF = ?

SQL Azure Federations Actual state and future trends: Actual Azure SQL Database tires (Basic, Standard and Premium ) don’t support Azure Federations Only Retired Tires (WEB and Business ) support SQL Azure Federations Federations functionality is deprecated and you cannot create new federations even for WEB and Business Tires for new databases

SQL Azure Federations Actual state and future trends : The Azure SQL Database Federations feature is being retired along with the Web/Business editions in September 2015 Migration could be realized using the Federations Migration Utility

Scaling Out Microsoft Azure SQL Databases Azure SQL Database Elastic Scale

Azure SQL Database Elastic Scale Overview Microsoft Azure Elastic Scale Concept?

Azure SQL Database Elastic Scale Overview What is Azure SQL Database Elastic Scale ? Azure DB Elastic Scale implements the infrastructure aspects of sharding and thus allows you to focus on the business logic of your application instead. Elastic Scale delivers both developer and management functionality which are provided through a set of .Net libraries and through Azure service templates It enables the data-tier of an application to scale out and in via industry-standard sharding practices

Azure SQL Database Elastic Scale Overview Microsoft Azure Elastic Scale Concept?

Azure SQL Database Elastic Scale Overview Main Challenge: Growing and shrinking capacity for the relational database as a part of your workload

Azure SQL Database Elastic Scale Overview Microsoft Azure Elastic Scale Concept Architecture Changes: Managing Shard Meta DB infra-structure Splitting 1 noisy customer data from one shard to another shard Scaling up a particular shard on need basis Querying data from multi shard Databases Merging shards to cut down costs

Azure SQL Database Elastic Scale Overview Microsoft Azure Elastic Scale Concept OLAP Challenges (Database Analysis & Warehousing): Developers will not be able to issue single query to fetch data from 2 different shards Conducting Data Analysis is hard

Azure SQL Database Elastic Scale Overview Elastic Scale Main Elements : Shard Map Manager (SMM) Data Dependent Routing (DDR) Multi Shard Query(MSQ) Split & Merge

Azure SQL Database Elastic Scale Overview Elastic Scale Elements:

Azure SQL Database Elastic Scale Overview Elastic Scale Elements: Data dependent routing: The functionality that enables an application to connect to a shard given a specific sharding key. Multi-shard query: The ability to issue a query against multiple shards; results sets are returned using UNION ALL semantics

Azure SQL Database Elastic Scale Overview Elastic Scale Elements: Shard: An Azure SQL database that stores data from a sharded data set. Shard Map: The set of mappings between sharding keys and their respective shards. Shard Set: The collection of shards that are attributed to the same shard map in the shard map manager.

Azure SQL Database Elastic Scale Overview Elastic Scale Elements: List Shard Map: A shard map in which sharding keys are mapped individually.  Range Shard Map: A shard map in which the shard distribution strategy is based on multiple ranges. Global shard map: The set of mappings between sharding keys and their respective shards within a shard set. Local shard map: Stored on a shard, the local shard map contains mappings for the shardlets that reside on the shard.

Azure SQL Database Elastic Scale Overview Elastic Scale Elements: Sharded tables: Tables that are sharded, i.e., whose data is distributed across shards based on their sharding key values. Reference tables: Tables that are not sharded but are replicated across shards. Shardlet: All of the data associated with a single value of a sharding key on a shard. 

Azure SQL Database Elastic Scale Overview Elastic Scale Elements: Shard Map Manager: A management object and data store that contains the shard map(s), shard locations, and mappings for one or more shard sets. Shard Elasticity (SE): The ability to perform both horizontal scaling and vertical scaling.

Azure SQL Database Elastic Scale Overview Elastic Scale Elements:

Azure SQL Database Elastic Scale Overview

Azure SQL Database Elastic Scale Overview Elastic Scale Actions: Horizontal scaling: The act of scaling out (or in) a collection of shards by adding or removing shards to a shard map. Vertical Scaling: The act of scaling up (or down) the performance level of an individual shard

Azure SQL Database Elastic Scale Overview Elastic Scale Actions: Shard: The act of horizontally partitioning identically structured data across multiple databases based on a sharding key. Split: The act of moving several shardlets from one shard to another (typically new) shard. Merge: The act of moving shardlets from two shards to one shard and updating the shard map Shardlet move: The act of moving a single shardlet to a different shard.

Horizontal Versus Vertical Scale Out Use horizontal scale out to increase capacity. Use vertical scale out to increase the performance of a shard.

Elastic Scale – Scaling Scenarios Scenarios for Split/Merge Grow Capacity – Splitting Ranges: Shrink Capacity – Merging Ranges: Manage Hotspots – Moving Shardlets

Elastic Scale – Scaling Scenarios Conceptual Overview of Split/Merge

Elastic Scale – Performance Performance of your split, merge and move operations depend on a variety of factors: Better performance is to be expected from the higher, more performant service tiers in Azure SQL Database The service also performs validation queries as part of its normal operations. These queries perform best when an index is present that has the sharding key as the leading column.

Best Practices & Troubleshooting Exercise your most important split/merge/move operations with the test tenant across several shard Keep the test tenant data size above the maximum data size of your largest tenant Ability to remove data from the source shard once the data has been successfully copied to the target Ensure that the sharding key is the leading column in your primary key or unique index definition Collocate your split/merge service in the region and data center where are your databases

Elastic Scale with Sharding Shard Elasticity The capability that enables administrators to automate vertical scaling (dialing up and down the edition of a single shard) Horizontal scaling (adding or removing shards from a shard map) of their sharded environment using PowerShell scripts and by means of the Azure Automation Service.

Elastic Scale with Sharding 1. Shard Map Management, 2 Data Dependent Routing, 3. Multi-Shard Queries (MSQ), 4. Split-Merge service

Elastic Scale with Sharding Common Sharding Patterns The total amount of data is too large to fit within the constraints of a single database The transaction throughput of the overall workload exceeds the capabilities of a single database Tenants may require physical isolation from each other, so separate databases are needed for each tenant Different sections of a database may need to reside in different geographies

Single Tenant vs. Multi Tenant

Elastic Scale Prerequisites Install Azure SQL Database Elastic Scale Client Library 1.0.0 It is a client library that allows ADO.NET developers to create applications that implement and use the pattern known as database sharding in Azure SQL Database. PM> Install-Package Microsoft.Azure.SqlDatabase.ElasticScale.Client

Elastic Scale with Automation Service Using Azure Automation Service for Scaling PowerShell Scripts

Elastic Scale How To… Constructing a ShardMapManager

Elastic Scale How To… Populating a Shard Map public void CreatePopulatedRangeMap(ShardMapManager smm, string mapName) { RangeShardMap<long> sm = null; if (!smm.TryGetRangeShardMap(mapName, out sm)) sm = smm.CreateRangeShardMap<long>(mapName); } Shard shard0 = null, shard1=null; // check if shard exists and if not, if (!sm.TryGetShard(new ShardLocation(shardServer, "sample_shard_0"), out shard0)) Shard0 = sm.CreateShard(new ShardLocation(shardServer, "sample_shard_0")); RangeMapping<long> rmpg=null; if (!sm.TryGetMappingForKey(0, out rmpg)) sm.CreateRangeMapping(new RangeMappingCreationInfo<long> (new Range<long>(0, 50), shard0, MappingStatus.Online));

Elastic Scale How To… Adding a Shard and its Range to an Existing Shard Map Shard shard2 = null; if (!sm.TryGetShard(new ShardLocation(shardServer, "sample_shard_2"),out shard2)) { shard2 = sm.CreateShard(new ShardLocation(shardServer, "sample_shard_2")); } // Create the mapping and associate it with the new shard sm.CreateRangeMapping(new RangeMappingCreationInfo<long> (new Range<long>(300, 400), shard2, MappingStatus.Online));

Elastic Scale How To… Splitting a Range and Assigning the Empty Portion to a Newly-added Shard Shard shard2 = null; if (!sm.TryGetShard(new ShardLocation(shardServer, "sample_shard_2"),out shard2)) { shard2 = sm.CreateShard(new ShardLocation(shardServer, "sample_shard_2")); } sm.SplitMapping(sm.GetMappingForKey(25), 25); // take existing mapping offline sm.MarkMappingOffline(sm.GetMappingForKey(25)); // now map while offline to a different shard and take online RangeMappingUpdate upd = new RangeMappingUpdate(); upd.Shard = shard2; sm.MarkMappingOnline(sm.UpdateMapping(sm.GetMappingForKey(25), upd));

Elastic Scale How To… Multi-Shard Querying { using (MultiShardConnection conn = new MultiShardConnection( myShardMap.GetShards(), myShardConnectionString) ) { using (MultiShardCommand cmd = conn.CreateCommand()) cmd.CommandText = "SELECT c1, c2, c3 FROM ShardedTable"; cmd.CommandType = CommandType.Text; cmd.ExecutionOptions = MultiShardExecutionOptions.IncludeShardNameColumn; cmd.ExecutionPolicy = MultiShardExecutionPolicy.PartialResults; using (MultiShardDataReader sdr = cmd.ExecuteReader()) while (sdr.Read()) var c1Field = sdr.GetString(0); var c2Field = sdr.GetFieldValue<int>(1); var c3Field = sdr.GetFieldValue<Int64>(2); }

Elastic Scale How To… Data Dependent Routing ShardMapManager smm = ShardMapManagerFactory.GetSqlShardMapManager(smmConnnectionString, ShardMapManagerLoadPolicy.Lazy); RangeShardMap<int> customerShardMap = smm.GetRangeShardMap<int>("customerMap");

Elastic Scale How To… Data Dependent Routing int customerId = 12345; int newPersonId = 4321; using (SqlConnection conn = customerShardMap.OpenConnectionForKey(customerId, Configuration.GetCredentialsConnectionString(), ConnectionOptions.Validate)) { SqlCommand cmd = conn.CreateCommand(); cmd.CommandText = @"UPDATE Sales.Customer SET PersonID = @newPersonID WHERE CustomerID = @customerID"; cmd.Parameters.AddWithValue("@customerID", customerId); cmd.Parameters.AddWithValue("@newPersonID", newPersonId); cmd.ExecuteNonQuery(); }

Elastic Scale How To… Drops all shards and the shard map manager database private static void DropAll() { RangeShardMap<int> shardMap = TryGetShardMap(); if (shardMap != null) foreach (Shard shard in shardMap.GetShards()) SqlDatabaseUtils.DropDatabase(shard.Location.DataSource, shard.Location.Database); } if (SqlDatabaseUtils.DatabaseExists(ShardMapManagerServerName, ShardMapManagerDatabaseName)) SqlDatabaseUtils.DropDatabase(ShardMapManagerServerName, ShardMapManagerDatabaseName); shardMapManager = null;

Scaling Out Microsoft Azure SQL Databases Azure Db Elastic Scale and Entity Framework

Data Dependent Routing using EF DbContext Create physical database connections through the Elastic Scale interfaces of the shard map manager Wrap the connection with the DbContext subclass Pass the connection down into the DbContext base classes to ensure all the processing on the EF side happens as well

Data Dependent Routing using EF DbContext Integrate DbContexts with data-dependent routing for scale-out public class ElasticScaleContext<T> : DbContext { public DbSet<Blog> Blogs { get; set; } … public ElasticScaleContext(ShardMap shardMap, T shardingKey, string connectionStr) : base(CreateDDRConnection(shardMap, shardingKey, connectionStr), true /* contextOwnsConnection */) } private static DbConnection CreateDDRConnection( ShardMap shardMap, T shardingKey, string connectionStr) Database.SetInitializer<ElasticScaleContext<T>>(null); SqlConnection conn = shardMap.OpenConnectionForKey<T> (shardingKey, connectionStr, ConnectionOptions.Validate); return conn;

Data Dependent Routing using EF DbContext Create and save new data. Console.Write("Enter a name for a new blog: "); var name = Console.ReadLine(); using (var db = new ElasticScaleContext<int>( sharding.ShardMap, tenantId1, connStrBldr.ConnectionString)) { var blog = new Blog { Name = name }; db.Blogs.Add(blog); db.SaveChanges(); // Display all Blogs for tenant 1 var query = from b in db.Blogs orderby b.Name select b; … }

Data Dependent Routing using EF DbContext Transient Faults Handling SqlDatabaseUtils.SqlRetryPolicy.ExecuteAction(() => { using (var db = new ElasticScaleContext<int>( sharding.ShardMap, tenantId1, connStrBldr.ConnectionString)) var blog = new Blog { Name = name }; db.Blogs.Add(blog); db.SaveChanges(); … } });

Elastic database Split-Merge tool Elastic database Split-Merge tool is used as a sample framework ( library ) to split – merge Azure SQL Database Requirements: Download the Split-Merge Libraries Configure and Deploy Split-Merge Service

Federations Migration The Azure SQL Database Federations feature will be retired in September 2015 Current Federations applications should be migrated to the Azure SQL DB Elastic Scale

Federations Migration ( for existing federations) 3 major steps for migrating an existing Federations application to one that uses the Elastic Scale APIs Create Shard Map Manager from a Federation Root Modify the Existing Application Switch Out Existing Federation Members

Federations Migration Utility Create Shard Map Manager from a Federation Root

Federations Migration Utility Use the Federations Migration Utility to clone the federation root metadata into the Elastic Scale Shard Map Manager.

Modify the Existing Application Change 1: Instantiate a Shard Map Manager object: //Instantiate ShardMapManger Object ShardMapManager shardMapManager = ShardMapManagerFactory.GetSqlShardMapManager( connectionStringSMM, ShardMapManagerLoadPolicy.Lazy); RangeShardMap<T> rangeShardMap = shardMapManager.GetRangeShardMap<T>(shardMapName)

Modify the Existing Application Change 2: Route Connections to the Appropriate Shard USE FEDERATION CustomerFederation(cid=100) WITH RESET, FILTERING=OFF //Connect and issue queries on the shard with key=100 using (SqlConnection conn = rangeShardMap.OpenConnectionForKey(100, csb)) { using (SqlCommand cmd = new SqlCommand()) { cmd.Connection = conn; cmd.CommandText = "SELECT * FROM customer"; using (SqlDataReader dr = cmd.ExecuteReader()) { //Perform action on dr } } }

Recommendations for Scaling Out New sharded applications should implement the Elastic Scale APIs and capabilities. Use a custom sharding approach if your application requires much more functionality beyond the capabilities of Elastic Scale. Existing Federations solutions can be migrated to employ sharding using the Elastic Scale’s migration utility.

Azure SQL Database Examples Elastic Scale with Azure DB - Getting Started https://code.msdn.microsoft.com/vstudio/Elastic-Scale-with-Azure-a80d8dc6 Elastic Scale with Azure DB - Entity Framework Integration https://code.msdn.microsoft.com/vstudio/Elastic-Scale-with-Azure-bae904ba Elastic Scale for Azure SQL DB – ShardSqlCmd https://code.msdn.microsoft.com/vstudio/Elastic-Scale-with-Azure-603e0cef Azure SQL Database Elastic Scale: Shard Elasticity https://gallery.technet.microsoft.com/scriptcenter/Elastic-Scale-Shard-c9530cbe

Useful links Scaling Out Azure SQL Databases, MSDN http://msdn.microsoft.com/en-us/library/azure/dn495641.aspx Get Started with Azure SQL Database Elastic Scale Preview, Microsoft Azure Documentation http://azure.microsoft.com/en-us/documentation/articles/sql-database-elastic-scale-get-started/ Introducing Elastic Scale preview for Azure SQL Database, Microsoft Azure Blog http://azure.microsoft.com/blog/2014/10/02/introducing-elastic-scale-preview-for-azure-sql-database/ Azure SQL Database Elastic Scale Overview, Microsoft Azure Documentation http://azure.microsoft.com/en-us/documentation/articles/sql-database-elastic-scale-introduction/

Thank you Thank You! Спасибі! Спасибо!

Scaling Out Microsoft Azure SQL Databases Q & A