1. Scaling Out Microsoft Azure SQL Databases
Azure SQL Database supports three methods to implement sharding: Elastic Scale, Custom Sharding and Federations
This session is a comparison and overview of these approaches, to be possible to choose the best one.
Mihail Mateev, Solutions Architect, PM and Senior Technical Evangelist, Strypes

2. Say Thank you to Volunteers:
They spend their FREE time to give you this event.
Because they are love our COMMUNITY.
Because they want YOU to learn from the BEST IN THE WORLD.
If you see one of these guys in the hall – buy them a beer/wine, they deserve it.

3. Vitaliy Popovych

4. Olena Smoliak

5. Maksim Garnets

6. Yevhen Nedashkivskyi

7. Oksana Borysenko

8. 3 Sponsor Sessions at 12:30
Don’t miss them, they might be providing some interesting and valuable information!
Bakotech (Dell) – Room “AE”
Intapp – Room “F”
DevArt – Room “Lazurny”

9. Our Sponsors:

10. Scaling Out Microsoft Azure SQL Databases
Welcome!
Ласкаво просимо!
Добро пожаловать!

11. About me
Mihail Mateev is a Solutions Strypes / ICT Automatisering, Community enthusiast, PASS RM for CEE and chapter lead, Microsoft Azure MVP
Mihail works in various areas related to Microsoft technologies : Silverlight, WPF, WP, LightSwitch, WCF, ASP.Net MVC, MS SQL Server and Microsoft Azure

12. Agenda Approaches to Scaling Out Introduction to Azure Federations
Custom Sharding
Elastic Scale
Introduction to Azure Federations
Azure SQL Database Elastic Scale
Recommendations for Scaling Out

13. DB Scaling solutions: Scale UP vs. Scale OUT
Background
The number of clients or sales or licenses grows over time as a linear function
We have a linear function for number of DB Transactions, and amount of data
The query response time grows exponentially

14. DB Scaling solutions: Scale UP vs. Scale OUT
Scale UP improving the hardware solution of the DB
faster processor
more RAM
faster Hard Drive
Scale OUT (assume some amount of re-work)
Table partitioning
Master/Slave configurations
Cluster Computing
Sharding

15. DB Scaling solutions: Scale OUT
Table partitioning:
The large tables are split into two or more physically separate partitions
The queries will end up hitting two separate tables.
The tables can also be located in different DB

16. DB Scaling solutions: Scale OUT
Master/Slave configurations
We have one database which is called “Master” and multiple databases called “Slaves”
The Master DB is set up for read/write access
Slave databases are read-only.
There is a data replication between the Master and the Slaves

17. DB Scaling solutions: Scale OUT
Cluster Computing:
Similar to the Master/Slave in the DB Cluster 
we have one database node which is a DB available for read/write and multiple nodes which are in the read-only mode
All data updates are received by the “Master” node and replicated into the read-only nodes
The tables can also be located in different DB When Master DB is failed one of the read-only nodes becomes the new “Master”
Sharding
A solution in which the application operates with “Shards” which can be physically separate databases.
There are queries which run inside the shard and queries which are distributed across multiple shardsets

18. DB Scaling solutions: Scale OUT
Sharding

19. DB Scaling solutions: Scale OUT
Sharding Example:

20. DB Scaling solutions: Scale OUT
Sharding :
Records are grouped by a particular key, known as a Sharding Key
Records that are associated with a particular Sharding Key are known as a Shardlet
The different databases that store the collections of Shardlets are known as Partitions

21. Why to Scale-Out your database
Scaling out your business is important
NoSQL and Scale-Out
Sharding / SQL Azure Federations

22. Scaling out your business
Social networking: the business problem
100s of million of users
10s of million of users concurrently
Terabytes to petabytes of data (Structured and unstructured )
Required (eventual) data consistency across users

23. NoSQL and Scale-Out Is NoSQL is better at “scaling out” than RDBMS?
RDBMS have ACID (Atomicity, Consistency, Isolation, Durability ) and supports transactions. Scaling "out" with RDBMS is harder to implement due to these concepts.
NoSQL solutions usually offer record-level atomicity, but cannot guarantee a series of operations will succeed (transaction).

24. SQL Azure Azure Custom Sharding
You can design and develop your own mechanisms for defining shards, routing connections, and managing data distribution among them.
Custom sharding typically requires significant development effort beyond application logic.

25. SQL Azure Custom Sharding

26. SQL Azure Azure Custom Sharding
Horizontal Partitioning and Vertical Partitioning
Horizontal partitioning, likes you use a knife to cut the table horizontally, which means split the table by rows

27. SQL Azure Azure Custom Sharding
Horizontal Partitioning and Vertical Partitioning
Vertical partitioning means split the table by columns

28. SQL Azure Federations
utilize the horizontal partitioning to split the tables in multiple databases.
make sure that all records in the tables that referred to the same ID must be in the same partition

29. SQL Azure Federations
Provides Data Partitioning/Sharding at the Data Platform
Enables applications to build elastic scale-out applications
Provides non-blocking SPLIT/DROP for shards
Auto-connect to right shard based on sharding keyvalue
Provides SPLIT resilient query mode

30. SQL Azure Federations
Azure Federation is a Sharding technology which allows distributing the DB transactions across multiple databases which are called Federation Members.

31. SQL Azure Federations
The SQL Azure Federation have following essential objects:
Federation object
This object is responsible for the distribution schema of the Federation Members

32. SQL Azure Federations
The SQL Azure Federation have following essential objects:
Federation Key
a column in the Federated Table. The value of the Federation Key determines in which Federation Member the data row is stored. 

33. SQL Azure Federations
The SQL Azure Federation have following essential objects:
Federation Member
It is a physically separate SQL Azure database.
it holds data related to a range of Federation Keys, configured for this Federation Member in the Federation object.
The Federation Member can have two types of tables: Federated tables and Non-Federated tables.

34. SQL Azure Federations
The SQL Azure Federation have following essential objects:
Atomic Unit
Atomic Unit represents all rows of data from Federated Table(s) which have a particular value of the Federation Key

35. SQL Azure Federations

36. SQL Azure Federations How does it work?:
Once you connect to the SQL Azure DB you create a Federation object. This database becomes Federation Root.
When you create a Federation object your first Federation Member, a separated SQL Azure DB, is also created.

37. SQL Azure Federations Federation “Orders_Fed”
Represents the data being sharded
Federation Root
Database that logically houses federations, contains federation meta data
Federation Key
Value that determines the routing of a piece of data (defines a Federation Distribution)
Atomic Unit
All rows with the same federation key value: always together!
Federation Member (aka Shard)
A physical container for a set of federated tables for a specific key range and reference tables
Federated Table
Table that contains only atomic units for the member’s key range
Reference Table
Non-sharded table
Azure DB with Federation Root
Federation Directories, Federation Users, Federation Distributions, …
Federation “Orders_Fed”
(Federation Key: CustomerID)
Member: PK [min, 100)
AU PK=5
AU PK=25
AU PK=35
Member: PK [100, 488)
AU PK=105
AU PK=235
AU PK=365
Connection Gateway
Member: PK [488, max)
AU PK=555
AU PK=2545
AU PK=3565
Sharded
Application 37

38. SQL Azure Federations
One federated database contains three types of table
Federated
Reference
Common
One federated table is created by appending FEDERATED ON to a CREATE TABLE statement. CREATE TABLE (…) FEDERATED ON (Customerid = custId)

39. SQL Azure Federations CREATE FEDERATION:
CREATE FEDERATION WADFederation (ID int RANGE) GO
ADDING A TABLE TO A FEDERATION
CREATE TABLE … FEDERATED ON (Customerid = custId)
ADDING DATA WITH FEDERATION DATA MIGRATION WIZARD

40. SQL Azure Federations CONSIDERATIONS:
SQL Azure Federations requires additional efforts:
administration of the federations Knowledge about the sharding architecture
It is not flexible during the development cycle
Code First approach with EF = ?

41. SQL Azure Federations Actual state and future trends:
Actual Azure SQL Database tires (Basic, Standard and Premium ) don’t support Azure Federations
Only Retired Tires (WEB and Business ) support SQL Azure Federations
Federations functionality is deprecated and you cannot create new federations even for WEB and Business Tires for new databases

42. SQL Azure Federations Actual state and future trends :
The Azure SQL Database Federations feature is being retired along with the Web/Business editions in September 2015
Migration could be realized using the Federations Migration Utility

43. Scaling Out Microsoft Azure SQL Databases
Azure SQL Database Elastic Scale

44. Azure SQL Database Elastic Scale Overview
Microsoft Azure Elastic Scale Concept?

45. Azure SQL Database Elastic Scale Overview
What is Azure SQL Database Elastic Scale ?
Azure DB Elastic Scale implements the infrastructure aspects of sharding and thus allows you to focus on the business logic of your application instead.
Elastic Scale delivers both developer and management functionality which are provided through a set of .Net libraries and through Azure service templates
It enables the data-tier of an application to scale out and in via industry-standard sharding practices

46. Azure SQL Database Elastic Scale Overview
Microsoft Azure Elastic Scale Concept?

47. Azure SQL Database Elastic Scale Overview
Main Challenge:
Growing and shrinking capacity for the relational database as a part of your workload

48. Azure SQL Database Elastic Scale Overview
Microsoft Azure Elastic Scale Concept
Architecture Changes:
Managing Shard Meta DB infra-structure
Splitting 1 noisy customer data from one shard to another shard
Scaling up a particular shard on need basis
Querying data from multi shard Databases
Merging shards to cut down costs

49. Azure SQL Database Elastic Scale Overview
Microsoft Azure Elastic Scale Concept
OLAP Challenges (Database Analysis & Warehousing):
Developers will not be able to issue single query to fetch data from 2 different shards
Conducting Data Analysis is hard

50. Azure SQL Database Elastic Scale Overview
Elastic Scale Main Elements :
Shard Map Manager (SMM)
Data Dependent Routing (DDR)
Multi Shard Query(MSQ)
Split & Merge

51. Azure SQL Database Elastic Scale Overview
Elastic Scale Elements:

52. Azure SQL Database Elastic Scale Overview
Elastic Scale Elements:
Data dependent routing: The functionality that enables an application to connect to a shard given a specific sharding key.
Multi-shard query: The ability to issue a query against multiple shards; results sets are returned using UNION ALL semantics

53. Azure SQL Database Elastic Scale Overview
Elastic Scale Elements:
Shard: An Azure SQL database that stores data from a sharded data set.
Shard Map: The set of mappings between sharding keys and their respective shards.
Shard Set: The collection of shards that are attributed to the same shard map in the shard map manager.

54. Azure SQL Database Elastic Scale Overview
Elastic Scale Elements:
List Shard Map: A shard map in which sharding keys are mapped individually. 
Range Shard Map: A shard map in which the shard distribution strategy is based on multiple ranges.
Global shard map: The set of mappings between sharding keys and their respective shards within a shard set.
Local shard map: Stored on a shard, the local shard map contains mappings for the shardlets that reside on the shard.

55. Azure SQL Database Elastic Scale Overview
Elastic Scale Elements:
Sharded tables: Tables that are sharded, i.e., whose data is distributed across shards based on their sharding key values.
Reference tables: Tables that are not sharded but are replicated across shards.
Shardlet: All of the data associated with a single value of a sharding key on a shard. 

56. Azure SQL Database Elastic Scale Overview
Elastic Scale Elements:
Shard Map Manager: A management object and data store that contains the shard map(s), shard locations, and mappings for one or more shard sets.
Shard Elasticity (SE): The ability to perform both horizontal scaling and vertical scaling.

57. Azure SQL Database Elastic Scale Overview
Elastic Scale Elements:

58. Azure SQL Database Elastic Scale Overview

59. Azure SQL Database Elastic Scale Overview
Elastic Scale Actions:
Horizontal scaling: The act of scaling out (or in) a collection of shards by adding or removing shards to a shard map.
Vertical Scaling: The act of scaling up (or down) the performance level of an individual shard

60. Azure SQL Database Elastic Scale Overview
Elastic Scale Actions:
Shard: The act of horizontally partitioning identically structured data across multiple databases based on a sharding key.
Split: The act of moving several shardlets from one shard to another (typically new) shard.
Merge: The act of moving shardlets from two shards to one shard and updating the shard map
Shardlet move: The act of moving a single shardlet to a different shard.

61. Horizontal Versus Vertical Scale Out
Use horizontal scale out to increase capacity.
Use vertical scale out to increase the performance of a shard.

62. Elastic Scale – Scaling Scenarios
Scenarios for Split/Merge
Grow Capacity – Splitting Ranges:
Shrink Capacity – Merging Ranges:
Manage Hotspots – Moving Shardlets

63. Elastic Scale – Scaling Scenarios
Conceptual Overview of Split/Merge

64. Elastic Scale – Performance
Performance of your split, merge and move operations depend on a variety of factors:
Better performance is to be expected from the higher, more performant service tiers in Azure SQL Database
The service also performs validation queries as part of its normal operations. These queries perform best when an index is present that has the sharding key as the leading column.

65. Best Practices & Troubleshooting
Exercise your most important split/merge/move operations with the test tenant across several shard
Keep the test tenant data size above the maximum data size of your largest tenant
Ability to remove data from the source shard once the data has been successfully copied to the target
Ensure that the sharding key is the leading column in your primary key or unique index definition
Collocate your split/merge service in the region and data center where are your databases

66. Elastic Scale with Sharding
Shard Elasticity
The capability that enables administrators to automate
vertical scaling (dialing up and down the edition of a single shard)
Horizontal scaling (adding or removing shards from a shard map) of their sharded environment
using PowerShell scripts and by means of the Azure Automation Service.

67. Elastic Scale with Sharding
1. Shard Map Management, 2 Data Dependent Routing, 3. Multi-Shard Queries (MSQ), 4. Split-Merge service

68. Elastic Scale with Sharding
Common Sharding Patterns
The total amount of data is too large to fit within the constraints of a single database
The transaction throughput of the overall workload exceeds the capabilities of a single database
Tenants may require physical isolation from each other, so separate databases are needed for each tenant
Different sections of a database may need to reside in different geographies

69. Single Tenant vs. Multi Tenant

70. Elastic Scale Prerequisites
Install Azure SQL Database Elastic Scale Client Library It is a client library that allows ADO.NET developers to create applications that implement and use the pattern known as database sharding in Azure SQL Database.
PM> Install-Package Microsoft.Azure.SqlDatabase.ElasticScale.Client

71. Elastic Scale with Automation Service
Using Azure Automation Service for Scaling
PowerShell Scripts

72. Elastic Scale How To…
Constructing a ShardMapManager

73. Elastic Scale How To… Populating a Shard Map
public void CreatePopulatedRangeMap(ShardMapManager smm, string mapName) {
RangeShardMap<long> sm = null;
if (!smm.TryGetRangeShardMap(mapName, out sm))
sm = smm.CreateRangeShardMap<long>(mapName); }
Shard shard0 = null, shard1=null;
// check if shard exists and if not,
if (!sm.TryGetShard(new ShardLocation(shardServer, "sample_shard_0"), out shard0))
Shard0 = sm.CreateShard(new ShardLocation(shardServer, "sample_shard_0"));
RangeMapping<long> rmpg=null;
if (!sm.TryGetMappingForKey(0, out rmpg))
sm.CreateRangeMapping(new RangeMappingCreationInfo<long>
(new Range<long>(0, 50), shard0, MappingStatus.Online));

74. Elastic Scale How To…
Adding a Shard and its Range to an Existing Shard Map
Shard shard2 = null;
if (!sm.TryGetShard(new ShardLocation(shardServer, "sample_shard_2"),out shard2)) {
shard2 = sm.CreateShard(new ShardLocation(shardServer, "sample_shard_2")); }
// Create the mapping and associate it with the new shard
sm.CreateRangeMapping(new RangeMappingCreationInfo<long>
(new Range<long>(300, 400), shard2, MappingStatus.Online));

75. Elastic Scale How To…
Splitting a Range and Assigning the Empty Portion to a Newly-added Shard
Shard shard2 = null;
if (!sm.TryGetShard(new ShardLocation(shardServer, "sample_shard_2"),out shard2)) {
shard2 = sm.CreateShard(new ShardLocation(shardServer, "sample_shard_2")); }
sm.SplitMapping(sm.GetMappingForKey(25), 25);
// take existing mapping offline
sm.MarkMappingOffline(sm.GetMappingForKey(25));
// now map while offline to a different shard and take online
RangeMappingUpdate upd = new RangeMappingUpdate();
upd.Shard = shard2;
sm.MarkMappingOnline(sm.UpdateMapping(sm.GetMappingForKey(25), upd));

76. Elastic Scale How To… Multi-Shard Querying {
using (MultiShardConnection conn = new MultiShardConnection( myShardMap.GetShards(), myShardConnectionString) ) {
using (MultiShardCommand cmd = conn.CreateCommand())
cmd.CommandText = "SELECT c1, c2, c3 FROM ShardedTable";
cmd.CommandType = CommandType.Text;
cmd.ExecutionOptions = MultiShardExecutionOptions.IncludeShardNameColumn;
cmd.ExecutionPolicy = MultiShardExecutionPolicy.PartialResults;
using (MultiShardDataReader sdr = cmd.ExecuteReader())
while (sdr.Read())
var c1Field = sdr.GetString(0);
var c2Field = sdr.GetFieldValue<int>(1);
var c3Field = sdr.GetFieldValue<Int64>(2); }

77. Elastic Scale How To… Data Dependent Routing
ShardMapManager smm = ShardMapManagerFactory.GetSqlShardMapManager(smmConnnectionString, ShardMapManagerLoadPolicy.Lazy);
RangeShardMap<int> customerShardMap = smm.GetRangeShardMap<int>("customerMap");

78. Elastic Scale How To… Data Dependent Routing int customerId = 12345;
int newPersonId = 4321;
using (SqlConnection conn = customerShardMap.OpenConnectionForKey(customerId,
Configuration.GetCredentialsConnectionString(), ConnectionOptions.Validate)) {
SqlCommand cmd = conn.CreateCommand();
cmd.CommandText Sales.Customer
SET PersonID
WHERE CustomerID
customerId);
newPersonId);
cmd.ExecuteNonQuery(); }

79. Elastic Scale How To…
Drops all shards and the shard map manager database
private static void DropAll() {
RangeShardMap<int> shardMap = TryGetShardMap();
if (shardMap != null)
foreach (Shard shard in shardMap.GetShards())
SqlDatabaseUtils.DropDatabase(shard.Location.DataSource, shard.Location.Database); }
if (SqlDatabaseUtils.DatabaseExists(ShardMapManagerServerName, ShardMapManagerDatabaseName))
SqlDatabaseUtils.DropDatabase(ShardMapManagerServerName, ShardMapManagerDatabaseName);
shardMapManager = null;

80. Scaling Out Microsoft Azure SQL Databases
Azure Db Elastic Scale and Entity Framework

81. Data Dependent Routing using EF DbContext
Create physical database connections through the Elastic Scale interfaces of the shard map manager
Wrap the connection with the DbContext subclass
Pass the connection down into the DbContext base classes to ensure all the processing on the EF side happens as well

82. Data Dependent Routing using EF DbContext
Integrate DbContexts with data-dependent routing for scale-out
public class ElasticScaleContext<T> : DbContext {
public DbSet<Blog> Blogs { get; set; } …
public ElasticScaleContext(ShardMap shardMap, T shardingKey, string connectionStr)
: base(CreateDDRConnection(shardMap, shardingKey, connectionStr),
true /* contextOwnsConnection */) }
private static DbConnection CreateDDRConnection(
ShardMap shardMap, T shardingKey, string connectionStr)
Database.SetInitializer<ElasticScaleContext<T>>(null);
SqlConnection conn = shardMap.OpenConnectionForKey<T>
(shardingKey, connectionStr, ConnectionOptions.Validate);
return conn;

83. Data Dependent Routing using EF DbContext
Create and save new data.
Console.Write("Enter a name for a new blog: ");
var name = Console.ReadLine();
using (var db = new ElasticScaleContext<int>(
sharding.ShardMap,
tenantId1,
connStrBldr.ConnectionString)) {
var blog = new Blog { Name = name };
db.Blogs.Add(blog);
db.SaveChanges();
// Display all Blogs for tenant 1
var query = from b in db.Blogs
orderby b.Name
select b; … }

84. Data Dependent Routing using EF DbContext
Transient Faults Handling
SqlDatabaseUtils.SqlRetryPolicy.ExecuteAction(() => {
using (var db = new ElasticScaleContext<int>(
sharding.ShardMap,
tenantId1,
connStrBldr.ConnectionString))
var blog = new Blog { Name = name };
db.Blogs.Add(blog);
db.SaveChanges(); … }
});

85. Elastic database Split-Merge tool
Elastic database Split-Merge tool is used as a sample framework ( library ) to split – merge Azure SQL Database
Requirements:
Download the Split-Merge Libraries
Configure and Deploy Split-Merge Service

86. Federations Migration
The Azure SQL Database Federations feature will be retired in September 2015
Current Federations applications should be migrated to the Azure SQL DB Elastic Scale

87. Federations Migration ( for existing federations)
3 major steps for migrating an existing Federations application to one that uses the Elastic Scale APIs
Create Shard Map Manager from a Federation Root
Modify the Existing Application
Switch Out Existing Federation Members

88. Federations Migration Utility
Create Shard Map Manager from a Federation Root

89. Federations Migration Utility
Use the Federations Migration Utility to clone the federation root metadata into the Elastic Scale Shard Map Manager.

90. Modify the Existing Application
Change 1: Instantiate a Shard Map Manager object:
//Instantiate ShardMapManger Object ShardMapManager shardMapManager =
ShardMapManagerFactory.GetSqlShardMapManager( connectionStringSMM,
ShardMapManagerLoadPolicy.Lazy);
RangeShardMap<T> rangeShardMap =
shardMapManager.GetRangeShardMap<T>(shardMapName)

91. Modify the Existing Application
Change 2: Route Connections to the Appropriate Shard
USE FEDERATION CustomerFederation(cid=100) WITH RESET, FILTERING=OFF
//Connect and issue queries on the shard with key=100
using
(SqlConnection conn = rangeShardMap.OpenConnectionForKey(100, csb))
{ using (SqlCommand cmd = new SqlCommand())
{ cmd.Connection = conn; cmd.CommandText = "SELECT * FROM customer";
using (SqlDataReader dr = cmd.ExecuteReader()) {
//Perform action on dr
} } }

92. Recommendations for Scaling Out
New sharded applications should implement the Elastic Scale APIs and capabilities.
Use a custom sharding approach if your application requires much more functionality beyond the capabilities of Elastic Scale.
Existing Federations solutions can be migrated to employ sharding using the Elastic Scale’s migration utility.

93. Azure SQL Database Examples
Elastic Scale with Azure DB - Getting Started
Elastic Scale with Azure DB - Entity Framework Integration
Elastic Scale for Azure SQL DB – ShardSqlCmd
Azure SQL Database Elastic Scale: Shard Elasticity

94. Useful links
Scaling Out Azure SQL Databases, MSDN
Get Started with Azure SQL Database Elastic Scale Preview, Microsoft Azure Documentation
Introducing Elastic Scale preview for Azure SQL Database, Microsoft Azure Blog
Azure SQL Database Elastic Scale Overview, Microsoft Azure Documentation

95. Thank you
Thank You!
Спасибі!
Спасибо!

96. Scaling Out Microsoft Azure SQL Databases
Q & A