ELASTIC DATABASE CAPABILITIES WITH AZURE SQL DB Silvia Doomra Azure SQL DB Program Management

ELASTIC DATABASE TOOLS & SERVICES - GOALS Simplify the creation and operation of SaaS solutions in Azure that grow to use large numbers of databases  Develop OLTP applications with scaled out data tiers in Azure SQL DB  Scale (grow or shrink) Azure SQL DB resources as needed  Manage operations over many Azure DB databases

SHARDING AND TENANCY MODELS Single tenant per database  Each tenant’s data is stored in a different database  Better isolation of tenants as compared to multi-tenant model Multiple tenants per database  Multiple tenants share the same database  Less isolation of tenants as compared to single tenant model Hybrid model  Some tenants share databases, others get their own database  E.g., premium or paying customers get their own databases, while free tier customers share databases

ELASTIC SCALE FOR YOUR DATA TIER  Shard map management: Manage membership in a scaled out data tier  Two types of shard maps  Range: contiguous values  List: explicit values  Types of sharding keys  INT, BIGINT, GUID, VARBINARY  All datetime types Shard Map Manager [shards_global] sidsmidDatasourc e Databasena me 11serverNameDB2 21serverNameDB3 [shardmaps_global] smidname 1RangeShardMap [shard_mappings_global] midsmidminma x Sid DB 1 [0-100)... DB 2 [ ) DB 3 [ ) DB 4 [ ) DB 5 [ ) DB 6 [ ) DB n [n-n+100)

DATA DEPENDENT ROUTING Scenario: query a shard with a specific shardlet key/tenant-id Application Developer Client App DDR APIs ( ) Admin/ DevOps SELECT * FROM customers WHERE customer ID = 104 Shard Map Manager DB 1 [0-100)... DB 2 [ ) DB 3 [ ) DB 4 [ ) DB 5 [ ) DB 5 [ ) DB n [n-n+100)

// Get a routed connection for a given shardingKey using (SqlConnection conn = ShardMap.OpenConnectionForKey( shardingKey, connectionString /* Credentials Only */, ConnectionOptions.Validate /* Validate */ )); { using (SqlCommand cmd = new SqlCommand() { cmd.Connection = conn; cmd.CommandText = "SELECT dbNameField, TestIntField, TestBigIntField FROM ShardedTable"; SqlDataReader sdr = cmd.ExecuteReader(); // Now consume results from the data reader… } Data Dependent Routing

DATA DEPENDENT ROUTING Caching: improve performance of shard operations  Global Shard Map (GSM) – state of all shards in the Shard Map  Local Shard Map (LSM) – state of all shards on a particular shard  Client Cache (eager/lazy) – state of all shards in the Shard Map/known shards Application Developer Client App DDR APIs ( ) Shard Map Manager DB 1 [0-100) GSM Cache LSM

ELASTIC SCALE CONNECTION OPENING FLOW  OpenConnectionForKey call with validation on  Check for shardlet key in cache  Cache miss: Fetch mapping info from GSM  Connect to shard  Validate on shard  Validation fails: Go back to cache miss Client Application Shard Map Manager DB 2 [100, 300) GSM Cache LSM ShardMap.OpenConnectionForKey( 104 /* Tenant ID */, “…”/* Credentials Only */, ConnectionOptions.Validate /* Validate */ )); [100, 200): DB2 spValidate

MULTI-SHARD QUERY Scenario: execute a query across a set of shards (returns a UNION ALL result set) Application Developer Client App MSQ APIs ( ) Admin/ DevOps SELECT count(*) FROM customers UNION ALL result set Shard Map Manager DB 1 [0-100)... DB 2 [ ) DB 3 [ ) DB 4 [ ) DB 5 [ ) DB 5 [ ) DB n [n-n+100)

using (MultiShardConnection conn = new MultiShardConnection(m_shardMap.GetAllShards(null), MultiShardTestUtils.GetTestSqlCredential())) { using (MultiShardCommand cmd = conn.CreateCommand()) { cmd.CommandText = "SELECT dbNameField, TestIntField, TestBigIntField FROM ShardedTable"; cmd.CommandType = CommandType.Text; cmd.Policy = MultiShardPolicy.PartialResults; using (MultiShardDataReader sdr = cmd.ExecuteReader(includeShardNameColumn: true)) { while (sdr.Read()) { var dbNameField = sdr.GetString(0); var testIntField = sdr.GetFieldValue (1); var testBigIntField = sdr.GetFieldValue (2); string shardIdPseudoColumn = sdr.GetFieldValue (3); } MULTI-SHARD QUERY

GETTING STARTED IS EASY! 5 minute experience to a running app in Visual studio! us/documentation/articles/sql-database-elastic-scale- documentation-map/

DEMO ELASTIC DATABASE TOOLS

DATA MOVEMENT: SPLIT/MERGE Scenario: perform a split or merge action  Split: create two distinct shards from one  Merge: create one shard from two distinct shards Application Developer Admin/ DevOps Customer Hosted Services (SM) Split Merge DB 1 [0-100)... DB 2 [ ) DB 3 [ ) DB 4 [ ) DB 5 [ ) DB 5 [ ) DB n [n-n+100) DB 5.1 [ ) DB 5.2 [ ) DB 2.1 [ )

ELASTIC DB QUERY DB OLTP Cloud Application Elastic Tools Libraries DB Azure SQL Database PowerBI SQL DB Elastic Query Azure SQL DB v12 SQL TDS, ODBC, JDBC, ADO

ELASTIC DB QUERY: QUERYING ACROSS MANY DATABASES  Connect to a single SQL DB database using familiar SQL DB connection strings  Simple setup with T-SQL DDL  Transparent querying of many databases from a single database  Familiar programming experience with T-SQL, ADO. NET Linq, EF etc.  Use familiar development and BI tools to query across many databases  Designed for interactive querying

DEMO ELASTIC DATABASE QUERY

ELASTIC DATABASE JOBS  Scenario: Perform management operations across many DBs (e.g. index maintenance, DDL and DML, even queries delivering merged results) Shard Map Manager DB Results DB Database Jobs (cloud service) Control DB Admin/ DevOps DB 1 [0-100)... DB 2 [ ) DB 3 [ ) DB 4 [ ) DB 5 [ ) DB 5 [ ) DB n [n-n+100)

ELASTIC DATABASE JOBS: COMMON USE CASES Make changes to many databases Deploy schema changes across many databases, using T-SQL Update data common to many databases, e.g., reference data Deploy new versions of stored procedures Maintain indexes across a set of databases Manage permissions and logins across a set of databases Collect results from many databases Collect database telemetry from DMVs to monitor data tier performance Gather application-specific metrics and KPIs

DATABASE JOBS – CAPABILITIES  Summary of key capabilities  Execute T-SQL scripts – on-demand, scheduled  Asynchronous, parallel (configurable) execution across databases  Monitor execution progress and review status  Automatic retry in case of failures  Operations across collections of Azure SQL Databases  Elastic Pools  Shard Sets  All DBs on a Server  Ad-hoc collections of DBs  Use stored secured credentials

ELASTIC DATABASE TRANSACTION

DEMO ELASTIC DATABASE TRANSACTIONS

using (var scope = new TransactionScope()) { using ( var conn1 = new SqlConnection(connStrDb1)) { conn1.Open(); SqlCommand cmd1 = conn1.CreateCommand(); cmd1.CommandText = string.Format("insert into T1 values(1)"); cmd1.ExecuteNonQuery(); } using (var conn2 = new SqlConnection(connStrDb2)) { conn2.Open(); var cmd2 = conn2.CreateCommand(); cmd2.CommandText = string.Format("insert into T2 values(2)"); cmd2.ExecuteNonQuery(); } scope.Complete(); } ELASTIC DATABASE TRANSACTION

DYNAMIC MANAGEMENT VIEWS (DMV) All DMVs related to transactions are relevant for distributed transactions in Azure SQL DB. sys.dm_tran_active_snapshot_database_transactionssys.dm_tran_active_transactions sys.dm_tran_current_snapshotsys.dm_tran_current_transaction sys.dm_tran_database_transactionssys.dm_tran_locks sys.dm_tran_session_transactionssys.dm_tran_top_version_generators sys.dm_tran_transactions_snapshotsys.dm_tran_version_store

ELASTIC DB TOOLS: SUMMARY OF CAPABILITIES Shard map management (SMM)  Define groups of shards for your application  Manage mapping of routing keys to shards Data dependent routing (DDR)  Route incoming requests to the correct shard, e.g., given a customer ID  Ensure correct routing as tenants move  Cache routing information for efficiency Multi-shard query (MSQ)  Interactive processing across several shards  Same statement executed on all shards with UNION all semantics Elastic DB Transactions  Support for distributed transactions across Azure SQL DBs Database Jobs (DJ) Asynchronous processing across several databases Split/Merge (SM) Grow or shrink capacity by adding or removing scale units Easily move data between scale units Client.NET APIsManagement Services Elastic Queries (EQ) Interactive reporting across several shards Rich query semantics beyond MSQ Familiar SQL DB connection experience and protocol support Querying Capabilities

ROADMAP & NEXT STEPS Elastic Scale APIs and customer-deployed services  Client Library and Split/Merge: Generally Available  Open Sourced at:  Elastic Database Jobs: currently in Preview  Elastic Database Query: currently in Preview  Elastic Database Transactions: Generally Available

THANKS! Silvia Doomra-