Download presentation
Presentation is loading. Please wait.
Published byWarren White Modified over 6 years ago
1
Design and Implement Cloud Data Platform Solutions
9/9/2018 Design and Implement Cloud Data Platform Solutions © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
2
03 | Implement SQL Database
3
Agenda 1 5 2 6 3 7 4 8 Microsoft Cloud Data Platform
Hybrid HA/DR Scenarios with SQL Server 2 Implement SQL Server on Azure VM 6 Design and Implement Security 3 Implement SQL Database 7 Monitor and Manage Implementations on Azure 4 SQL Database High Availability and Disaster Recovery 8 Design and Implement Database Solutions for SQL Server and SQL Database
4
In this module What is SQL Database Elastic Database Tools
Data Tiers / Elastic data tiers SQL Database performance Scalability Options Service Tier Advisor Elastic Databases
5
Demo Deploy VM using Azure Resource Manager (ARM)
6
Data platform continuum
Hybrid Cloud On premises Off premises Shared Lower cost Platform as a service Azure SQL Database Virtualized Databases Software as a service Infrastructure as a service SQL Server in Azure VM Virtualized Machines Virtual SQL Server Private Cloud Virtualized Machines + Appliances Physical SQL Server Physical Machines (raw iron) Dedicated Higher cost Higher administration Lower administration
7
What is SQL Database A relational database as a service, fully managed by Microsoft. For cloud-designed apps when near-zero administration and enterprise-grade capabilities are key. Perfect for cloud architects and developers looking for programmatic DBA-like functionality. Elastic scale and performance Business continuity and data protection Familiar and self-managed Predictable performance levels Programmatic scale-out Dashboard views of database metrics Self-service restore Disaster recovery Compliance-enabled Familiar and compatible Programmatic Self-managed
8
SQL Database single-database service tiers
Premium Basic Standard Built for Light transactional workloads Medium transactional workloads Heavy transactional workloads Availability SLA 99.99%* Database max size 2 GB 250 GB 1 TB Point-in-time restore (“oops” recovery) Any point within 7 days Any point within 14 days Any point within 35 days Business continuity Geo-restore Geo-replication, standby secondary Active geo-replication, up to four readable secondaries Security Auditing, Row-Level Security, dynamic data masking Performance objectives Transactions per hour Transactions per minute Transactions per second Database Transaction units (DTUs) Basic: 5 S0: 10 S1: 20 S2: 50 S3: 100 P1: 125 P2: 250 P4: 500 P6: 1,000 P11: 1,750 Available tiers ($/month) and GA price Basic: $4.99 S0: $15 S1: $30 S2: $75 S3: $150 P1: $465 P2: $930 P4: $1,860 P6: $3,720 P11: $7,001 *The 99.99% availability SLA does not apply to the existing Web and Business editions, which will continue to be supported at 99.9% availability.
9
Demo Implementing SQL Database
10
How is it different from virtual machines?
SQL Server in a virtual machine Azure SQL Database Best for TCO benefits Scalability Resources Migrating existing applications with no modifications, or very minor changes Applications that need elastic scale and/or reduced infrastructure management overhead IT has adequate resources and bandwidth for support and maintenance Customer does not want to add additional IT resources for support and maintenance Removing CAPEX Avoiding CAPEX and OPEX Scale up to 20,000 IOPS Scale out to thousands of databases, process terabytes of OLTP data
11
SQL Database Elastic Database service tiers
Standard Elastic Preview Elastic Database Pools Resizable eDTU range 200–1200 DTUs Resizable storage range 200–1200 GB Maximum databases per pool 100 databases Elastic database DTUs (eDTU) Each eDTU includes 1 GB of storage $4.5 / eDTU per month (50% pricing for preview – GA price listed above) Elastic Databases eDTU max per database Up to 100 DTUs Storage max per database Up to 250 GB Elastic database fee $2.5 / Database per month (50% pricing for preview – GA price listed above)
12
Hands-on Lab Implementing SQL Database – Part 1
13
Canonical cloud app architecture
Build 2014 9/9/2018 Canonical cloud app architecture Classic 3-tier enterprise architecture Required to scale to 10,000s of users and process terabytes of relational data Scaling out (and in) web and worker tiers is relatively easy How to scale data tier if hard limits of the biggest scale unit (for example, P3 instance) are reached: both storage size and throughput? Microsoft Azure Traffic Manager Azure Queues Write operations mystuff.cloudapp.net Web Roles Worker Roles Azure SQL Database © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
14
Common database scalability patterns
9/9/2018 Common database scalability patterns Fabrikam Single large database Inventory Order Invoice Vertically partitioned Root Cust.#2 1 tenant: 1 database (SaaS ISV) Cust.#1 Cust.#n Root Shard #1 Shard #2 Shard #n Other partitioning scheme © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
15
Scalability options in Azure SQL Database
Vertical: Scale up or scale down Change service tiers for a given database as capacity needs fluctuate Horizontal: Scale out or scale in Add or remove databases (sharded and/or in a pool) as more or less capacity is needed Premium Premium Standard Scale up/down Basic Basic Basic Basic Scale out/in
16
Elastic databases
17
Targeting applications with Azure SQL Database
Cloud-born SaaS Applications ISV SaaS Applications Custom Customer-facing Applications Employee-facing Applications New SaaS providers Goal: Compete & replace on-premises competitors’ applications Needs: Hyper-scale, HADR, Performance Write code once Predictable & flexible cost structure to build a business model Existing SQL ISV Re-architecting for SaaS Goal: Replace existing LOB packaged software with a SaaS solution Needs: Hyper-scale, HADR, Performance SQL compatibility write code once Predictable & flexible cost structure to build a business model All Entities Goal: Ability to scale as the customer base grows Needs: Time to solution Hyper-scale, HADR, Performance Predictable & flexible cost structure to build a business model Enterprises building custom applications Goal: Create or re-architect apps to cloud or use SaaS solutions Needs: Hyper-scale, HADR, Performance SQL compatibility Packaged LOB Applications – Delivered by ISVs Elastic databases Single databases
18
Managing large numbers of databases
Predictable workloads Single databases or partitioned data across multiple databases; scale between service tiers and performance levels as capacity needs fluctuate. Unpredictable workloads For large numbers of databases with unpredictable performance demands; pool resources to be shared between these databases. Single database or partitioned databases Elastic Database Pool reads/writes reads/writes reads/writes reads/writes Databases consume resources as needed P2 Scale databases up as needed S1 #N Customer 1 Customer 2 Customer 3 … Customer #N B … Scale out/in the pool
19
Elastic Database Model
PREVIEW Elastic Database Model Elastic databases in elastic database pools Pooled resources are used by many databases Standard elastic pools provide 200–1,200* database throughput units (DTUs) for up to 100* databases Elastic Standard databases can burst up to 100 DTUs (S3 level) Create/configure pools by using the portal, Azure PowerShell, REST APIs Move databases in/out by using the portal, Azure PowerShell, REST APIs, and T-SQL Databases remain online throughout Monitoring and alerting is available on both pools and databases Max per-database burst level DTUs 200 400 800 1,200 *Additional pricing tiers might be introduced, and the ranges and limits might be increased during the preview
20
What is difficult about building SaaS?
A SaaS model requires supporting a large number of customers Managing many customers and databases is hard How can you deal with isolation versus packing lots of customers into a database? What are your resource needs and economic/business considerations? How can you future proof for scale? How can you handle changing business models? Purchasing individual databases and overprovisioning to meet the variable and peak demand for each database is often not cost efficient. One architectural pattern for SaaS applications is to create one database per end customer.
21
Elastic databases at a glance
Elastic database pools and elastic database pricing model (preview) Elastic database tools—client library and split-merge service (GA) Elastic database job (preview) Elastic database queries and transactions
22
Concepts An elastic database pool is a collection of DTUs and storage that is used by multiple databases Elastic database jobs allow you to perform tasks across databases in the pool Scenarios include Performing administrative tasks, such as deploying new schema Update reference data; for example, product information common across all databases Rebuild indexes to improve query performance Customer 1 Customer 2 Customer 3 Customer #N … Elastic pool Databases consume resources as needed Elastic database tools
23
Performance settings Set DTU guarantee for the group
The DTU guarantee should be within the system limits set for the elastic pool Use historic utilization of the group or the desired guarantee per database and utilization of concurrently active databases to inform the size of the DTU guarantee Set DTU cap and guarantees for the databases DTU cap per database is applied to all databases in the group Use peak utilization patterns of databases within the group and tolerance for overcommitting the DTU guarantee of the group to inform the cap DTU guarantee per database sets the DTU guarantee for all databases in the group; this setting is optional
24
Demo Elastic Pool
25
Creating an elastic database pool
From the Azure SQL Database server blade, click on Add pool. Click on Preview Terms and accept the notice in the Preview Terms blade.
26
Define the elastic database pool
Enter the Name for the pool. Specify the pricing tier; currently there is only one option. Click ADD DATABASES and select the databases from the Add Databases blade.
27
Configure the performance
Select the POOL DTU/GB value of 200, 400, 800 and price scales lineally Specify the elastic database DTU min and max values of 0, 10, 20, 50, 100. Click OK to create the pool. Azure creates the pool on the Starboard to show the status.
28
Configuration is complete
Azure displays the ELASTIC DATABASE POOL (PREVIEW) blade when it completes provisioning. Database blade includes the elastic pool name.
29
Add and remove databases from a pool
Click on the Databases link to show the Elastic databases blade. Click Add database to add another to the pool. Click Remove database to remove a set of databases.
30
Adding alerts Click on the Alert rules in the pool blade to add alert rules. Click Add rule to add a new rule. Select the Resource pool and the metric along with the other information to add an alert, and then click OK to save the changes.
31
Hands-on Lab Implementing SQL Database – Part 2
32
Elastic database tools
33
Elastic database jobs overview
Elastic database jobs enables you to run T-SQL scripts (jobs) against all of the databases in an elastic database pool Define, maintain, and persist T-SQL scripts to be executed across an elastic database pool Execute T-SQL scripts reliably with automatic retry and at scale Track job execution state
34
Getting started with elastic database jobs
Perform management operations on databases in Elastic Pool (the preview is just scoped to this) Index maintenance, DDL, DML, queries delivering merged results For the preview release, it is available as a customer- hosted service Install and deploy the service through the portal
35
Enter in the job credentials used for services
Think of this process as enabling SQL Server Agent with the execution account credentials
36
Add a new elastic job for the resource group
The service allows you to run T-SQL scripts against all of the databases in the pool in a single operation. For example, you can set the policy on each database to allow only people with the right credentials to view sensitive data.
37
Elastic database job dependencies
Cloud service uses a Extra Small worker role VM for managing job execution Storage account used for job management Azure SQL Database used for job management
38
Creating a job In the elastic database job pool blade, click Create job. Type in the name and password of the database administrator (created at installation). In the Create Job blade, type a name for the job. Paste or type in the T-SQL script. Click Save and then click Run.
39
Checking job status Click on the name (a) of a job. The STATUS can be “Completed“ or “Failed.“ The job’s details appear (b) with its date and time of creation and running. The list below the details (c) shows the progress of the script against each database in the pool, giving its date and time details.
40
When is sharding the right consideration?
Application workload can be partitioned across a number of scale units, and workload directed to each partition can fit into the biggest scale unit available. Application workload doesn’t contain large scans or aggregations that need to touch the entire data set. Total capacity demands (CPU, IO, memory, storage) of the application exceed the hard limits of a single Azure SQL Database scale unit. Application requires on-demand or automatic policy-driven scale out or scale in. Caveat: Typical data warehousing does not easily fit the patterns for Elastic Scale.
41
Scalability = Capacity * Density
Adding more capacity Using capacity more efficiently Identifying and breaking contention and choke points How to add additional capacity to a solution? Subtle constraints to consider... Traditional performance tuning Maximize application throughput (for example, leveraging batching) Improving network performance
42
High scale OLTP Large, (mostly) uniform active data set
Distribute partitionable tables across as many shards as necessary Some shards can become more active than others Pinning hot users to dedicated databases Move users to balance workload between databases Scale to the proper service tier (Premium) to deal with increased workload Split-merge actions are required to fully exploit elasticity Mostly data dependent routing with key lookup queries Few fan-out queries may be required (for example, leaderboards and inventory management)
43
Sharding and tenancy models
Single tenant per database Each tenant’s data is stored in a different database Better isolation of tenants as compared to multitenant model Multiple tenants per database Multiple tenants share the same database Less isolation of tenants as compared to single-tenant model Typically more cost-effective than the single-tenant model Hybrid model Some tenants share databases, others get their own database For example, Premium or paying customers get their own databases, while free tier customers share databases Temporal model Sharding based on date/time Most recent shard is constantly loaded with newly arriving data New shards added when most recent shard nears capacity
44
Elastic database client library: Sharding
3 Cross-shard capabilities Client library Cross- shard operations Cross-shard extensions Cross- shard operations Application developer Admin/ DevOps Elastic Scale app Elastic Scale manageability shard1 shardi shardj shardn Client library Shard-local operations 1 Shard-local operations … … … 4 2 Grow/shrink capacity Admin/DevOps
45
Elastic database client library: Key capabilities
Client .NET APIs Management services Shard Map Management (SMM) Define groups of shards for your application Manage mapping of routing keys to shards Data Dependent Routing (DDR) Route incoming requests to the correct shard (for example, given a customer ID) Ensure correct routing as tenants move Cache routing information for efficiency Multi-shard Query (MSQ) Interactive processing across several shards Same statement executed on all shards with UNION all semantics Shard Set Execution (SSE) Interactive processing across several shards Same statement executed on all shards with UNION all semantics Split/Merge (SM) Grow or shrink capacity by adding or removing scale units Dynamically adjust scale factor of scale unit Trigger adjustment dynamically through policies Shard Elasticity (SE)
46
Elastic database client library: Overview
[shardmaps_global] smid name 1 RangeShardMap [shards_global] sid smid Datasource Databasename 1 serverName DB2 2 Two types of shard maps Range: contiguous values List: explicit values Four types of sharding keys INT, BIGINT, GUID, VARBINARY Shard Map Manager [shard_mappings_global] mid smid min max Sid 1 100 2 200 . . . DB1 [0-100) DB3 [ ) DB5 [ ) DB6 [ ) DB2 [ ) DB4 [ ) DBn [n-n+100) Shard Set
47
Elastic database client library: Overview
Two classes of operations: Individual shards Cross shards Two Personas: Application developer Admin/DevOps Application developer Admin/ DevOps Net client API Customer hosted services Query a specific shard Query multiple shards Capacity, cost management Database maintenance, DDL . . . DB1 [0-100) DB3 [ ) DB5 [ ) DB6 [ ) DB2 [ ) DB4 [ ) DBn [n-n+100)
48
Data Dependent Routing (DDR)
Application developer Admin/ DevOps Scenario: Query a shard with a specific shardlet key Shard Map Manager Client app DDR APIs ( ) SELECT * FROM customers WHERE customer ID = 104 . . . DB1 [0-100) DB3 [ ) DB5 [ ) DB6 [ ) DB2 [ ) DB4 [ ) DBn [n-n+100)
49
Shard Set Executer (SSE)
Scenario: Perform an admin action across set of shards Application developer CREATE TABLE foo (col1 INT) Customer Hosted Services (SSE) Admin/ DevOps . . . DB1 [0-100) DB3 [ ) DB5 [ ) DB6 [ ) DB2 [ ) DB4 [ ) DBn [n-n+100)
50
Split/Merge (SM) Perform a split or merge action Merge Split . . .
Scenario: Perform a split or merge action Split: Create two distinct shards from one Merge: Create one shard from two distinct shards Application developer Admin/ DevOps Customer Hosted Services (SM) Merge Split DB1 [0-100) DB2 [ ) DB3 [ ) DB4 [ ) DB5 [ ) DB6 [ ) DBn [n-n+100) . . . DB2.1 [ ) DB5.1 [ ) DB5.2 [ )
51
Shard Elasticity (SE) Scenario: Automation to vertically scale a shard or horizontally scale a shard set Vertical scale: Increase/decrease the performance level of the shard Horizontal scale: Add/remove a shard to the shard set Application developer Admin/ DevOps Azure Automation (SE) Vertical scaling Horizontal scaling DB1 [0-100) DB2 [ ) DB3 [ ) DBn [n-n+100) . . . DB4 [ ) DB5 [ ) DB5 [ ) Basic Basic Basic S2 P1 P3 Time
52
Performance
53
Performance predictability
9/9/2018 Performance predictability Web / Business Basic / Standard / Premium Machine Compute Writes Reads Memory Machine Compute Writes Reads Memory Noisy neighbor! DB 1 workload DB 1 workload DB 2 workload DB 2 workload DB 1 workload DB 2 work-load DB 3 workload DB 4 workload DB 3 work-load DB 7 work-load DB 3 workload DB 4 workload DB 4 workload Introduction of bounding boxes to eliminate the noisy neighbor problem DB 4 workload DB 5 workload DB 7 workload DB 5 workload DB 6 workload DB 7 work-load DB 8 work-load DB 6 workload DB 6 workload DB 5 workload © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
54
Database Throughput Unit – DTU
9/9/2018 Database Throughput Unit – DTU 30% Bounding box Monitoring database workload utilization within bounding box CPU 75% Represents the relative power (resources) assigned to the database Blended measure of CPU, memory, and read-write rates Compare the power across performance levels Simplifies talking about performance, think IOPS vs. % Writes Reads Utilization 50% Memory 60% © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
55
Azure SQL Database Benchmark – ASDB
9/9/2018 Azure SQL Database Benchmark – ASDB An example representing meaningful OLTP-workload Uses six tables of varying sizes, some of which are always larger than available memory and scale with the throughput Uses nine transaction types A transaction is a combination of multiple SELECT, DELETE, INSERT, and UPDATE statements © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
56
Resource governance S2 P1
CPU Writes Reads P1 Governor CPU Writes Reads Governor CPU Writes Reads Governor SELECT * FROM a JOIN b ON … RESULT SELECT * FROM b JOIN c ON … RESULT SELECT * FROM c JOIN d ON … RESULT SELECT * FROM c JOIN d ON … RESULT Simulate the behavior of a dedicated small computer Resource requests are queued instead of rejected Overloading can result in long-running transactions and command timeouts
57
Resource monitoring master.sys.resource_stats
9/9/2018 Resource monitoring master.sys.resource_stats Based on 5-minute averages userdb.sys.dm_db_resource_ stats Based on 15-second averages Percentages relative to performance level Accessible though Azure Portal Enables configuration of alerting © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
58
Service Tier Advisor
59
Overview Service Tier Advisor surfaces insights into the database workload to: Determine the optimal service tier and performance level for the database Choosing right service tier / performance level is dependent on: Feature usage and features available in different Service Tiers Resource usage and bounding boxes of different Performance Levels Service Tier Advisor takes into account: Resource consumption Database Size Disaster Recovery plan Service Tier Advisor does not take into account the following metrics as they are subjective and not measurable: Uptime SLA Point in time Restore Performance Objective
60
Recommended Perf Level Constraints
Recommended performance level must support current database size and disaster recovery plan of a database. Recommended performance level must be suitable for database resource consumption 95% of the time.
61
Service Tier Advisor Components
Basis of the Service Tier Advisor recommendations Feature usage analysis Resource consumption analysis
62
Feature Usage Analysis
Decision tree on the right takes into account the following features: Database Size Disaster Plan Feature that requires larger service tier is marked as dominant as it sets the minimum service tier that supports all used features of a database.
63
Resource consumption analysis
Resource consumption distribution shows what percentage of active period database was consuming resources within the boundaries of different performance levels This chart is an example of resource consumption for a single database
64
Predictable performance levels
Across the service tiers, each performance level is assigned a defined level of throughput for a streamlined experience Redefined Measure of power Introducing the Database Throughput Unit (DTU), which represents database power and replaces hardware specs % CPU % read % write % memory DTU is defined by the bounding box for the resources required by a database workload and measures power across the six performance levels Basic – 5 DTU S0 – 10 DTU S1 – 20 DTU S2 – 50 DTU S3 – 100 DTU P1 – 125 DTU P2 – 250 DTU P3 – 1000 DTU
65
Hands-on Lab Implementing SQL Database – Part 3
Similar presentations
© 2024 SlidePlayer.com. Inc.
All rights reserved.