Managing batch processing Transient Azure SQL Warehouse Resource Cloud Migration Managing batch processing Transient Azure SQL Warehouse Resource

Cloud Migration Concepts

Azure SQL Warehouse Concurrency Batch processing Large memory needs for performance Partitioning Indexing and statistics

KNOWING THE VARIOUS BIG DATA SOLUTIONS CONTROL EASE OF USE Any Hadoop technology, any distribution Workload optimized, managed clusters Data Engineering in a Job-as-a-service model Azure Marketplace Azure Data Lake Analytics IaaS Clusters Managed Clusters Big Data as-a-service Azure HDInsight Frictionless & Optimized Spark clusters SQL Data Warehouse SQL Logic Reduced Administration BIG DATA ANALYTICS Azure Data Lake Analytics Polybase Azure Data Lake Store Azure Storage BIG DATA STORAGE

Distributions Azure SQL Warehouse base of 60 distributions (hash or round-robin) Elastic 1 – 60 distributions associated to a unit of compute. Limit amount of data skew Limit amount of data movement by aligning processing with hash-key distribution

Service level Max concurrent queries Compute nodes Distributions per Compute node Max memory per distribution (MB) Max memory per data warehouse (GB) Cost DW100 4 1 60 400 24 $1.21/hour DW200 8 2 30 800 48 $2.42/hour DW300 12 3 20 1,200 72 $3.63/hour DW400 16 15 1,600 96 $4.839/hour DW500 5 2,000 120 $6.049/hour DW600 6 10 2,400 144 $7.259/hour DW1000 32 4,000 240 $12.10/hour DW1200 4,800 288 $14.52/hour DW1500 6,000 360 $18.15/hour DW2000 8,000 480 $24.20/hour DW3000 12,000 720 $36.30/hour DW6000 24,000 1440 $72.59/hour

DWU smallrc mediumrc largerc xlargerc DW100 100 200 400 DW200 800 DW300 DW400 1,600 DW500 DW600 DW1000 3,200 DW1200 DW1500 DW2000 6,400 DW3000 DW6000 12,800

Processing in Memory Joining polybase tables in this case is not recommended as we do want to complete all operations in memory so all polybase tables are initially loaded directly to temporary tables. All intermediate work tables should be temporary tables not physical.