Azure SQL DWH: Tips and Tricks for developers Sergiy Lunyakin Azure SQL DWH: Tips and Tricks for developers

Sponsors!

About me I’m Ukrainian DWH/BI Consultant at ITMagination Data Platform MVP, MCSE BI, MCSA Cloud Platform Leader of Speaker at SQL Conferences Organizer of SQLSaturday Lwow Contacts: sergey.lunyakin@gmail.com @slunyakin

Agenda What is Azure SQL DW Architecture of Azure SQL DW Limitations Check compatibility Handling cross-database query Handling Identity Handling ANSI Update/Delete/Merge/SCD Handling Compute columns Handling Cursor

What is Azure SQL DW Microsoft Azure Platform as a Service It’s a Massively Parallel Processing system (MPP) Distributed Compute and Distributed Storage Scale up and down in several minutes Pause compute resources Supports a subset of T-SQL Join with external data in Azure Blob Storage/Data Lake

Architecture of Azure SQL DW Dist_DB_1 Dist_DB_2 Dist_DB_15 Dist_DB_16 Dist_DB_17 Dist_DB_30 Dist_DB_46 Dist_DB_47 Dist_DB_60 … … … … … …

Logical Overview Control Compute Storage Microsoft Build 2016 4/10/2019 1:58 AM Logical Overview Compute Control Storage © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Distributions Distribution – SQL Database which stores one or more distributed table Splits data table to 60 buckets through compute nodes Hash distributed table * Round-Robin distributed table * Replicate table - New type of table * Selecting the right distribution method is key for good performance

Distributed queries Query Result Control Compute Storage Microsoft Build 2016 4/10/2019 1:58 AM Distributed queries Query Result Control Compute Storage Scale-out distributed query engine © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Distributed Query SELECT COUNT_BIG(*) FROM dbo.[FactInternetSales] ; SELECT SUM(*) FROM dbo.[FactInternetSales] ; Control Compute SELECT COUNT_BIG(*) FROM dbo.[FactInternetSales] ; SELECT COUNT_BIG(*) FROM dbo.[FactInternetSales] ; SELECT COUNT_BIG(*) FROM dbo.[FactInternetSales] ; SELECT COUNT_BIG(*) FROM dbo.[FactInternetSales] ;

Limitations Primary/Foreign Keys Identity Computed Columns Triggers Cross-database joins Sequences Cursors MERGE ANSI joins on updates/deletes More limitations: https://azure.microsoft.com/en-us/documentation/articles/sql-data-warehouse-migrate-code/ https://feedback.azure.com/forums/307516-sql-data-warehouse

Check compatibility Data warehouse migration utility Free tool Helps to identify unsupported features Helps to identify HASH distribution column Migrate scheama Migrate data (BCP tool)

Cross-database query Azure SQL DW doesn’t support cross-database query. Use ELT approach. Separate schemas. Use External tables as staging tables.

CTAS CTAS is super-charched version of SELECT...INTO Parallelized Better for Data import Data copy Workarounds CREATE TABLE [dbo].[FactInternetSales_new] WITH ( DISTRIBUTION = ROUND_ROBIN , CLUSTERED COLUMNSTORE INDEX ) AS SELECT * FROM [dbo].[FactInternetSales];

Identity Handle it on source side IDENTITY property Explicit import Doesn’t support CTAS Custom Identity with ROW_NUMBER

ANSI JOINS Update/Del/Merge Update/Delete doesn’t support JOINS in FROM Use CTAS for preparing interim table with JOINS Use CTAS for Merge workaround Split Merge to operation steps and use UNION ALL Use interim table for big number of steps Use partitioning for big tables, don’t reload the whole table

Compute columns Handle it in a source system Use CTAS during import Create a View Use explicit data type and nullability check in you calculation expressions Wrong data during migration Schema error during partition switch

Cursor Use WHILE for lopping Prepare a list of elements as a table Loop through this list using While clause and variable Do some action

Summary MPP PaaS Service in Azure Cloud Storing and processing huge amount of structure data Limitation: Identity, ANSI JOINS, MERGE CTAS - Super-charged version of SELECT...INTO CTAS good way for workarounds Better reload data with CTAS than Row-By-Row operations

Sponsors!

The end