Azure SQL DWH: Tips and Tricks for developers Sergiy Lunyakin, ITMagination Azure SQL DWH: Tips and Tricks for developers

SQLSat Kyiv Team Yevhen Nedashkivskyi Alesya Zhuk Eugene Polonichko Oksana Borysenko Mykola Pobyivovk Oksana Tkach

Sponsor Sessions Starts at 13:10 Don’t miss them, they might be providing some interesting and valuable information! Room A Room B Room C 13:00 - 13:20 DevArt Microsoft Eleks 13:20 - 13:50 DB Best Intapp DataArt NULL means no session in that room at that time 

Our Awesome Sponsors

Session will begin very soon :) Please complete the evaluation form from your pocket after the session. Your feedback will help us to improve future conferences and speakers will appreciate your feedback! Enjoy the conference!

About me DWH/BI Consultunt at ITMagination Data Platform MVP, MCSE BI, MCSA Cloud Platform Leader of Speaker at SQLSaturdays Organizer of SQLSaturday Lwow Contacts: sergey.lunyakin@gmail.com @slunyakin

Agenda What is Azure SQL DW Architecture of Azure SQL DW Limitations Check compatibility Handling cross-database query Handling Identity Handling ANSI Update/Delete/Merge/SCD Handling Compute columns Handling Cursor

What is Azure SQL DW Microsoft Azure Platform as a Service It’s a Massively Parallel Processing system (MPP) Distributed Compute and Distributed Storage Scale up and down in couple minutes Pause compute resources Supports a subset of T-SQL Join with external data in Azure Blob Storage

Architecture of Azure SQL DW Dist_DB_1 Dist_DB_2 Dist_DB_15 Dist_DB_16 Dist_DB_17 Dist_DB_30 Dist_DB_46 Dist_DB_47 Dist_DB_60 … … … … … …

Logical Overview Control Compute Storage Microsoft Build 2016 11/30/2018 2:09 PM Logical Overview Compute Control Storage © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Distributed queries Query Result Control Compute Storage Microsoft Build 2016 11/30/2018 2:09 PM Distributed queries Query Result Control Compute Storage Scale-out distributed query engine © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Distributed Query SELECT COUNT_BIG(*) FROM dbo.[FactInternetSales] ; SELECT SUM(*) FROM dbo.[FactInternetSales] ; Control Compute SELECT COUNT_BIG(*) FROM dbo.[FactInternetSales] ; SELECT COUNT_BIG(*) FROM dbo.[FactInternetSales] ; SELECT COUNT_BIG(*) FROM dbo.[FactInternetSales] ; SELECT COUNT_BIG(*) FROM dbo.[FactInternetSales] ;

Limitations Primary/Foreign Keys Identity Computed Columns Triggers Cross-database joins Sequences Cursors MERGE ANSI joins on updates/deletes More limitations: https://azure.microsoft.com/en-us/documentation/articles/sql-data-warehouse-migrate-code/ https://feedback.azure.com/forums/307516-sql-data-warehouse

Check compatibility Data warehouse migration utility Free tool Helps to identify unsupported features Helps to identify HASH distribution column Migrate scheama Migrate data (BCP tool)

Cross-database query Azure SQL DW doesn’t support cross-database query. Use ELT approach. Separate schemas. Use External tables as staging tables.

CTAS CTAS is super-charched version of SELECT...INTO Parallelized Better for Data import Data copy Workarounds CREATE TABLE [dbo].[FactInternetSales_new] WITH ( DISTRIBUTION = ROUND_ROBIN , CLUSTERED COLUMNSTORE INDEX ) AS SELECT * FROM [dbo].[FactInternetSales];

Identity Handle it on source side IDENTITY property Explicit import Doesn’t support CTAS Custom Identity with ROW_NUMBER

ANSI JOINS Update/Del/Merge Update/Delete doesn’t support JOINS in FROM Use CTAS for preparing interim table with JOINS Use CTAS for Merge workaround Split Merge to operation steps and use UNION ALL Use interim table for big number of steps Use partitioning for big tables, don’t reload the whole table

Compute columns Handle it in source system Use CTAS during import Create a View Use explicit data type and nullability check Wrong data during migration Schema error during partition switch

Cursor WHILE for lopping List of elements as a table Loop through list using While clause and variable

Summary MPP PaaS Service in Azure Cloud Storing and processing huge amount of structure data Limitation: Identity, ANSI JOINS, MERGE CTAS - Super-charched version of SELECT...INTO CTAS good way for workarounds Better reload data with CTAS than Row-By-Row operations

Question?

Our Awesome Sponsors