Why Is My SQL DW Query Slow? Microsoft Ignite 2016 6/14/2018 11:55 PM Why Is My SQL DW Query Slow? DA443 Rob Farley, LobsterPot Solutions & Bhavik Merchant, Microsoft © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

About Us Rob Farley, LobsterPot Solutions Bhavik Merchant, Microsoft 6/14/2018 11:55 PM About Us Rob Farley, LobsterPot Solutions SQL MVP, MCM, MCT Consultant and Trainer in APS/PDW and SQL DW Bhavik Merchant, Microsoft Data Platform Solution Architect © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6/14/2018 11:55 PM Abstract / Agenda Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Key takeaway – SQL DW is there for scale out 6/14/2018 11:55 PM Abstract / Agenda Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. Bhavik will cover the basic concepts – but quickly because you got it this morning already from Matt Usher’s session. Key takeaway – SQL DW is there for scale out © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Bhavik will talk about why SQL DW is fast. 6/14/2018 11:55 PM Abstract / Agenda Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. Bhavik will talk about why SQL DW is fast. It really is. Rob agrees. © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Rob will talk about what’s going on when a query runs. 6/14/2018 11:55 PM Abstract / Agenda Rob will talk about what’s going on when a query runs. Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

…and highlight where things can be slower than you intend 6/14/2018 11:55 PM Abstract / Agenda …and highlight where things can be slower than you intend Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

…because after all, you should get the performance you deserve! 6/14/2018 11:55 PM Abstract / Agenda Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. …because after all, you should get the performance you deserve! © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6/14/2018 11:55 PM Abstract / Agenda Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Cortana Intelligence The Azure Data Platform at a Glance Build 2015 6/14/2018 11:55 PM Cortana Intelligence The Azure Data Platform at a Glance Information Management Azure Data Factory Data Catalog Event Hub Big Data Stores Azure Data Lake Store Azure SQL Data Warehouse Machine Learning and Analytics Azure Machine Learning Hadoop, Spark, Storm, Hbase Managed Clusters (Azure Data Lake) Azure Stream Analytics Dashboards and Visualizations Power BI Business apps Custom apps Sensors and devices Personal Digital Assistant Cortana People Perceptual Intelligence Face, vision Speech, text Azure Data Lake Analytics Automated Systems Business Scenarios Recommendations, customer churn, forecasting, etc. DATA INTELLIGENCE ACTION © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

SMP vs MPP SMP/NUMA Massively parallel processing (MPP) 6/14/2018 SMP vs MPP Multiple CPUs are used to complete individual processes simultaneously All CPUs share the same memory (SMP) OR different Groups of CPUs use different sets of memory on the same machine (NUMA) All SQL Server implementations up until now have been SMP/NUMA SMP/NUMA Multiple Nodes (computers) get utilized to process a single task Many separate CPUs running in parallel across multiple nodes to execute a single task Each set of CPUs has its own memory Applications must be segmented, using high-speed communications between nodes Massively parallel processing (MPP) © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

What’s in the Box? Compute Scale compute up or down when required Storage and Compute are de-coupled, enabling a true elastic service and separate charging for both compute and storage Application or User connection Control Node DMS (Data Movement Service) executes across all database nodes SQL DB Data Loading DMS Massively Parallel Processing (MPP) Engine Compute Scale compute up or down when required Pause, Restart, Stop, Start. DMS DMS DMS DMS SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node Storage Add/Load data to WASB(S) without incurring compute costs Blob storage [WASB(S)] Azure Infrastructure + Storage

Control Node Control Node Endpoint for connections SQL DB Control Node SQL DB Massively Parallel Processing (MPP) Engine SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node Endpoint for connections Regular SQL endpoint (TCP 1433) Coordinates compute activity Blob storage [WASB(S)] HDInsight

Compute Nodes Compute Node(s) Azure SQL Database SQL DB Control Node SQL DB Massively Parallel Processing (MPP) Engine SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node We can change the number of compute nodes by changing DWU Blob storage [WASB(S)]

Compute Nodes Again Compute Node(s) Azure SQL Database SQL DB Control Node SQL DB Massively Parallel Processing (MPP) Engine SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node An increase of DWU will increase the number of compute nodes Blob storage [WASB(S)]

DW Who? Data Warehouse Unit (DWU) Derived from tests, gives competitive performance Blocks of 100, from 100 to 6000. Not all multiples. How many do I need? Start small, test performance Increase till you reach required performance levels Scales linearly Changes applied within 5 mins

Architecture of SQL DW for DWU200 6/14/2018 Architecture of SQL DW for DWU200 Azure SQL Data Warehouse Engine Worker2 Worker1 Azure Storage Blob(s) D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15 D16 D17 D18 D19 D20 D21 D22 D23 D24 D25 D26 D27 D28 D29 D30 D31 D32 D33 D34 D35 D36 D37 D38 D39 D40 D41 D42 D43 D44 D45 D46 D47 D48 D49 D50 D51 D52 D53 D54 D55 D56 D57 D58 D59 D60 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Architecture of SQL DW for DWU600 6/14/2018 Architecture of SQL DW for DWU600 Azure SQL Data Warehouse Engine Worker4 Worker1 Worker5 Worker3 Worker2 Worker6 Azure Storage Blob(s) D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15 D16 D17 D18 D19 D20 D21 D22 D23 D24 D25 D26 D27 D28 D29 D30 D31 D32 D33 D34 D35 D36 D37 D38 D39 D40 D41 D42 D43 D44 D45 D46 D47 D48 D49 D50 D51 D52 D53 D54 D55 D56 D57 D58 D59 D60 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Query Execution Control Node accepts the query 6/14/2018 11:55 PM Query Execution Control Node accepts the query MPP Query Optimizer works out what to do …using statistics on the control node which are brought up from the compute nodes Data Movement as required Instructions given out to nodes EXPLAIN plan Local SQLPLANs …using statistics on the individual nodes © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo Running queries OPTION (LABEL = ‘Some Label') 6/14/2018 11:55 PM Demo Running queries OPTION (LABEL = ‘Some Label') Exploring space used Query steps MPP Plans Local plans Data Movement Workers © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

When your choices hurt Loading Skew Joins & Aggregations 6/14/2018 11:55 PM When your choices hurt Loading SSIS currently not parallel ..use AZCopy to transfer flat files to blob storage, then PolyBase Skew A disproportionate amount of data on a single distribution Joins & Aggregations Too much data movement going on Distribute for hefty joins and aggregations © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

When your choices hurt Over-partitioning Statistics & Cardinality 6/14/2018 11:55 PM When your choices hurt Over-partitioning Remember you already have 60 distributions Columnstore row groups are 1M rows So if you have under 60M rows – don’t partition any more Statistics & Cardinality Manage statistics yourself – currently no auto-update for control-node level statistics No uniqueness, so write queries that aggregate the fact table first © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Summary Don’t just accept the defaults Know the impact of choices 6/14/2018 11:55 PM Summary Don’t just accept the defaults Know the impact of choices Read the best practices guides online Keep abreast of announcements Statistics!! Monitor Skew - DBCC PDW_SHOWSPACEUSED('dbo.SomeTable'); What’s happening with your Distributed Plans What’s happening with your SQL Plans © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Questions? Or email us! rob@lobsterpot.com.au 6/14/2018 11:55 PM Questions? Or email us! rob@lobsterpot.com.au bhavik.merchant@microsoft.com © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Continue your Ignite learning path 6/14/2018 11:55 PM Continue your Ignite learning path Visit Channel 9 to access a wide range of Microsoft training and event recordings https://channel9.msdn.com/ Head to the TechNet Eval Centre to download trials of the latest Microsoft products http://Microsoft.com/en-us/evalcenter/ Visit Microsoft Virtual Academy for free online training visit https://www.microsoftvirtualacademy.com © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6/14/2018 11:55 PM Thank you Chat with me in the Speaker Lounge Find us: @rob_farley or rob@lobsterpot.com.au Bhavik.Merchant@microsoft.com © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.