Download presentation
Presentation is loading. Please wait.
1
Why Is My SQL DW Query Slow?
Microsoft Ignite 2016 6/14/ :55 PM Why Is My SQL DW Query Slow? DA443 Rob Farley, LobsterPot Solutions & Bhavik Merchant, Microsoft © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
2
About Us Rob Farley, LobsterPot Solutions Bhavik Merchant, Microsoft
6/14/ :55 PM About Us Rob Farley, LobsterPot Solutions SQL MVP, MCM, MCT Consultant and Trainer in APS/PDW and SQL DW Bhavik Merchant, Microsoft Data Platform Solution Architect © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
3
6/14/ :55 PM Abstract / Agenda Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
4
Key takeaway – SQL DW is there for scale out
6/14/ :55 PM Abstract / Agenda Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. Bhavik will cover the basic concepts – but quickly because you got it this morning already from Matt Usher’s session. Key takeaway – SQL DW is there for scale out © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
5
Bhavik will talk about why SQL DW is fast.
6/14/ :55 PM Abstract / Agenda Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. Bhavik will talk about why SQL DW is fast. It really is. Rob agrees. © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
6
Rob will talk about what’s going on when a query runs.
6/14/ :55 PM Abstract / Agenda Rob will talk about what’s going on when a query runs. Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
7
…and highlight where things can be slower than you intend
6/14/ :55 PM Abstract / Agenda …and highlight where things can be slower than you intend Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
8
…because after all, you should get the performance you deserve!
6/14/ :55 PM Abstract / Agenda Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. …because after all, you should get the performance you deserve! © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
9
6/14/ :55 PM Abstract / Agenda Azure SQL Data Warehouse offers some great technology to help you scale out. Loading data can be lightning fast, and analytical queries can be super quick. What about when it all slows down? Let’s explore some internals of SQL DW, show you where things can go wrong with loading, transforming, and querying, and how to get the performance that you would expect. © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
10
Cortana Intelligence The Azure Data Platform at a Glance
Build 2015 6/14/ :55 PM Cortana Intelligence The Azure Data Platform at a Glance Information Management Azure Data Factory Data Catalog Event Hub Big Data Stores Azure Data Lake Store Azure SQL Data Warehouse Machine Learning and Analytics Azure Machine Learning Hadoop, Spark, Storm, Hbase Managed Clusters (Azure Data Lake) Azure Stream Analytics Dashboards and Visualizations Power BI Business apps Custom apps Sensors and devices Personal Digital Assistant Cortana People Perceptual Intelligence Face, vision Speech, text Azure Data Lake Analytics Automated Systems Business Scenarios Recommendations, customer churn, forecasting, etc. DATA INTELLIGENCE ACTION © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
11
SMP vs MPP SMP/NUMA Massively parallel processing (MPP)
6/14/2018 SMP vs MPP Multiple CPUs are used to complete individual processes simultaneously All CPUs share the same memory (SMP) OR different Groups of CPUs use different sets of memory on the same machine (NUMA) All SQL Server implementations up until now have been SMP/NUMA SMP/NUMA Multiple Nodes (computers) get utilized to process a single task Many separate CPUs running in parallel across multiple nodes to execute a single task Each set of CPUs has its own memory Applications must be segmented, using high-speed communications between nodes Massively parallel processing (MPP) © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
12
What’s in the Box? Compute Scale compute up or down when required
Storage and Compute are de-coupled, enabling a true elastic service and separate charging for both compute and storage Application or User connection Control Node DMS (Data Movement Service) executes across all database nodes SQL DB Data Loading DMS Massively Parallel Processing (MPP) Engine Compute Scale compute up or down when required Pause, Restart, Stop, Start. DMS DMS DMS DMS SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node Storage Add/Load data to WASB(S) without incurring compute costs Blob storage [WASB(S)] Azure Infrastructure + Storage
13
Control Node Control Node Endpoint for connections
SQL DB Control Node SQL DB Massively Parallel Processing (MPP) Engine SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node Endpoint for connections Regular SQL endpoint (TCP 1433) Coordinates compute activity Blob storage [WASB(S)] HDInsight
14
Compute Nodes Compute Node(s) Azure SQL Database
SQL DB Control Node SQL DB Massively Parallel Processing (MPP) Engine SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node We can change the number of compute nodes by changing DWU Blob storage [WASB(S)]
15
Compute Nodes Again Compute Node(s) Azure SQL Database
SQL DB Control Node SQL DB Massively Parallel Processing (MPP) Engine SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node SQL DB Compute Node An increase of DWU will increase the number of compute nodes Blob storage [WASB(S)]
16
DW Who? Data Warehouse Unit (DWU)
Derived from tests, gives competitive performance Blocks of 100, from 100 to Not all multiples. How many do I need? Start small, test performance Increase till you reach required performance levels Scales linearly Changes applied within 5 mins
17
Architecture of SQL DW for DWU200
6/14/2018 Architecture of SQL DW for DWU200 Azure SQL Data Warehouse Engine Worker2 Worker1 Azure Storage Blob(s) D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15 D16 D17 D18 D19 D20 D21 D22 D23 D24 D25 D26 D27 D28 D29 D30 D31 D32 D33 D34 D35 D36 D37 D38 D39 D40 D41 D42 D43 D44 D45 D46 D47 D48 D49 D50 D51 D52 D53 D54 D55 D56 D57 D58 D59 D60 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
18
Architecture of SQL DW for DWU600
6/14/2018 Architecture of SQL DW for DWU600 Azure SQL Data Warehouse Engine Worker4 Worker1 Worker5 Worker3 Worker2 Worker6 Azure Storage Blob(s) D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15 D16 D17 D18 D19 D20 D21 D22 D23 D24 D25 D26 D27 D28 D29 D30 D31 D32 D33 D34 D35 D36 D37 D38 D39 D40 D41 D42 D43 D44 D45 D46 D47 D48 D49 D50 D51 D52 D53 D54 D55 D56 D57 D58 D59 D60 © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
19
Query Execution Control Node accepts the query
6/14/ :55 PM Query Execution Control Node accepts the query MPP Query Optimizer works out what to do …using statistics on the control node which are brought up from the compute nodes Data Movement as required Instructions given out to nodes EXPLAIN plan Local SQLPLANs …using statistics on the individual nodes © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
20
Demo Running queries OPTION (LABEL = ‘Some Label')
6/14/ :55 PM Demo Running queries OPTION (LABEL = ‘Some Label') Exploring space used Query steps MPP Plans Local plans Data Movement Workers © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
21
When your choices hurt Loading Skew Joins & Aggregations
6/14/ :55 PM When your choices hurt Loading SSIS currently not parallel ..use AZCopy to transfer flat files to blob storage, then PolyBase Skew A disproportionate amount of data on a single distribution Joins & Aggregations Too much data movement going on Distribute for hefty joins and aggregations © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
22
When your choices hurt Over-partitioning Statistics & Cardinality
6/14/ :55 PM When your choices hurt Over-partitioning Remember you already have 60 distributions Columnstore row groups are 1M rows So if you have under 60M rows – don’t partition any more Statistics & Cardinality Manage statistics yourself – currently no auto-update for control-node level statistics No uniqueness, so write queries that aggregate the fact table first © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
23
Summary Don’t just accept the defaults Know the impact of choices
6/14/ :55 PM Summary Don’t just accept the defaults Know the impact of choices Read the best practices guides online Keep abreast of announcements Statistics!! Monitor Skew - DBCC PDW_SHOWSPACEUSED('dbo.SomeTable'); What’s happening with your Distributed Plans What’s happening with your SQL Plans © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
24
Questions? Or email us! rob@lobsterpot.com.au
6/14/ :55 PM Questions? Or us! © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
25
Continue your Ignite learning path
6/14/ :55 PM Continue your Ignite learning path Visit Channel 9 to access a wide range of Microsoft training and event recordings Head to the TechNet Eval Centre to download trials of the latest Microsoft products Visit Microsoft Virtual Academy for free online training visit © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
26
6/14/ :55 PM Thank you Chat with me in the Speaker Lounge Find us: @rob_farley or © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Similar presentations
