2. A developers guide to Azure SQL Data Warehouse
Microsoft Build 2016
9/19/2018 7:06 PM
A developers guide to Azure SQL Data Warehouse
James Rowland-Jones (JRJ)
Azure SQL Data Warehouse explained
Provisioning the database
Designing tables
Loading data
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. What is SQL DW? (and when to use it)

4. Analytical workloads Store large volumes of data
Microsoft Build 2016
9/19/2018 7:06 PM
Analytical workloads
Store large volumes of data
Consolidate disparate data into a single location
Shape, model, transform and aggregate data
Perform query analysis across large datasets
Ad-hoc reporting across large data volumes
All using simple SQL constructs
Petabyte scale
“SQL on SQL”
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Unsuitable workloads Operational workloads (OLTP)
High frequency reads & writes
Large numbers of singleton selects
High volumes of single row inserts
Procedural ETL
Row by row processing needs
Incompatible formats (JSON, XML)

6. Logical Overview Control Compute Storage Microsoft Build 2016
9/19/2018 7:06 PM
Logical Overview
Compute
Control
Storage
De-coupled storage from compute & control
Completely elastic
Pay for the data you store and the compute you provision
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Distributed queries Query Result Control Compute Storage
Microsoft Build 2016
9/19/2018 7:06 PM
Distributed queries
Result
Query
Control
Compute
Scale-out distributed query engine
Storage
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Fully managed PaaS Microsoft Build 2016 9/19/2018 7:06 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Geo-redundant Microsoft Build 2016 9/19/2018 7:06 PM
Platform as a Service (PaaS)
Leverages the cloud
Database snapshots are taken on a regular basis to support a global 8 hour RPO
A geo redundant snapshot can be used to bring online the database in any region
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Connectivity Windows or Linux ODBC JDBC ADO.NET PHP
Microsoft Build 2016
9/19/2018 7:06 PM
Connectivity
Windows or Linux
ODBC
JDBC
ADO.NET
PHP
Customers also using FreeTDS from Linux
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Summary Scale-out distributed query engine
Microsoft Build 2016
9/19/2018 7:06 PM
Summary
Scale-out distributed query engine
De-coupled storage from compute
Fully managed
Completely elastic
Platform as a Service (PaaS)
Petabyte scale
Leveraging cloud ecosystem
Broad range of connectivity options
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. IoT Scenario Persist Export Query Stream Import Write Red Stars
Archive on the way in to the warehouse
Geo-redundant blob storage guarantees your data
Allows for RPO 0 Data Warehouse architectures
Allows for point in time restart-ability of ELT
Resume / Pause Pattern

13. Provisioning SQL DW Microsoft Build 2016 9/19/2018 7:06 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Demo: Provisioning

15. Summary Full provisioning experience Partial provisioning experience
PowerShell
Azure portal
REST API
Partial provisioning experience
T-SQL CREATE DATABASE
T-SQL scale with ALTER DATABASE
sys.database_service_objectives shows current configuration
sys.dm_operation_status shows progress performing provisioning operations

16. Designing tables Microsoft Build 2016 9/19/2018 7:06 PM
Two types of table
ROUND_ROBIN
HASH
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. ROUND ROBIN DISTRIBUTION01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

18. HASH DISTRIBUTION 02 01 01 03 HASH ( ) 01 02 03 04 05 06 07 08 09 1011 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

19. Demo: Table design

20. Guidance Keep column definitions strongly typed
Distribution key is read only
Aim for a not null definition for distribution key
Ensure columns are defined consistently
Especially true for the distribution key
Hash distribution optimises data layout
JOIN and GROUP BY columns are often the best candidates
Partition for data management
Consider columns used in the WHERE clause
Evaluate query date ranges (month, quarter) as part of the partitioning strategy

21. Loading data

22. Demo: Loading Data

23. Loading compressed text
Guidance
Evenly split the data into multiple files
One file per reader
DWU
Readers
Writers
DW100 8 60
DW200 16
DW300 24
DW400 32
DW500 40
DW600 48
DW1000

24. Batching trickle loads
>= 102,400 rows per distribution = Columnstore
<102,400 rows per distribution = Row storage
Assuming even distribution:
6,144,000+ rows required in Bulk Insert
500
Rows/Sec
1000
2000
Load
threshold exceeded (hours)
<3.5 hours
< 2 hours
< 1 hour

25. Other loading methods Azure Data Factory SSIS Bcp
3rd party data loading tools

26. Resources PolyBase loading: http://aka.ms/acom-polybase-load
SQL DW docs:
Re-visit Build on Channel 9
Continue your education at Microsoft Virtual Academy online