2. Context about the Data Warehouse

3. Growing Market Large entry cost High switching cost Small Market
Small entry cost
Lower switching cost
Large Market
November 2011
BigQuery
February 2013
Redshift
July 2016
Azure SQL DW

4. Today, 80% of organizations adopt cloud-first strategies
Data will grow to 44 ZB in 2020
Today, 80% of organizations adopt cloud-first strategies
AI investment increased by
300%

5. A foundation of any analytics platform
Cloud Data Warehouse
Big Data
Modern Data Warehouse
Advanced Analytics
Advanced Analytics on Big Data
Real-Time
Real-Time Advanced Analytics

6. Cloud Data Warehouse INGEST STORE PREP & TRAIN MODEL & SERVE
Business/custom apps
(Structured)
Azure Data Factory
Polybase
Azure SQL Data Warehouse
Azure Analysis Services
Analytical dashboards
Azure Blob Storage

7. Modern Data Warehouse INGEST STORE PREP & TRAIN MODEL & SERVE
Logs, files and media
(unstructured)
Azure Data Factory
Azure HDInsight
Azure Databricks
Azure Blob Storage
Hot Path
Business/custom apps
(Structured)
Polybase
Operational Reports
&
Analytical Dashboards (PowerBI)
Azure Data Factory
Azure SQL Data Warehouse
Azure Analysis Services

8. Advanced Analytics on Big Data
INGEST
STORE
PREP & TRAIN
MODEL & SERVE
Logs, files and media
(unstructured)
Azure Machine Learning
SQL Machine Learning
Azure Data Factory
Azure HDInsight
Azure Blob Storage
Azure Databricks
Hot Path
Business/custom apps
(Structured)
Polybase
Operational Reports
&
Analytical Dashboards (PowerBI)
Azure Data Factory
Azure SQL Data Warehouse
Azure Analysis Services

9. Real time advanced analytics
INGEST
STORE
PREP & TRAIN
MODEL & SERVE
Logs, files and media
(unstructured)
Azure Machine Learning
SQL Machine Learning
Azure IoT Hub
Sensors and IoT
(unstructured)
Azure HDInsight (Kafka)
Azure HDInsight
Azure Databricks
Hot Path
Business/custom apps
(Structured)
Polybase
Operational Reports
&
Analytical Dashboards (PowerBI)
Azure Data Factory
Azure Blob Storage
Azure SQL Data Warehouse
Azure Analysis Services

10. SQL DW evolution

11. 2/5/ :04 AM
Generation 1
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Legacy: tightly coupled compute and storage
2/5/ :04 AM
Legacy: tightly coupled compute and storage
$$$
Compute
Storage
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Cloud: Separated Compute and Storage
2/5/ :04 AM
Cloud: Separated Compute and Storage
Control
Compute
Remote
Storage
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Scale on Demand Control Compute Remote Storage 2/5/2019 11:04 AM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Pause & Resume Control Compute Remote Storage 2/5/2019 11:04 AM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Generation 1 Cores Memory SSD TempDB Cores Memory SSD TempDB Cores
2/5/ :04 AM
Generation 1
Cores
Memory
SSD TempDB
Control
Cores
Memory
SSD TempDB
Cores
Memory
SSD TempDB
Cores
Memory
SSD TempDB
Compute
Remote Storage
Snapshot backups
Data
Log
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. 2/5/ :04 AM
Generation 2
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Cloud: Separated Compute and Storage
2/5/ :04 AM
Cloud: Separated Compute and Storage
Control
Compute
Intelligent
Cache
Remote
Storage
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Generation 2 Cores Memory SSD TempDB Cache TempDB NVMe SSD Cores
2/5/ :04 AM
Generation 2
Cores
Memory
SSD TempDB
Control
Cache
TempDB
NVMe SSD
Cores
Memory
Cache
TempDB
NVMe SSD
Cores
Memory
Cache
TempDB
NVMe SSD
Cores
Memory
Compute
Data
Remote storage
Snapshot backups
Log
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

20. Automated tiering of storage
2/5/ :04 AM
Automated tiering of storage
Node
Capacity
Access
Time
Memory
Cache
Remote Storage
400GB+
70 ns
1.5TB
20 µs ∞
100 ms
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Intelligent Cache Latest hardware
2/5/ :04 AM
Intelligent Cache
Latest hardware
Keeps data close to the CPUs
Removes the remote I/O from query
Data maintained in cache based on LRU2 algorithms
Pre-emptive caching will improve performance further
1.5 TB per 500 compute DWU
Hashed data maintained on the compute nodes
Maximizes performance for distribution aligned queries
Generation 1 hashed data in storage layer
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. ∞ 5x Storage Scalability 5x 2/5/2019 11:04 AM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Compute Scalability 2/5/2019 11:04 AM
5X more compute scale on Gen2. Bigger and faster nodes.
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. Data Warehouse Units Normalized amount of compute
Converts to billing units i.e. what you pay
DWU
100
200
300
400
500
600
1000
1200
1500
2000
3000
6000
CPU
RAM
I/O
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Benefits from workload isolation Cost optimization
2/5/ :04 AM
Find consumption patterns and leverage Azure Functions to
Auto-scale
Time-scale
And save money
Coming soon: template to auto/time-scale DW (can apply to SQL DB)
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Starting point: Sizing by capacity
Microsoft Build 2016
2/5/ :04 AM
Starting point: Sizing by capacity
0-4 TBs
4-8 TBs
8-12 TBs
12-16 TBs
16-20 TBs
20-36 TBs
36-48 TBs
48-60 TBs
60-80 TBs
TBs
>160 TBs
100
200
300
400
500
600
1000
1200
1500
2000
3000
6000
Recommended starting point
Flexibility to select any range of DWUs
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. Changing Service Level Objectives (SLO)
2/5/ :04 AM
Changing Service Level Objectives (SLO)
ALTER DATABASE sqltelemetry
MODIFY
(service_objective = 'DW1000' ) ;
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. SQL DW Architecture 2/5/2019 Control – “The Brain”
Application or User connection
Control – “The Brain”
Connection and tool endpoint. Coordinates storage/compute activity.
Control
Engine
Data Loading
(ADF, SSIS, REST, OLE, ODBC, ADF, AZCopy, PowerShell)
DMS
SQL DB
Compute – “The Brawn”
Handles query processing, ability to scale up/down
Compute
Compute
Compute
Compute
DMS
DMS
DMS
DMS
SQL DB
SQL DB
SQL DB
SQL DB
Dist_DB_1
Dist_DB_2
Dist_DB_15
Dist_DB_16
Dist_DB_17
Dist_DB_30
Dist_DB_31
Dist_DB_32
Dist_DB_45
Dist_DB_46
Dist_DB_47
Dist_DB_60
Data Movement Services
Coordinates data movement between nodes/storage … … … …
Storage
Database data and log files are stored on WASB separate from compute
Blob storage [WASB(S)]
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29. 2/5/ :04 AM
Query Execution
Compute
Dist_DB_1
Dist_DB_2
Dist_DB_15
DMS
SQL DB
Dist_DB_16
Dist_DB_17
Dist_DB_30 …
Dist_DB_31
Dist_DB_32
Dist_DB_45
Dist_DB_46
Dist_DB_47
Dist_DB_60
Engine
Shell DB 1
CONTROL
Queries 5
0. Batch (batch, sp)
1. Query submitted
Submit Time
2. Parsed
3. Optimized – MEMO generated, plan chosen, check some perms
Start Time
4. Object locks acquired
End Compile Time
5. System locks acquired – wait for concurrency slot
6. Executed
End Time 2 3 6 …
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

30. Distributed queries Query Result Control Compute Storage
Microsoft Build 2016
Distributed queries
2/5/ :04 AM
Query
Result
Control
Compute
Scale-out distributed query engine
Storage
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

31. Simple example SELECT COUNT_BIG(*) FROM dbo.[FactInternetSales] ;
Microsoft Build 2016
2/5/ :04 AM
Simple example
SELECT COUNT_BIG(*)
FROM dbo.[FactInternetSales] ;
SELECT SUM(*)
FROM dbo.[FactInternetSales] ;
Control
Compute
SELECT COUNT_BIG(*)
FROM dbo.[FactInternetSales] ;
SELECT COUNT_BIG(*)
FROM dbo.[FactInternetSales] ;
SELECT COUNT_BIG(*)
FROM dbo.[FactInternetSales] ;
SELECT COUNT_BIG(*)
FROM dbo.[FactInternetSales] ;
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

32. Demo: Azure Portal 2/5/2019 11:04 AM
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

33. 2/5/ :04 AM
Thank You
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.