Azure SQL Data Warehouse for Beginners David Alzamendi Azure SQL Data Warehouse for Beginners

Thanks to our Sponsors

Lunch Time Session from Sponsors

Raffle Prize Draw: Don’t miss the opportunity Enter Raffle Prize Draw for each sponsor Be at lecture Keynote room at 5pm $350 Flight Centre voucher Super Fast Dash - Hamptown Downs (worth $170) Two Vouchers for Power BI Training Videos (each worth $260 USD )

About Me

About Me I have been working as a Business Intelligence consultant the past few years. I have worked in all different areas of Business Intelligence solutions, providing high standard responses. Likewise, I have acquired international experience through living and working abroad in Argentina, Spain and Australia. Hobbies: Travel Cooking Sports @david_alzamendi www.linkedin.com/in/dalzamendi david@davidalzamendi.com

Get Involved “Sharing is Caring”

Today’s Session Introduction to ADW Architecture (Change of rules) Demo 1 (Provisioning) Pros Cons Demo 2 (Polybase) Best Practices Security

Introduction to Azure Data Warehouse Azure SQL Data Warehouse is a massively parallel processing (MPP) cloud-based, scale-out, relational database capable of processing large amounts of data.

Azure Data Warehouse

TIMELINE On-Premises Cloud 2008 2010 2011/2013 2014 2015 Microsoft Acquired DATAllegro, company that makes data warehouse appliances 2010 Microsoft Launched Fast Track Architecture Data Warehouse reference using SMP. DW best practices offered with leading H/W Partners. 2011/2013 Parallel Data Warehouse v1 and v2. Data Allegro product on Windows & SQL First DW appliance by Microsoft in partnership with Dell and HP 2014 Analytics Platform System (APS). Introduction of Hadoop region within appliance and new naming to reflect broader Big Data capabilities. Cloud 2015 Introduction of Azure SQL DW. Service based on APS’s MPP capabilities.

When to use Azure Data Warehouse You plan to move your DW to the cloud Tight deadlines for delivering a DW solution When you don’t want to invest millions of dollars Not enough skills for infrastructure, security, network, etc..PAAS! Store large amounts of data Integrate data from one or many sources in a single database Run complex queries or Ad-hoc reports against large amounts of data

When NOT to use Azure Data Warehouse OLTP Systems Row by row processing High number of single transactions Frequent reads and writes

VS Symmetric Multiprocessor System (SMP) Architecture Massively Parallel Processing (MPP) VS

Architecture Tables Data is stored in 60 distributions Round Robin: Distributes information randomly. Hash Distributed: Distributes data bases on hashing values from a single column Columnstore Index by default (10X compression + 100X query performance)

Load Data You can use load data using: Polybase Bcp Azure Data Factory SSIS

But what is the difference? Blob Storage Table Storage File Storage Queue Cosmos DB Azure DB Azure Data Lakes Azure DW

Azure Data Warehouse vs Azure Data Lakes Structured Data Structured, semi structured and non-structured data Processing It has shape (schema-on-write) No, shape (schema-on-read) Agility Less agile, fixed configuration Highly agile, configure and reconfigure as needed Security Mature Maturing Final Users Business professionals All users (data scientists)

Azure Data Warehouse vs Azure SQL Database Architecture Relational DB using MPP Relational DB using SMP System Online Analytical Processing (OLAP) Online Transaction Processing (OLTP) Storage Unlimited Up to 4 TB Polybase Yes No Concurrent Queries Up to 32 (not DMVs) Up to 30 000

Example

Provisioning ADW using: Azure Resource Manager Power Shell T-SQL Demo 1 Provisioning ADW using: Azure Resource Manager Power Shell T-SQL

Azure Data Warehouse PROS PaaS Azure Ecosystem Decouples storage from compute Scale Out Price Pause service Availability and Backups

PaaS

Azure Ecosystem More

Decouples storage from compute Grow or shrink storage size independent of compute. Grow or shrink compute power without moving data. Pause compute capacity while leaving data intact, only paying for storage.

Scale Out Deploy it in a few minutes From DW100 to DW6000 (DW9000, DW18000 * Public Preview) DW100 = 1 compute node DW6000 = 60 compute nodes DW 18000= 180 compute nodes Automatically Scale out using: T-SQL Rest API Power Shell

Price DWU (Data Warehouse Units) (DWUs are a measure of underlying resources like CPU, memory, and I/O bandwidth which are allocated to your SQL Data Warehouse. Increasing the number of DWUs increases resources and performance.) DWU Calculator to rescue! 1 compute node (100 DWU) x $1.21/hour (USD) 1TB x $0.17/hour = $122/month (USD) 200$ free credit in your first Azure Subscription 25$/12 months Visual Studio Dev Essentials

Pause the Service Compute and memory resources are returned to the pool of available resources in the data center DWU costs are zero for the duration of the pause. Data storage is not affected and your data stays intact. SQL Data Warehouse cancels all running or queued operations. Pause and Resume using Power Shell, ARM and REST APIs.

Availability and Backups Available in 30 regions (SE AUS Melbourne- SE ASIE Singapore) Locally Redundant Storage (LRD) for free Available Read Access Geo Redundant Storage (RA-GRS) Snapshot every 4 to 8 hours Retention for 7 days

Azure Data Warehouse CONS Maintain statistics Data Types (geography,geometry,hierarchyid,image,text,ntext,sql_variant,timestamp,xml) Primary key, Foreign keys, Unique and Check Table Constraints Unique Indexes, Computed columns, User-Defined Types, Global temporary tables, Synonyms, Indexed Views, Triggers, Sequences In Memory OLTP is not available

Load data using Polybase: Create Credentials Polybase 338 000 000 rows /DW1000 Demo 2 Load data using Polybase: Create Credentials Create External Data Source Load Data using CTAS

Best Practices Pause the service Scale when necessary Use SSAS on top of the ADW to increase concurrency (better together) Maintain Statistics Load the data using Polybase Hash large tables Not Over Partition Temporary Heap Tables to transform data Use Large Resource Class for Memory Consuming Transactions Use DMVS to monitor queries

Security TDE (Transparent Data Encryption) It doesn’t support: Always Encrypted. Grant permission to different users through Schemas and Roles

Wrap Up Easy to use Go and create your first DW in a few minutes PaaS DW Scale out and down as needed Low cost DW

Questions?

Thanks to our Sponsors

Don’t forget your feedback please! Thank You! Don’t forget your feedback please! Documentation