Where Should My Data Live (and Why)? Matt Gordon, Data Platform Solution Architect, DMI

Data Platform Solution Architect, DMI 15+ Years SQL Server Experience I began as a database developer, which evolved into a DBA role, then datacenter management, and now consulting. I’ve Managed Data and People I’ve managed servers, datacenters, and development teams in my career so I’ve approached these issues from many angles. Matt Gordon Data Platform Solution Architect, DMI I Drive Race Cars The picture to the left is of me and my car in Gasoline Alley at Indianapolis Motor Speedway this past June. I will talk your ear off about this if given the opportunity! /in/sqlatspeed @sqlatspeed

Agenda Where Does Our Data Live Now? Why Does Our Data Live Where It Does? Cloud, On-Premises, or Both? Case Studies Wrap-up

Definitions

Discussion Points Cloud is just somebody else’s computer in somebody else’s datacenter Rapid development from cloud providers constantly expands options Are you locked into deployment locations for certain platforms? Database engine always on-premises Hadoop always in cloud Blending of technologies and platforms may/may not be the right answer

Where Does Our Data Live Now?

Where Does Our Data Live Now? How many of us do not have a single data environment in the cloud? How many of us have only test/dev/QA data environments in the cloud? How many of us have a “trial” production data environment in the cloud? How many of us have all production data environments in the cloud? How many of us have all (or nearly all) data environments in the cloud?

Why Does Our Data Live Where It Does?

Why Does Our Data Live Where It Does? On-Premises Pros Cost Leveraging “investments” Can cost less if uptime is not critical Comfort level “I can go see it” Physical control and security Data accessible even when all external telecom is down Licensing

Why Does Our Data Live Where It Does? On-Premises Cons Generally requires large up-front investment Requires corresponding infrastructure Rackspace, cooling, cabling, telecom, fire suppression, etc. May require backup datacenter Depends on uptime requirements On-site personnel often needed to maintain operations More expensive from a resource perspective

Why Does Our Data Live Where It Does? Cloud Pros Cost Buy only what you need Scalability (vertical and horizontal) Global redundancy Storage durability Data availability from all locations PaaS often satisfactory to government security audits/approvals High availability and disaster recovery often built-in*

Why Does Our Data Live Where It Does? Cloud Cons Can require robust Internet connectivity VPN cost can be significant Minimal to no control over underlying infrastructure “Noisy neighbors” Design apps to deal with connection hiccups more efficiently Perception of lighter security “Things happen by magic”

Cloud, On-Premises, or Both?

Cloud Deployment Options (Azure) SQL Server (IaaS) Azure SQL DW Azure SQL DB Hadoop Mimics on-premises behavior but resources are on Azure Full control of configuration Full control of maintenance MPP cloud-based, scale-out, relational database Separates storage and compute Can pause compute capacity when not needed PaaS flavor of SQL Server database Very limited control of maintenance Limited control of configuration Microsoft’s flavor is known as HDInsight Used for semi-structured data Can connect from database engine using PolyBase

Cloud Deployment Options (Amazon) SQL Server on EC2 Amazon Redshift Amazon RDS Hadoop Mimics on-premises behavior but resources are on Amazon EC2 Full control of configuration Full control of maintenance Amazon equivalent of Azure SQL DW Fully managed Easily scalable Amazon PaaS offering Supports six database engines Minimal configuration control Amazon’s HDInsight equivalent is EMR Supports traditional Hadoop tooling Can connect from database engine using PolyBase

Cloud Deployment Options (Google) Google Compute Engine DW? Google Cloud SQL Hadoop SQL Server on Google Cloud Platform is IaaS offering Full control of configuration Full control of maintenance Multiple versions and editions supported No real Google equivalent in this space yet PaaS flavor of database engines Supports MySQL and PostgreSQL (beta) Fully-managed Google’s fully-managed flavor is known as Google Cloud Dataproc Used for semi-structured data Can connect from database engine using PolyBase

On-Premises Deployment Options SQL Server Microsoft APS PaaS Database Hadoop Traditional deployment of the database engine Full control of configuration Full control of maintenance MPP appliance Evolution of Parallel Data Warehouse Architecture of Azure SQL DW based on this design No on-premises equivalent of Azure SQL Database Microsoft’s flavor is known as HDInsight Many other non-Microsoft deployment options Can connect from database engine using PolyBase

Hybrid Deployment Options/Scenarios On-Premises App Servers & Azure SQL DB Availability Groups with Azure Replica(s) Easy to create and destroy databases as needed for development and deployment Removes management responsibility from devs Good choice if DBA team short on resources Uses Azure as backup datacenter(s) Requires robust network infrastructure Good for minimum datacenter proximity requirements

Hybrid Deployment Options/Scenarios Replication to Azure IaaS VM Replication to Azure SQL Database Tried and true technology in use Identical to doing this on-premises other than network portion Good way to ease into comfort with the cloud Azure SQL Database can be a replication subscriber Eases DBA team into cloud and PaaS interactions Straightforward setup

Setting up replication to Azure SQL Database Demo Setting up replication to Azure SQL Database

Hybrid Deployment Options/Scenarios Log Shipping to Azure IaaS VM PolyBase to Azure Blob Storage Popular with customers who want a copy of data stored completely off-site Straightforward setup Expands environment without requiring cluster or other complication infrastructure Great for querying large quantities of semi-structured data Good way to introduce team to PolyBase Subject of our first case study

Case Studies

Transportation Planning Agency Statistical models generating TBs of output every year Storage costs spiraling upward and difficult to manage Output stored in relational database tables requiring constant maintenance Output generated as text files which were fed into the relational tables Loaded output files into Azure Blob Storage (cold) Query performance increased Storage costs decreased by 96% ($2k per year vs. $75k per year)

Querying statistical model outputs stored in Azure Blob Storage Demo Querying statistical model outputs stored in Azure Blob Storage

Geospatial Research Center Hosted Hadoop cluster Hosted HDFS storage storing Excel, CSV, XML, JSON, etc. SQL Server installed on Azure VMs Database engine, DQS, MDS, and SSAS in use PolyBase used to query semi-structured data from main SQL Server databases Data consumers presented with common interface to access heterogeneous data

Wrap-up

Discussion Points Cloud is just somebody else’s computer in somebody else’s datacenter Rapid development from cloud providers constantly expands options Are you locked into deployment locations for certain platforms? Database engine always on-premises Hadoop always in cloud Blending of technologies and platforms may/may not be the right answer

Recommendations Set expectations what cloud technologies are and what they can do Management Team HA/DR isn’t done by magic – it’s just different Stay abreast of new technologies Research Training Azure Stack Embrace it all!

Learn more from Matt Gordon @sqlatspeed mgordon@dminc.com