Polybase and Time Travel (Temporal Tables) Stan Geiger #492 | Phoenix 2016

Platinum Level Sponsors Gold Level Sponsors Pre Conference Sponsor Venue Sponsor Key Note Sponsor

Silver Level Sponsors Bronze Level Sponsors

About Me Sr. Product Manager with Idera Performance Monitoring of Microsoft BI stack Backup and Recovery of Microsoft SQL Server Geek Sync Presenter Blog Contributor HSSUG presenter Over 25 years experience BI, Data Architect DBA.Net Developer Data Analyst

Where in the world are we? 5 … data warehousing has reached the most significant tipping point since its inception. The biggest, possibly most elaborate data management system in IT is changing. – Gartner, “The State of Data Warehousing in 2012” Data sources ETL Data warehouse BI and analytics

The Cool Kid’s Data Warehouse 6

The Data Warehouse of the Future? Diverse Big Data Workload Centric Approach Data stored on multiple platforms Physically distributed data warehouse data warehouse appliances columnar RDBMSs NoSQL databases MapReduce tools, and HDFS. 7

The Data Warehouse of the Future…Its Here! 8

SQL Server Technology Drivers PolyBase JSON Data Temporal Tables In Memory Table ColumnStore Index

PolyBase 10

PolyBase Use T-SQL to store data in SQL Server from Hadoop or Azure as tables. Knowledge of Hadoop or Azure is not required to use. Pushes computation to where data resides Export relational data into Hadoop or Azure 11

PolyBase - External Tables, Data Sources & File Formats 12 SQL Server w/ PolyBase Social Apps Sensor &RFID Mobile Apps Web Apps Data Scientists, BI Users, DB Admins Your Apps PowerPivot PowerView PolyBase Split-Based Query Processing External Table External Data Source External File Format Hadoop Relational DW

PolyBase Scenarios Querying Run T-SQL over HDFS Combine data from different Hadoop clusters Join relational with non-relational data ETL Subset of Hadoop in Columnar Format Enable data aging scenarios to more economic storage Allows building of multi-temperate DW platforms SQL Server acts as hot query engine processing most recent data sets Aged data immediately accessible via external tables No need to groom data Hybrid (Azure Integration) Mesh-up on-premise and cloud apps Bridge between on-premise and Azure 13

PolyBase 1.Create external data source (Hadoop). 2.Create external file format (delimited text file). 3.Create external table pointing to file stored in Hadoop. 14 CREATE EXTERNAL TABLE [dbo].[CarSensor_Data] ( [SensorKey] int NOT NULL, [CustomerKey] int NOT NULL, [GeographyKey] int NULL, [Speed] float NOT NULL, [YearMeasured] int NOT NULL ) WITH (LOCATION='/Demo/car_sensordata.tbl', DATA_SOURCE = hdp2, FILE_FORMAT = ff2, REJECT_TYPE = VALUE, REJECT_VALUE = 0 CREATE EXTERNAL DATA SOURCE hdp2 with ( TYPE = HADOOP, LOCATION ='hdfs://10.xxx.xx.xxx:xxxx', RESOURCE_MANAGER_LOCATION='10.xxx.xx.xxx:xxxx') CREATE EXTERNAL FILE FORMAT ff2 WITH ( FORMAT_TYPE = DELIMITEDTEXT, FORMAT_OPTIONS (FIELD_TERMINATOR ='|', USE_TYPE_DEFAULT = TRUE)

PolyBase - Ad-Hoc Query joining relational with Hadoop data Who drives faster than 35 Miles > joining structured customer data stored in SQL Server with sensor data 15 SELECT DISTINCT Insured_Customers.FirstName, Insured_Customers.LastName, Insured_Customers.YearlyIncome, Insured_Customers.MaritalStatus into Fast_Customers from Insured_Customers INNER JOIN ( select * from CarSensor_Data where Speed > 35 ) as SensorD ON Insured_Customers.CustomerKey = SensorD.CustomerKey ORDER BY YearlyIncome CREATE CLUSTERED COLUMNSTORE INDEX CCI_FastCustomers ON Fast_Customers;

CHANGE THE CONFIG FILE!!!!!!!!!!!!!!!!!! 16 The user that is used to access Hadoop has to be configured. This block will need to be inserted into the DWEngineService.exe.config file in the applicationSettings block which should already exist. Anything besides PDW_User

Demo

Temporal Tables Temporal Table is really two tables. Data Table Historical Table (PERIOD) A temporal table can be defined as a table for which PERIOD definition exists comprising of system columns Slowly Changing Dimension Data Table is Type 1 Historical Table is Type 2 Recover accidental data changes 18

TEMPORAL TABLES  Slowly Changing Dimensions Type 1 Records will be updated with no history log Type 2 Keep history with a From Date and To Date Set existing record as obsolete (with setting the To Date) Create a new record (with new From Date) Type 3 Limited History Type 4 Similar to Type 2 but broken out as 2 tables.

Temporal Tables Requirements/Limitations Primary Key Two columns (start and end date as datetime2) In-Memory tables cannot be used INSERT and UPDATE not allowed on SYSTEM_TIME period columns History Table data cannot be changed. Regular queries only affect data in the current table. 20

Temporal Tables Example: 21 CREATE TABLE dbo.TestTemporal (ID int primary key, A int, B int, C AS A*B, SysStartTime datetime2 GENERATED ALWAYS AS ROW START NOT NULL, SysEndTime datetime2 GENERATED ALWAYS AS ROW END NOT NULL, PERIOD FOR SYSTEM_TIME (SysStartTime, SysEndTime) ) WITH (SYSTEM_VERSIONING = ON);

TEMPORAL TABLES  Creating through SSMS 22

Temporal Tables The SELECT statement FROM claus e has a new clause FOR SYSTEM_TIME with four temporal-specific sub-clauses to query data across the current and history tables. Point in time: AS OF Exclusive bounds: FROM TO Inclusive lower bound, exclusive upper bound: BETWEEN AND Inclusive bounds: CONTAINED IN (, ) 23

Temporal Tables For example, if you want to look at the values active for customer 27 on the first of the year: … FROM Customer FOR SYSTEM_TIME AS OF ' ' WHERE CustomerID = 27 If instead you want to see every version of the users records for that day you could write: … FROM Customer FOR SYSTEM_TIME BETWEEN ' ' AND ' 'WHERE CustomerID = 27 24

Demo

Try any of our tools for free!