Orchestration and data movement with Azure Data Factory v2 Simon Peck Orchestration and data movement with Azure Data Factory v2 Welcome

Thanks to our sponsors Gold Sponsors Bronze Sponsors

About me Data Architect – Data Engineers Ltd 20+ years working with data. Varigence certified Biml expert Varigence consulting partner BimlFlex data warehouse implementer Co-author “The Biml Book” Ara 28 years ago database programming SQL Server 20 years

Agenda ADF v2 Introduction Demo Q & A Entities Coming from SSIS Integrated Runtime Demo API data to Azure SQL Automation with Biml Q & A Poll

What is Azure Data Factory? ADF is a cloud-based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation It is a Platform-as-a-Service offering in Azure that was first released in 2015 v1. v2 announced at PASS Summit 2017 and is still in public preview

Quick compare SSIS SSIS (ETL) ADF v2 (ELT) Connection Manager Linked Service Source / Destination Adapters Dataset Package Pipeline Tasks Activity

Entities Adapters Task Package Connection Manager Activity – USQL, Hive, Stored Proc, Copy Activity Consumes and produces Data sets which represents data items stored in Linked Service. Pipleline is a collection or logical grouping of Activities Connection Manager

Entities Linked Services Datasets Pipeline Activity Linked services are much like connection strings, which define the connection information needed for Data Factory to connect to external resources. Referenced by datasets Datasets Datasets identify data within different data stores, such as tables, files, folders, documents and endpoints. These reference the data you want to use in your activities as inputs and outputs Pipeline Activity – USQL, Hive, Stored Proc, Copy Activity Consumes and produces Data sets which represents data items stored in Linked Service. Pipleline is a collection or logical grouping of Activities A pipeline is a logical grouping (container) of activities that together perform a task (workflow) Activity The activities in a pipeline define actions to perform on your data. Can have constraints and dependencies between activities (like SSIS)

Quick compare SSIS SSIS (ETL) ADF v2 (ELT) Connection Manager Linked Service Source / Destination Adapters Dataset Package Pipeline Tasks Activity

Integration Runtime (IR) The Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory to provide the following data integration capabilities across different network environments: IR type Public network Private network Azure Data movement Activity dispatch   Self-hosted Azure-SSIS SSIS package execution The Integration Runtime or IR is the compute infrastructure used by ADF V2, it determines where your activity runs on, or gets dispatched from. There are three IR types: During the demo we’ll look at Azure and Azure-SSIS

Demo IOS field app used by farm managers Data is locked down in private cloud IOS field app has limited reporting capability and expensive to change IOS field app syncs with cloud database via API calls Leverage ADF v2 + ADF v2 SSIS Integrated Runtime Extract data to Azure SQL DB for Power BI and Excel analysis, reports and dashboards Part of a greater precision agriculture project Client talk about cloud so it’s time to start. Really good data for machine learning and data science experiements.

Demo – Agriculture Field App We want to land the XML files in blob storage or data lake for reuse for other over arching projects We need a linked service to the HTTP endpoint, Blob Storage and Azure SQL DB We need datasets to describe

Automation with Biml 50+ Weather Stations 5 Years Data Every 6 Minutes 127,000 Copy Activities 30 Million Weather Observations with up to 10 data points per observation Add something here about Varigences partnership with Microsoft and creating first class ADF model into the Biml Engine.

12/2/2018 12:56 PM Biml Basics Biml is a XML dialect to describe BI objects Just plain XML text Used for Tables, Views, SSIS, SSAS (both), ADF Cut to demo 1. Cut back after metadata Not particularly exciting. Demo, add 2 ingredients (Biml Script and metadata) © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Biml Script is where the magic lives 12/2/2018 12:56 PM Biml Script is where the magic lives Loop 1 Loop 2 Biml is a XML dialect to describe BI objects Just plain XML text Used for Tables, Views, SSIS, SSAS (both), ADF Cut to demo 1. Cut back after metadata Not particularly exciting. Demo, add 2 ingredients (Biml Script and metadata) Loop 3 © 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

simon@dataengineeers. co. nz @biguynz https://nz. linkedin

Thank for attending South Island SQLSaturday#!