1. Orchestration and data movement with Azure Data Factory v2
Simon Peck
Orchestration and data movement with Azure Data Factory v2
Welcome

2. Thanks to our sponsors
Gold Sponsors
Bronze Sponsors

3. About me Data Architect – Data Engineers Ltd
20+ years working with data.
Varigence certified Biml expert
Varigence consulting partner
BimlFlex data warehouse implementer
Co-author “The Biml Book”
Ara 28 years ago database programming
SQL Server 20 years

4. Agenda ADF v2 Introduction Demo Q & A Entities Coming from SSIS
Integrated Runtime
Demo
API data to Azure SQL
Automation with Biml
Q & A
Poll

5. What is Azure Data Factory?
ADF is a cloud-based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation
It is a Platform-as-a-Service offering in Azure that was first released in 2015 v1. v2 announced at PASS Summit 2017 and is still in public preview

6. Quick compare SSIS SSIS (ETL) ADF v2 (ELT) Connection Manager
Linked Service
Source / Destination Adapters
Dataset
Package
Pipeline
Tasks
Activity

7. Entities Adapters Task Package Connection Manager
Activity – USQL, Hive, Stored Proc, Copy
Activity Consumes and produces Data sets which represents data items stored in Linked Service.
Pipleline is a collection or logical grouping of Activities
Connection Manager

8. Entities Linked Services Datasets Pipeline Activity
Linked services are much like connection strings, which define the connection information needed for Data Factory to connect to external resources. Referenced by datasets
Datasets
Datasets identify data within different data stores, such as tables, files, folders, documents and endpoints. These reference the data you want to use in your activities as inputs and outputs
Pipeline
Activity – USQL, Hive, Stored Proc, Copy
Activity Consumes and produces Data sets which represents data items stored in Linked Service.
Pipleline is a collection or logical grouping of Activities
A pipeline is a logical grouping (container) of activities that together perform a task (workflow)
Activity
The activities in a pipeline define actions to perform on your data. Can have constraints and dependencies between activities (like SSIS)

9. Quick compare SSIS SSIS (ETL) ADF v2 (ELT) Connection Manager
Linked Service
Source / Destination Adapters
Dataset
Package
Pipeline
Tasks
Activity

10. Integration Runtime (IR)
The Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory to provide the following data integration capabilities across different network environments:
IR type
Public network
Private network
Azure
Data movement Activity dispatch
Self-hosted
Azure-SSIS
SSIS package execution
The Integration Runtime or IR is the compute infrastructure used by ADF V2, it determines where your activity runs on, or gets dispatched from. There are three IR types:
During the demo we’ll look at Azure and Azure-SSIS

11. Demo IOS field app used by farm managers
Data is locked down in private cloud
IOS field app has limited reporting capability and expensive to change
IOS field app syncs with cloud database via API calls
Leverage ADF v2 + ADF v2 SSIS Integrated Runtime
Extract data to Azure SQL DB for Power BI and Excel analysis, reports and dashboards
Part of a greater precision agriculture project
Client talk about cloud so it’s time to start.
Really good data for machine learning and data science experiements.

12. Demo – Agriculture Field App
We want to land the XML files in blob storage or data lake for reuse for other over arching projects
We need a linked service to the HTTP endpoint, Blob Storage and Azure SQL DB
We need datasets to describe

13. Automation with Biml 50+ Weather Stations 5 Years Data Every 6 Minutes
127,000 Copy Activities
30 Million Weather Observations with up to 10 data points per observation
Add something here about Varigences partnership with Microsoft and creating first class ADF model into the Biml Engine.

14. 12/2/ :56 PM
Biml Basics
Biml is a XML dialect to describe BI objects
Just plain XML text
Used for Tables, Views, SSIS, SSAS (both), ADF
Cut to demo 1. Cut back after metadata
Not particularly exciting. Demo, add 2 ingredients (Biml Script and metadata)
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Biml Script is where the magic lives
12/2/ :56 PM
Biml Script is where the magic lives
Loop 1
Loop 2
Biml is a XML dialect to describe BI objects
Just plain XML text
Used for Tables, Views, SSIS, SSAS (both), ADF
Cut to demo 1. Cut back after metadata
Not particularly exciting. Demo, add 2 ingredients (Biml Script and metadata)
Loop 3
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. simon@dataengineeers. co. nz @biguynz https://nz. linkedin

17. Thank for attending South Island SQLSaturday#!