1. Orchestration and data movement with Azure Data Factory v2
Simon Peck
Orchestration and data movement with Azure Data Factory v2

2. Simon Peck – Data Engineers
Data Architect – Data Engineers Ltd
28 years working with data mainly BI/DW
Varigence certified Biml expert (BimlHero)
Microsoft MPP (Big Data), MCSE (Data/BI)
BimlFlex DW automation implementer
Co-author “The Biml Book”

3. Agenda ADF v2 Introduction Demo Q & A if time Overview Entities
Coming from SSIS
Integrated Runtime
Demo
API data to Azure SQL
Automation fun with Biml and PowerShell
Q & A if time

4. Overview ADF v2
A fully managed data integration service for big data analytics in Azure
Batch ingest data at scale
Orchestrate and schedule
Monitor and manage

5. SSIS – ADF v2 Broad Comparison
SSIS (ETL)
ADF v2 (ELT)
Connection Manager
Linked Service
Source / Destination Adapters
Dataset
Package
Pipeline
Tasks
Activity

6. Entities
Adapters
Task
Package
Connection Manager

7. Entities Linked Services Datasets Pipeline Activity
Linked services are much like connection strings, which define the connection information needed for Data Factory to connect to external resources. Referenced by datasets
Datasets
Datasets identify data within different data stores, such as tables, files, folders, documents and endpoints. These reference the data you want to use in your activities as inputs and outputs
Pipeline
A pipeline is a logical grouping (container) of activities that together perform a task (workflow)
Activity
The activities in a pipeline define actions to perform on your data. Can have constraints and dependencies between activities (like SSIS)

8. Integration Runtime IR type Public network Private network Azure
The Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory to provide the following data integration capabilities across different network environments:
IR type
Public network
Private network
Azure
Data movement Activity dispatch
Self-hosted
Azure-SSIS
SSIS package execution

9. Walkthrough

10. Solution Overview Process Flow Start $$I$ IR Get Data Wait Load Data
Stop $$I$ IR
Find HWM
Form URL
HTTP Source
Blob Store Sink
Webhook
Azure Automation
For SSIS IR to start
(sproc)
Execute
SSIS
Package(s)
Webhook
Azure Automation

11. Solution Overview
Activities
Master Pipeline

12. Solution Overview
Extract Pipeline

13. Solution Overview
Extract Pipeline

14. Demo Automated development and deployment with Biml and PowerShell
Metadata driven automation using Biml
Automated

15. Demo
Automated development and deployment with Biml and PowerShell

16. simon@dataengineeers. co. nz @biguynz https://nz. linkedin

17. Thanks to all sponsors