Azure Data Factory v2: What’s new?

SQLSat Kyiv Team Denis Reznik Yevhen Nedashkivskyi Oksana Borysenko Eugene Polonichko Denis Reznik Mykola Pobyivovk Oksana Tkach

Sponsor Sessions Starts at 13:00 Don’t miss them, they might be providing some interesting and valuable information! Congress Hall DevArt Conference Hall Simplement Room AC DB Best Predslava1 Intapp NULL means no session in that room at that time 

Sponsors

Session will begin very soon :) Please complete the evaluation form from your pocket after the session. Your feedback will help us to improve future conferences and speakers will appreciate your feedback! Enjoy the conference!

About me Eugene Polonychko, Chapter Pass SQL Server User Group Over 6 years experience of software development, mostly focused on data. Have designed and implemented data warehouses using custom coding as well as with ETL tools. Experience developing front end applications, BI reporting and database administration. Have worked with MS SQL, MySQL and other databases. Social network: https://www.linkedin.com/in/eugenepolonichko/ https://twitter.com/EvgenPolonichko Hello. Thank you for joining me today. My name is Eugene Polonychko. I’m here today to talk to you about Azure Data Factory. But before I begin my presentation I want to tell you about myself. I’m from Ukraine. I’m DWH\BI architect and MCP. I have a more than 6-year experience of software development, mostly focused on BI. During my career I’ve designed and implemented BI solutions using Microsoft BI stack, Oracle and other technOlogies. You can connect with me on social media, the links to my twitter and LinkedIn accounts are in the slide. I’ve always been interested in ETL. That’s why I’m going to to talk about cloud ETL.

What are we going to talk about? What is Azure Data Factory? Concepts Dataset Pipeline Linked Services New in Data Factory v2 Trigger Control flow SSIS I’ve divided my talk into 4 sections. First, I’ll tell YOU about what ADF is Second, I’ll explain some important concepts of ADF including dataset, pipeline and linked services Third, we’ll look at the difference between SSIS and ADF Finally, I’ll describe monitoring for this technology. I’ll conclude with Q&A session and I’ll be glad to answer your questions at the end of the talk.

What is Azure Data Factory? Data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data. Okay, first, I’m going to give you an idea about what Azure Data Factory is . It’s a cloud data integration service . Why has Microsoft created it? Because we needed a tool which helps to import data from one cloud data source to another. So data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data. Azure Data Factory itself does not store any data. It lets you create data-driven flows to orchestrate movement of data between supported data stores and processing of data using computed services in other regions or in an on-premises environment.

What is Azure Data Factory? Look at this scheme.  Using Azure Data Factory, you can create and schedule data-driven workflows that can ingest data from disparate data stores, process/transform the data by using compute services such as Azure HDInsight Hadoop, Spark, Azure Data Lake Analytics, and Azure Machine Learning, and publish output data to data stores such as Azure SQL Data Warehouse for business intelligence (BI) applications to consume. Now when you have an idea what ADF is we can move on to the main concepts.

Concepts Data Source Dataset Pipeline is a grouping of logically related activities. It is used to group activities into a unit that performs a task Concepts Linked services computing environment Activities define the actions to perform on your data. Each activity takes zero or more datasets as inputs and produces one or more datasets as output. Activity We have four main concepts. First. Dataset is data source Second. Activity. It’s actions to perform on your data Third. Pipeline. It’s a grouping of logically related activities And finally. Linked services are computing environment or external resource. For example It’ s Hive, Machine Learning, stored procedure Let’s look at these concepts in more detail

Expressions & Parameters String functions – concat, substring, replace, indexof etc. Collection functions – length, union, first, last etc. Logic functions – equals, less than, greater than, and, or, not etc. Conversation functions – coalesce, xpath, array, int, string, json etc. Math functions – add, sub, div, mod, min, max etc. Date functions – utcnow, addminutes, addhours, format etc.

System variables Pipeline scope Trigger scope Variable Name @pipeline().DataFactory @pipeline().Pipeline @pipeline().RunId @pipeline().TriggerType @pipeline().TriggerId @pipeline().TriggerName @pipeline().TriggerTime Variable Name trigger().scheduledTime trigger().startTime

Development Create ADF Objects and Deploy to ADFv2 .net using .net PowerShell: Create ADF Objects and Deploy to ADFv2 Edit & PowerShell: Create ADF Objects per copy and paste and Deploy json artefacts using Powershel

Trigger

Type of triggers Manual execution Schedule trigger: A trigger that invokes a pipeline on a wall-clock schedule. Tumbling window trigger: A trigger that operates on a periodic interval, while also retaining state.

Schedule trigger A schedule trigger runs pipelines on a wall-clock schedule. This trigger supports periodic and advanced calendar options. For example, the trigger supports intervals like "weekly" or "Monday at 5:00 PM and Thursday at 9:00 PM."

Tumbling window trigger Tumbling window triggers are a type of trigger that fires at a periodic time interval from a specified start time, while retaining state. Tumbling windows are a series of fixed-sized, non-overlapping, and contiguous time intervals.

Control flow

Control flow Filter activity ForEach activity Execute Pipeline Get metadata If Condition activity Web activity Lookup activity Wait activity Until activity

Branching On success On failure On completion On skip

DEMO

SSIS

Integration Runtime Azure Self-hosted Azure-SSIS

Integration Runtime

Azure integration runtime An Azure integration runtime is capable of: Running copy activity between cloud data stores Dispatching the following transform activities in public network: HDInsight Hive activity, HDInsight Pig activity, HDInsight MapReduce activity, HDInsight Spark activity, HDInsight Streaming activity, Machine Learning Batch Execution activity, Machine Learning Update Resource activities, Stored Procedure activity, Data Lake Analytics U-SQL activity, .Net custom activity, Web activity, Lookup activity Get Metadata activity.

Self-hosted integration runtime An Azure integration runtime is capable of: Running copy activity between a cloud data stores and a data store in private network Dispatching the following transform activities against compute resources in On-Premise or Azure Virtual Network:

Azure-SSIS Integration Runtime To lift and shift existing SSIS workload, you can create an Azure-SSIS IR to natively execute SSIS packages.

DEMO

Do you have any questions?

Sponsors