Azure Data Factory V2 Templates New Features Joseph Barth, Senior BI, DriveTime

Thank you to our Sponsors

Joseph Barth Senior BI, DriveTime SQL Server Enthusiasts Sports Guy 11 year SQL Server Veteran, Developer, ELT and ETL, some DBA work, query tuning and Data Architecture. Was Chapter Leader Arizona SQL Server Users Group, Co-Founder CASSUG in Albany 2010 Sports Guy Joseph Barth Played football 20 years, rugby 5 years, golf, basketball, triathlons, tough mudder, I like to stay active, hiking, fishing, hunting and camping. Senior BI, DriveTime Family Man /datamovementguru Married 10 years – blessed to have 5 year old named Mattigan, keeps me and my wife on our toes, also wife was a large part of our SQLSaturday’s in Phoenix, she knows PASS and understands what this is about. @datamovemntguru Joe Barth

Azure Data Factory v2 Using a brand new product in Preview and now in GA, we will step through a quick overview of the following concepts, to get you up to speed! Integration Runtimes Linked Services, Datasets, Pipelines Expressions, Parameters Azure DevOps Azure Key Vault Templates ( New Feature) Databricks Dataflows (New feature)

Integration Runtimes On Premise Sources Cloud Services Self Hosted Integration Runtime Cloud Services Azure Default(auto region) Azure Region Dependent Azure SSIS in the cloud Deploy your packages to Azure SQL DB SSIS and run from hosted scale set.

Understanding IR Concepts

Let’s talk speed, size and time

Linked Services, Datasets, Pipelines

Expressions , Parameters

Azure DevOps Azure DevOps– is Microsoft’s full development life cycle offering in the cloud. Manage your projects, agile, scrum, build and release, testing and version control. Today we integrate Azure DevOps git using ARM templates. Version control using Azure DevOps Git repository. ADFv2 you can configure your Azure Data Factory to using git repository to generate an ARM Template for deployment. ARM Template – Azure Resource Manager , using JSON structure to deploy objects to Azure. Azure Data Factory V2 allows us to using in conjunction with GIT to deploy changes from DEV, QA and to Productions using Parameters.

Azure Key Vault Linked Service Connection string Azure DevOps Release

Azure Date Factory Templates From Existing

Azure Databricks Dataflow

Databricks Dataflow Monitoring

ADFv2 The Good , The Bad , The Ugly New GUI online to create pipeline Not Intuitive GUI, you have to dig to find and understand Still have to do json structures for most activities, control in the GUI , know power shell Azure SSIS in the cloud – same deployment and packages Azure DevOps Git Continuous Integration Limitation on throughput is regulated by the system, express route is a must and also must be implemented for PaaS objects. Requires VNET and other setup to get to On-Prem sources for Azure SSIS

References page (pictures and graphics not presenters original work) https://www.datamovementguru.com/blog Data Movement Guru youtube -https://www.youtube.com/channel/UC4pLbBQfjnK06qHNGJ0KItg http://joshrichards.space/2016/12/16/space-the-good-the-bad-and-the-ugly-mars-one-update/ https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance https://docs.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime https://blogs.sentryone.com/melissacoates/getting-started-with-automated-arm-deployments-in-azure/ https://www.coffeesmurf.com/git-authentication-on-linux-with-visual-studio-team-services/ https://apphub.webex.com/categories/developer-tools/integrations/visual-studio-team-services-cisco-systems

Implementation Setup and Run!

Thank you to our Sponsors

Learn more from Joseph Barth @datamovemntguru joe@datamovementguru.com