Continuous Integration and Delivery (CI/CD) in Azure Data Factory Rayis Imayev SQL Saturday Providence, 2019 Aug 24th, 2019 Continuous Integration and Delivery (CI/CD) in Azure Data Factory

Thank you to our SQL Saturday #892 Sponsors

"Coming together is a beginning, staying together is progress, and working together is success.“ Henry Ford

Continuous Integration and Delivery (CI/CD) in Azure Data Factory Session plan: Introduction to the Azure Data Factory (ADF) Continuous Integration and Delivery (CI/CD) overview ADF pipeline case study Merging ADF with CI/CD ADF integration with a GitHub repository Azure DevOps review Creating integration of GitHub based ADF code and DevOps pipelines ADF pipeline amendments and affect on DevOps pipeline Conclusion

Introduction to the Azure Data Factory (ADF) Cloud platform to compose data-driven workflows with steps to move and transform data. Control Flow Pipeline Activity Datasets Linked Services Triggers Mapping Data Flows Source/Sink Transformation Tasks Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Continuous Integration and Delivery overview CI/CD process helps to establish a good software development practice and aims to build a healthy relationship between development, quality assurance, and other supporting teams. 1. Develop code 2. Unit test 3. Integrate 4. Acceptance test 5. Deploy to production Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

ADF pipeline case study (Demo) Sample ADF pipeline Copy all files across multiple containers from one file-based storage store to another. Activities: Get Metadata For Each Loop container Copy Data … more items Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Merging ADF with CI/CD Questions: Can multiple people work within the same ADF? Can I integrate my ADF changes into other people's ADF code? Can I have a process to frequently ship the ADF solution code to various environments? Can a continuous deployment be automated to a production environment? Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

ADF integration with a GitHub repository (Demo) Setting a GitHub Repository for your Azure Data Factory Saving your ADF code in a GitHub repository Adding changes to the existing ADF code Publishing changes to a live Data Factory and creating ARM templates http://datanrg.blogspot.com/2019/02/azure-data-factory-integration-with.html Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Azure DevOps review Plan, track, and discuss work across your teams. Collaborate on code development using free Git public and private repositories, pull requests, and code review. Azure Pipelines helps you implement a build, test, and deployment pipeline for any app. Using testing services for your apps. Code once and share packages across your organization Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Creating DevOps project to deploy ADF code (Demo) New DevOps project New Release Pipeline Variable groups for my deployment solution Adding Git artifact for the deployment Deployment stages and deployment steps within each stage Release pipeline variables Testing a deployment release http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

ADF code change and its affect in DevOps pipeline (Demo) Changing ADF code Merging the ADF code to your GitHub repository Publishing / Creaing ARM templates to deploy ADF code Monitoring ADF code deployment Validating code changes in other environments http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Continuous Integration and Delivery (CI/CD) in Azure Data Factory Session plan: Introduction to the Azure Data Factory (ADF) Continuous Integration and Delivery (CI/CD) overview ADF pipeline case study Merging ADF with CI/CD ADF integration with a GitHub repository Azure DevOps review Creating integration of GitHub based ADF code and DevOps pipelines ADF pipeline amendments and affect on DevOps pipeline Conclusion Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Conclusion: ADF code deployment could be automated with the help of Azure DevOps. Don’t forget to include Disable/Enable deployment steps if your ADF has triggers. Make sure to use Azure Key Vault to store your secrets and reuse them during deployment. Links: http://datanrg.blogspot.com/2019/02/continuous-integration-and-delivery.html https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment https://docs.microsoft.com/en-us/azure/devops/learn/what-is-devops Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

Thank you for attending my session! Rayis Imayev Toronto, Canada Consultant / Big Data Integration & Analytics Senior BI & ETL Developer at Mosaic North America http://datanrg.blogspot.com/ https://twitter.com/RayisImayev Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev