1. Continuous Integration and Delivery (CI/CD) in Azure Data Factory
Rayis Imayev
SQL Saturday Providence, 2019
Aug 24th, 2019
Continuous Integration and Delivery (CI/CD) in Azure Data Factory

2. Thank you to our SQL Saturday #892 Sponsors

3. "Coming together is a beginning, staying together is progress, and working together is success.“ Henry Ford

4. Continuous Integration and Delivery (CI/CD) in Azure Data Factory
Session plan:
Introduction to the Azure Data Factory (ADF)
Continuous Integration and Delivery (CI/CD) overview
ADF pipeline case study
Merging ADF with CI/CD
ADF integration with a GitHub repository
Azure DevOps review
Creating integration of GitHub based ADF code and DevOps pipelines
ADF pipeline amendments and affect on DevOps pipeline
Conclusion

5. Introduction to the Azure Data Factory (ADF)
Cloud platform to compose data-driven workflows with steps to move and transform data.
Control Flow
Pipeline
Activity
Datasets
Linked Services
Triggers
Mapping Data Flows
Source/Sink
Transformation Tasks
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

6. Continuous Integration and Delivery overview
CI/CD process helps to establish a good software development practice and aims to build a healthy relationship between development, quality assurance, and other supporting teams.
1. Develop code
2. Unit test
3. Integrate
4. Acceptance test
5. Deploy to production
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

7. ADF pipeline case study (Demo)
Sample ADF pipeline
Copy all files across multiple containers from one file-based storage store to another.
Activities:
Get Metadata
For Each Loop container
Copy Data
… more items
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

8. Merging ADF with CI/CD Questions:
Can multiple people work within the same ADF?
Can I integrate my ADF changes into other people's ADF code?
Can I have a process to frequently ship the ADF solution code to various environments?
Can a continuous deployment be automated to a production environment?
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

9. ADF integration with a GitHub repository (Demo)
Setting a GitHub Repository for your Azure Data Factory
Saving your ADF code in a GitHub repository
Adding changes to the existing ADF code
Publishing changes to a live Data Factory and creating ARM templates
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

10. Azure DevOps review Plan, track, and discuss work across your teams.
Collaborate on code development using free Git public and private repositories, pull requests, and code review.
Azure Pipelines helps you implement a build, test, and deployment pipeline for any app.
Using testing services for your apps.
Code once and share packages across your organization
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

11. Creating DevOps project to deploy ADF code (Demo)
New DevOps project
New Release Pipeline
Variable groups for my deployment solution
Adding Git artifact for the deployment
Deployment stages and deployment steps within each stage
Release pipeline variables
Testing a deployment release
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

12. ADF code change and its affect in DevOps pipeline (Demo)
Changing ADF code
Merging the ADF code to your GitHub repository
Publishing / Creaing ARM templates to deploy ADF code
Monitoring ADF code deployment
Validating code changes in other environments
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

13. Continuous Integration and Delivery (CI/CD) in Azure Data Factory
Session plan:
Introduction to the Azure Data Factory (ADF)
Continuous Integration and Delivery (CI/CD) overview
ADF pipeline case study
Merging ADF with CI/CD
ADF integration with a GitHub repository
Azure DevOps review
Creating integration of GitHub based ADF code and DevOps pipelines
ADF pipeline amendments and affect on DevOps pipeline
Conclusion
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

14. Conclusion:
ADF code deployment could be automated with the help of Azure DevOps.
Don’t forget to include Disable/Enable deployment steps if your ADF has triggers.
Make sure to use Azure Key Vault to store your secrets and reuse them during deployment.
Links:
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev

15. Thank you for attending my session!
Rayis Imayev Toronto, Canada
Consultant / Big Data Integration & Analytics
Senior BI & ETL Developer at Mosaic North America
Continuous Integration and Delivery (CI/CD) in Azure Data Factory - Rayis Imayev