1. My Data Wandered Lonely As A Cloud: Azure Data Factory Julie Smith SQL Server MVP Innovative Architects @juliechix

2. My Data Flew amongst the Clouds quickly and without errors! 2

3. www.InnovativeArchitects.com 3 About Me Julie Smith IA Ambassador SQL Server MVP Julie.Smith@innovativearchitects. com @JulieChix Datachix.com

4. www.InnovativeArchitects.com 4 INNOVATIVE ARCHITECTS #WorkSomeplaceAwesome

5. Foundation

6. Intro: What is Azure Data Factory? Azure Data Factory is a cloud service that orchestrates, manages, and monitors the integration and transformation of structured and unstructured data from on-premises and cloud sources at scale.

7. Data Integration in the Cloud Between other cloud services and On Prem Sources, Destinations, Transformations Like DTS Like SSIS 7

8. Where Portal.azure.com New>Data+Analytics>Data Factory

9. Three Main Elements Linked Services Datasets Pipeline Activities 9

10. Linked Services 10 Data Stores Data Gateways

11. Data Gateway for On Prem 11

12. Data Gateways & ADF 12 Supplies key Install Gateway on each On Prem resource (server, laptop, etc)

13. Data Gateways & ADF 13

14. Data Gateways & ADF 14

15. Data Stores Contain credentials and connection information for Sources and Destinations. An On Prem Data Store MUST reference a Data Gateway 15

16. After you set up the gateway, set up your linked services. They have to have a gateway if they are going to an on prem source or destination.

17. After you set up the gateway, set up your linked services. When you pick SQL Server (on prem). You HAVE to have a gateway:

18. Azure Data Stores Don’t Require Gateway

19. Types of connections, in context: 19

20. Once you have Linked Services Datasets (tables) Pipelines Activities 20

21. Author and Deploy 21

22. Author and Deploy 22

23. Diagram 23

24. Datasets (Tables) 24

25. JSON pronounced Jay-Sahn JavaScript Object Notation http://json.org/

26. JSON JSON is built on two structures: A collection of name/value pairs. In various languages, this is realized as an object, record, struct, dictionary, hash table, keyed list, or associative array. { } An ordered list of values. In most languages, this is realized as an array, vector, list, or sequence. [ ] JavaScript Object Notation http://json.org

27. JSON in ADF { "name": "OnPremActorSrce", "properties": { "published": false, "type": "SqlServerTable", "linkedServiceName": "NorthWindStg", "typeProperties": { "tableName": "Actor" }, "availability": { "frequency": "Day", "interval": 1 }, "policy": { "externalData": { "retryInterval": "00:01:00", "retryTimeout": "00:10:00", "maximumRetry": 3 }

28. JSON specific to ADF https://msdn.microsoft.com/en- us/library/azure/dn835050.aspx

29. Pipelines Activities Copy 29

30. Weird Things One Data Factory. So your diagram gets messy. Goes against SSIS best practices of one package per destination. Scheduling is clumsy. Pipeline and destination have to be in sync in their availability.

31. One Downloadable Demo Points

32. Within the pipeline: Activities

33. Slices Each unit of data consumed and produced by an activity run is called a data slice. They have StartTime and EndTime and those are accessible to the pipeline activity via ADF System Variables: "sqlReaderQuery": "$$Text.Format('select * from MyTable where timestampcolumn >= \\'{0:yyyy-MM-dd HH:mm}\\' AND timestampcolumn < \\'{1:yyyy-MM-dd HH:mm}\\'', WindowStart, WindowEnd)"

34. Scripting Reference https://msdn.microsoft.com/en-us/library/azure/dn835050.aspx As of July 16 th BEWARE

35. Visual Studio Extension Azure SDK 2.7 and above for Visual Studio 2013 You get templates You can reverse engineer You can connect to your factory and deploy from VS Came out JULY 22, 2015

36. Visual Studio Extension

37. Customer Case Studies https://azure.microsoft.com/en-us/documentation/articles/data- factory-customer-case-studies/ https://azure.microsoft.com/en-us/documentation/articles/data- factory-customer-case-studies/

38. WHY

39. Data Management Gateway Configuration Manager http://www.microsoft.com/en-us/download/details.aspx?id=39717 Instructions on use: https://azure.microsoft.com/en- us/documentation/articles/data-factory-move-data-between-onprem-and- cloud/#using-the-data-gateway-step-by-step-walkthroughhttps://azure.microsoft.com/en- us/documentation/articles/data-factory-move-data-between-onprem-and- cloud/#using-the-data-gateway-step-by-step-walkthrough For on prem machines. Load the Gateway on the machine. Then go to the Azure Data Factory. Create the Linked Service Gateway there. Get the key from the ADF linked service, copy and paste it into the final step of the Gateway setup on the On Prem Machine. The Gateway is for the entire server. The entire machine. The Linked service will use that gateway for other things and must be configured for each service i.e. Sql databases. Be patient. Refresh rate is slow and can make it seem like it didn’t work when it did.

40. Data Management Gateway Configuration Manager http://www.microsoft.com/en-us/download/details.aspx?id=39717 Instructions on use: https://azure.microsoft.com/en- us/documentation/articles/data-factory-move-data-between- onprem-and-cloud/#using-the-data-gateway-step-by-step- walkthroughhttps://azure.microsoft.com/en- us/documentation/articles/data-factory-move-data-between- onprem-and-cloud/#using-the-data-gateway-step-by-step- walkthrough For dev purposes, for you own machine. Use Express Set up. It will take about 10 minutes, but it works. You’ll have the data management on your laptop bam.

41. Learning Path https://azure.microsoft.com/en-us/documentation/learning- paths/data-factory/https://azure.microsoft.com/en-us/documentation/learning- paths/data-factory/

42. Webinar Wee Hyong Tok’s webcast

43. Resources Reza Rad’s blog