1. Data Platform and Analytics Foundational Training
Microsoft C+E Technology Training
Data Platform and Analytics
Foundational Training
Solution Area
Data Analytics
Solution
Advanced Analytics
Technology
Data Factory
[Speaker Notes]

2. Azure Data Factory Service A data integration service in the cloud
TechReady 23
5/13/ :27 PM
Azure Data Factory Service A data integration service in the cloud
Data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Azure Data Factory Service A data integration service in the cloud
TechReady 23
5/13/ :27 PM
Azure Data Factory Service A data integration service in the cloud
Data Factory service allows you to create data pipelines that move and transform data, and then run the pipelines on a specified schedule (hourly, daily, weekly, etc.).
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Azure Data Factory Service Pipelines and Activities
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Pipelines and Activities
In a Data Factory solution, you create one or more data pipelines
Activities define the actions to perform on your data
Data movement
Data transformation
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Azure Data Factory Service Activities » Data Movement
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Activities » Data Movement
The Copy Activity in Data Factory copies data from a source data store to a sink data store
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Azure Data Factory Service Activities » Data Movement » Azure
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Activities » Data Movement » Azure
Data Store
Supported as Source
Supported as Sink
Azure Blob storage  
Azure Data Lake Store 
Azure SQL Database 
Azure SQL Data Warehouse 
Azure Table storage 
Azure DocumentDB 
Azure Search Index 
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Azure Data Factory Service Activities » Data Movement » Databases
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Activities » Data Movement » Databases
Data Store
Supported as Source
Supported as Sink
SQL Server * 
Oracle *
MySQL * 
DB2 *
Teradata * 
PostgreSQL * 
Sybase * 
Cassandra * 
MongoDB *
Amazon Redshift
* Data store can be on-premises or on Azure IaaS, and require you to install Data Management Gateway
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Azure Data Factory Service Activities » Data Movement » File
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Activities » Data Movement » File
Data Store
Supported as Source
Supported as Sink
File System *  
HDFS *  
Amazon S3 
FTP
* Data store can be on-premises or on Azure IaaS, and require you to install Data Management Gateway
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Azure Data Factory Service Activities » Data Movement » Others
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Activities » Data Movement » Others
Data Store
Supported as Source
Supported as Sink
Salesforce  
Generic ODBC * 
Generic OData 
Web Table (HTML) 
GE Historian *
* Data store can be on-premises or on Azure IaaS, and require you to install Data Management Gateway
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Azure Data Factory Service Activities » Data Transformation
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Activities » Data Transformation
Data Factory supports numerous transformation activities that can be added to pipelines either individually or chained with another activity
Data Transformation Activity
Compute Environment
Hive
HDInsight [Hadoop]
Pig
MapReduce
Hadoop Streaming
Machine Learning activities: Batch Execution and Update Resource
Azure VM
Stored Procedure
Azure SQL Azure SQL Data Warehouse, or SQL Server
Data Lake Analytics U-SQL
Azure Data Lake Analytics
.NET (custom activity)
HDInsight [Hadoop] or Azure Batch
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Azure Data Factory Service Linked Data Services
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Linked Data Services
Linked services define the information needed for Data Factory to connect to external resources
Linked services are used for two purposes:
To represent a data store including, but not limited to, an on-premises SQL Server database, Oracle database, file share, or an Azure Blob Storage account
To represent a compute resource that can host the execution of an activity
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Azure Data Factory Service Datasets
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Datasets
Linked services link data stores to a Data factory
Datasets represent data structures within the data stores
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Azure Data Factory Service Relationship Between Data Factory entities
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Relationship Between Data Factory entities
Data Factory has a few key entities that work together to define:
Input and output data
Processing events, and
The schedule and resources required to execute the desired data flow
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Azure Data Factory Service Data Flow Concepts
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Data Flow Concepts
Datasets
(Collection of files, database table, etc)
Pipeline: a sequence of
activities (logical group)
Activity: a processing step
(Hadoop job, custom code, ML model, etc)
Data Sources
Ingest
Transform & Analyze
Publish
On Premises
Data Mart
Call Log Files
Customer Table
Azure Data Lake
Call Log Files
Customer Table
Customer Call Details
Customers Likely to Churn
Transform, Combine, etc
Analyze
Move
Azure DW
Customer Churn Table
Visualize
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Azure Data Factory Service Supported Regions
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Supported Regions
Currently, you can create data factories in the following three regions:
West US
East US, and
North Europe
However, a data factory can access data stores and compute services in other Azure regions to move data between data stores or process data using compute services
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. Azure Data Factory Service Supported Regions
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Supported Regions
Data Factory itself does not store any data
It lets you create data-driven flows to orchestrate movement of data between supported data stores and processing of data using compute services in other regions or in an on-premises environment
It also allows you to monitor and manage workflows using both programmatic and UI mechanisms
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Azure Data Factory Service Supported Regions
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Supported Regions
Even though Azure Data Factory is available in only three regions, the service powering the data movement in Data Factory is available globally in several regions
In case a data store sits behind a firewall then a Data Management Gateway (DMG) can be installed on-premises environment moves the data instead
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Azure Data Factory Service Resources
TechReady 23
5/13/ :27 PM
Azure Data Factory Service Resources
Introduction to Azure Data Factory Service, a data integration service in the cloud
Tutorial: Build your first pipeline to process data using Hadoop cluster
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. © 2016 Microsoft Corporation. All rights reserved
© 2016 Microsoft Corporation. All rights reserved. Microsoft, Windows, Microsoft Azure, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION