Data Platform and Analytics Foundational Training Microsoft C+E Technology Training Data Platform and Analytics Foundational Training Solution Area Data Analytics Solution Advanced Analytics Technology Data Factory [Speaker Notes]

Azure Data Factory Service A data integration service in the cloud TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service A data integration service in the cloud Data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service A data integration service in the cloud TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service A data integration service in the cloud Data Factory service allows you to create data pipelines that move and transform data, and then run the pipelines on a specified schedule (hourly, daily, weekly, etc.). © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Pipelines and Activities TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Pipelines and Activities In a Data Factory solution, you create one or more data pipelines Activities define the actions to perform on your data Data movement Data transformation © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Activities » Data Movement TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Activities » Data Movement The Copy Activity in Data Factory copies data from a source data store to a sink data store © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Activities » Data Movement » Azure TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Activities » Data Movement » Azure Data Store Supported as Source Supported as Sink Azure Blob storage   Azure Data Lake Store  Azure SQL Database  Azure SQL Data Warehouse  Azure Table storage  Azure DocumentDB  Azure Search Index  © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Activities » Data Movement » Databases TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Activities » Data Movement » Databases Data Store Supported as Source Supported as Sink SQL Server *  Oracle * MySQL *  DB2 * Teradata *  PostgreSQL *  Sybase *  Cassandra *  MongoDB * Amazon Redshift * Data store can be on-premises or on Azure IaaS, and require you to install Data Management Gateway © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Activities » Data Movement » File TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Activities » Data Movement » File Data Store Supported as Source Supported as Sink File System *   HDFS *   Amazon S3  FTP * Data store can be on-premises or on Azure IaaS, and require you to install Data Management Gateway © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Activities » Data Movement » Others TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Activities » Data Movement » Others Data Store Supported as Source Supported as Sink Salesforce   Generic ODBC *  Generic OData  Web Table (HTML)  GE Historian * * Data store can be on-premises or on Azure IaaS, and require you to install Data Management Gateway © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Activities » Data Transformation TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Activities » Data Transformation Data Factory supports numerous transformation activities that can be added to pipelines either individually or chained with another activity Data Transformation Activity Compute Environment Hive HDInsight [Hadoop] Pig MapReduce Hadoop Streaming Machine Learning activities: Batch Execution and Update Resource Azure VM Stored Procedure Azure SQL Azure SQL Data Warehouse, or SQL Server Data Lake Analytics U-SQL Azure Data Lake Analytics .NET (custom activity) HDInsight [Hadoop] or Azure Batch © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Linked Data Services TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Linked Data Services Linked services define the information needed for Data Factory to connect to external resources Linked services are used for two purposes: To represent a data store including, but not limited to, an on-premises SQL Server database, Oracle database, file share, or an Azure Blob Storage account To represent a compute resource that can host the execution of an activity © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Datasets TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Datasets Linked services link data stores to a Data factory Datasets represent data structures within the data stores © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Relationship Between Data Factory entities TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Relationship Between Data Factory entities Data Factory has a few key entities that work together to define: Input and output data Processing events, and The schedule and resources required to execute the desired data flow © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Data Flow Concepts TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Data Flow Concepts Datasets (Collection of files, database table, etc) Pipeline: a sequence of activities (logical group) Activity: a processing step (Hadoop job, custom code, ML model, etc) Data Sources Ingest Transform & Analyze Publish On Premises Data Mart Call Log Files Customer Table Azure Data Lake Call Log Files Customer Table Customer Call Details Customers Likely to Churn Transform, Combine, etc Analyze Move Azure DW Customer Churn Table Visualize © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Supported Regions TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Supported Regions Currently, you can create data factories in the following three regions: West US East US, and North Europe However, a data factory can access data stores and compute services in other Azure regions to move data between data stores or process data using compute services © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Supported Regions TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Supported Regions Data Factory itself does not store any data It lets you create data-driven flows to orchestrate movement of data between supported data stores and processing of data using compute services in other regions or in an on-premises environment It also allows you to monitor and manage workflows using both programmatic and UI mechanisms © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Supported Regions TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Supported Regions Even though Azure Data Factory is available in only three regions, the service powering the data movement in Data Factory is available globally in several regions In case a data store sits behind a firewall then a Data Management Gateway (DMG) can be installed on-premises environment moves the data instead © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Azure Data Factory Service Resources TechReady 23 5/13/2018 10:27 PM Azure Data Factory Service Resources Introduction to Azure Data Factory Service, a data integration service in the cloud https://docs.microsoft.com/en-us/azure/data-factory/data-factory-introduction Tutorial: Build your first pipeline to process data using Hadoop cluster https://docs.microsoft.com/en-us/azure/data-factory/data-factory-build-your-first-pipeline © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

© 2016 Microsoft Corporation. All rights reserved © 2016 Microsoft Corporation. All rights reserved. Microsoft, Windows, Microsoft Azure, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION