Matthew Roche Senior Program Manager Microsoft @SQLAllFather https://ssbipolar.com STOCKHOLM Integrating Power BI and Azure Data Lake with dataflows and CDM Folders

Our sponsors

I really wanted to come to Stockholm… 

Ask me questions on Twitter!

Introducing Power BI dataflows, three ways

Power BI dataflows are part of the evolution of BI Self-Service BI Data Warehouse Reports and dashboards OLTP systems Data Lake Data Preparation / ETL OLAP / Analytics Models

Power BI dataflows are part of the evolution of BI Self-Service BI Data Warehouse Reports and dashboards OLTP systems Data Lake Data Preparation / ETL OLAP / Analytics Models

Power BI dataflows are another object/artifact type 5/9/2019 8:17 AM Power BI dataflows are another object/artifact type Reports & dashboards Datasets Dataflows Azure Data Lake Storage Gen2 CDM folder CDM folder CDM folder Business analysts Low/no code © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Power BI dataflows are like Excel 5/9/2019 8:17 AM Power BI dataflows are like Excel Sources 20 Ingest from Dynamics Sales 22 entities Clean and enrich sales data 10 entities 8 CRM – Production Dynamics 365 10 Final Business View 11 entities 4 1 1 IoT Signal Azure Data Lake Storage Product Telemetry in Azure 5 entities Add Telemetry Customer Attributes 6 entities 4 1 Product Usage Dataset Sales/Telemetry Reference Data External Dataflow © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo: Power BI dataflows end to end

Self-Service BI Data Warehouse Reports and dashboards OLTP systems Data Lake Data Preparation / ETL OLAP / Analytics Models

Integrating Power BI dataflows with Azure Data Lake Storage Gen2

Deliver ready-made insights to Power BI users from Azure Business analysts No code, low code IT professionals, Data scientists Low to high code Visualize and report Power BI dataflows Ingest Model & serve Azure SQL Data Warehouse Train & predict Azure Machine Learning Advance data prep Azure Databricks Orchestrate & move Azure Data Factory Ingest *Business Analysts read the diagram from left to right, while IT reads the diagram from right to left (to stay consistent with our Modern Data Warehouse pattern)* *use presentation mode to trigger animations on click* And here’s how everything I’ve showed you fits all together. Data ingested into Azure Data Lake Storage can be consumed by business analysts using Power BI or *Click* further enriched and leveraged by IT pros and data scientists using Azure. At any point, data processed by any Azure Data Service can be written back to new CDM folders or *Click* connected directly to Power BI, making the insights created in Azure accessible to Power BI and other CDM-enabled apps or tools. These integrations enable Power BI and Azure Data Services to be better when used together. For your organization, this means freeing valuable time and resources previously spent extracting and unifying data from different sources, and fully harnessing your data with more powerful analytics than ever. CDM folders Azure Data Lake Storage

Assign workspace to ADLS account Enabling ADLSg2 Integration in Power BI What “done” means: Workspace Admins can now configure workspace to store its dataflows in ADLS account Global Admin configured Dataflow CDM folder is stored in ADLSg2 and is accessible by its creator ONLY. Other people in workspace can get data if they are authorized to the CDM folder in ADLSg2 Now data can be read from CDM Folder via Azure Data Services or LOB solutions that are CDM folder aware In Power BI Create and configure Storage account Attach storage account to Power BI Enable people to use it Done Global Admin Assign workspace to ADLS account Create and refresh dataflow Create v2 workspace Done Workspace Admins In Azure Find dataflow CDM folder storage location Get authorized to storage location Attach to CDM folder from Azure data services Done Developers and Data scientists

Demo: Azure Integration, two ways

What is the Common Data Model?

The Common Data Model (1 of 2) The Common Data Model is a metadata system that simplifies data management and application development by unifying data into a known form and applying structural and semantic consistency across multiple apps and deployments. © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

The Common Data Model (2 of 2) In addition to the metadata system, the Common Data Model includes a set of standardized, extensible data schemas that Microsoft and its partners have published. This collection of predefined schemas includes entities, attributes, semantic metadata, and relationships. The schemas represent commonly used concepts and activities, such as Account and Campaign, to simplify the creation, aggregation, and analysis of data. https://docs.microsoft.com/en-us/common-data-model/use Industry accelerators docs: https://docs.microsoft.com/en-us/common-data-model/industry-accelerators © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

What are Common Data Model folders?

Common Data Model folders (CDM folders) A CDM folder is a folder in a data lake that conforms to specific, well-defined, and standardized metadata structures and self-describing data. These folders facilitate metadata discovery and interoperability between data producers and data consumers. © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

CDM folders include two types of content 5/9/2019 8:17 AM CDM folders include two types of content model.json A metadata file in a folder in an Azure Data Lake Storage Gen2 instance that follows the Common Data Model metadata format. Data files CSV data files in a Common Data Model folder have a well-defined structure and format and are referenced in the model.json file. model.json docs: https://docs.microsoft.com/en-us/common-data-model/model-json © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Common Data Model folders (CDM folders) © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Common Data Model folders (CDM folders) © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Why am I asking these questions?

CDM folders are the “magic glue” between Azure and Power BI Business analysts No code, low code IT professionals, Data scientists Low to high code Visualize and report Power BI dataflows Ingest Model & serve Azure SQL Data Warehouse Train & predict Azure Machine Learning Advance data prep Azure Databricks Orchestrate & move Azure Data Factory Ingest *Business Analysts read the diagram from left to right, while IT reads the diagram from right to left (to stay consistent with our Modern Data Warehouse pattern)* *use presentation mode to trigger animations on click* And here’s how everything I’ve showed you fits all together. Data ingested into Azure Data Lake Storage can be consumed by business analysts using Power BI or *Click* further enriched and leveraged by IT pros and data scientists using Azure. At any point, data processed by any Azure Data Service can be written back to new CDM folders or *Click* connected directly to Power BI, making the insights created in Azure accessible to Power BI and other CDM-enabled apps or tools. These integrations enable Power BI and Azure Data Services to be better when used together. For your organization, this means freeing valuable time and resources previously spent extracting and unifying data from different sources, and fully harnessing your data with more powerful analytics than ever. CDM folders Azure Data Lake Storage

Power BI dataflows Positioning and use cases

Power BI dataflows sort-of-FAQ 5/9/2019 8:17 AM Power BI dataflows sort-of-FAQ A new capability for self-service data preparation in Power BI Delivered in a familiar Power Query experience Built on the foundation of Azure Data Lake Storage gen2 Utilize the CDM folder format for data storage A tool for business users to drive data reuse without requiring IT involvement Enable Excel-like data lineage and orchestration NOT a replacement for datasets NOT a replacement for a data warehouse NOT a replacement for Azure Data Factory or SSIS NOT a Premium-only feature NOT an additional cost or fee NOT spelled with a space or any capital letters © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Positioning Power BI dataflows Dataflows are for Power Query users Easy to build reusable data entities Easier to compose and orchestrate Multi-stage, multi-user data prep workflows Dataflows fill a self-service gap in the end-to-end story Without dataflows, users will Be blocked on IT involvement Use Excel and manual processes Require 3rd party data preparation tools like Alteryx, Datameer, Trifacta, etc. Dataflows are a key bridge between Power BI and Azure CDM folders and integration with ADLSg2 enable simple collaboration between business and IT CDM and CDM folders are strategic technologies beyond Power BI

Canonical Production Customer Scenario Sources Dataflows Datasets Metric Base Tables Ingest PostgreSQL Link Ingest Metric Final calculated cleansed data Spark (ODBC) Workspace Link See https://ssbipolar.com/2018/12/05/a-spectrum-of-data-enrichment-and-reuse/ Metric – specific business line Filtered to the specific line of business Metric – specific business line Filtered to the specific line of business Metric – specific business line Metric – specific business line Workspace Workspace Workspace Workspace See also: https://ssbipolar.com/2018/12/05/a-spectrum-of-data-enrichment-and-reuse/ © Microsoft Corporation

Session resources Dataflows on public Microsoft sites: Tech Ready 15 5/9/2019 Session resources Dataflows on public Microsoft sites: Dataflows documentation: https://docs.microsoft.com/en-us/power-bi/service-dataflows-overview Dataflows roadmap / release notes: https://docs.microsoft.com/en-us/business-applications-release-notes/April19/business-intelligence/power-bi-service/power-bi-dataflows/self-service-data-prep-with-dataflows Dataflows on Power BI Ideas: https://ideas.powerbi.com/forums/265200-power-bi-ideas?category_id=341638 Dataflows on Power BI community forum: link Common Data Model on public Microsoft sites: : Common Data Model documentation: https://docs.microsoft.com/en-us/common-data-model/ CDM Folder model metadata: https://docs.microsoft.com/en-us/common-data-model/model-json Common Data Model on GitHub: https://github.com/Microsoft/CDM End to end CDM partner sample: https://aka.ms/cdmadsblog Matthew’s blog: Dataflows landing page: https://ssbipolar.com/2018/10/23/dataflows-in-power-bi/ Dataflows FAQ: https://ssbipolar.com/2018/11/27/power-bi-dataflows-faq/ © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Thank you!