Introducing Power BI dataflows Matthew Roche Senior Program Manager Microsoft @SQLAllFather https://ssbipolar.com Introducing Power BI dataflows

Please Thank & Support our Sponsors SQL Saturday is made possible with the generous support of these sponsors. You can support them by opting-in and visiting them in the sponsor area. Global Alliance Partner: Venue & Internet Sponsor: Silver Sponsor: Operations Support: Endless Support from: OGMA Consulting Corp. Bronze Sponsor:

Introducing Power BI dataflows, Three Ways

Power BI dataflows are part of the evolution of BI Self-Service BI Data Warehouse Reports and dashboards OLTP systems Data Lake Data Preparation / ETL OLAP / Analytics Models

Power BI dataflows are another object/artifact type 5/5/2019 7:53 PM Power BI dataflows are another object/artifact type Reports & dashboards Datasets Dataflows Azure Data Lake Storage Gen2 CDM folder CDM folder CDM folder Business analysts Low/no code © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Power BI dataflows are like Excel 5/5/2019 7:53 PM Power BI dataflows are like Excel Sources 20 Ingest from Dynamics Sales 22 entities Clean and enrich sales data 10 entities 8 CRM – Production Dynamics 365 10 Final Business View 11 entities 4 1 1 IoT Signal Azure Data Lake Storage Product Telemetry in Azure 5 entities Add Telemetry Customer Attributes 6 entities 4 1 Product Usage Dataset Sales/Telemetry Reference Data External Dataflow © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo: Power BI dataflows end to end

Self-Service BI Data Warehouse Reports and dashboards OLTP systems Data Lake Data Preparation / ETL OLAP / Analytics Models

Power BI dataflows sort-of-FAQ 5/5/2019 7:53 PM Power BI dataflows sort-of-FAQ A new capability for self-service data preparation in Power BI Delivered in a familiar Power Query experience Built on the foundation of Azure Data Lake Storage gen2 Utilize the CDM folder format for data storage A tool for business users to drive data reuse without requiring IT involvement Enable Excel-like data lineage and orchestration NOT a replacement for datasets NOT a replacement for a data warehouse NOT a replacement for Azure Data Factory or SSIS NOT a Premium-only feature NOT an additional cost or fee NOT spelled with a space or any capital letters © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Integrating Power BI dataflows with Azure Data Lake Storage gen2

Using an organizational ADLSg2 resource with Power BI 5/5/2019 7:53 PM Using an organizational ADLSg2 resource with Power BI Reports & dashboards Azure Data Factory Azure Databricks Azure SQL DW Azure ML Datasets Dataflows Azure Data Lake Storage Gen2 CDM folder CDM folder CDM folder Data scientists Data engineers Low to high code Business analysts Low/no code © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Assign workspace to ADLS account BYOSA E2E – 3 steps What “done” means: Workspace Admins can now configure workspace to store its dataflows in ADLS account Global Admin configured Dataflow CDM folder is stored in ADLS and is accessible by its creator ONLY. Other people in workspace can get data if they are authorized to the CDM folder outside of Power BI Now data can be read from CDM Folder via Azure Data Services or LOB solutions that are CDM folder aware In Power BI Create and configure Storage account Attach storage account to Power BI Enable people to use it Done Global Admin Assign workspace to ADLS account Create and refresh dataflow Done Create v2 workspace Workspace Admins In Azure Find dataflow CDM folder storage location Attach to CDM folder from “Azure Data services" Get authorized to storage location Done Developers and Data scientists

Bring your own storage account 5/5/2019 7:53 PM Bring your own storage account Use your organizational ADLSg2 resource and attach it to Power BI A Global Administrator account and an Azure subscription is required Once a dataflow storage location is configured, it cannot be changed By default, only owners of a dataflow stored in Azure Data Lake Storage Gen2 can access its data To authorize additional users to the dataflows stored in Azure, you must add them to the dataflow’s CDM folder Creating dataflows with linked entities is only possible when they are stored in the same storage account © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Deliver ready-made insights to Power BI users from Azure 5/5/2019 7:53 PM Deliver ready-made insights to Power BI users from Azure Self service customizations in Power BI Dynamics 365 Azure Data Factory Azure Databricks Azure SQL DW Azure ML Dataflow Dynamics 365 data Azure Data Lake Storage Gen2 CDM folder © Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo: Azure Integration, Two Ways

Wrapping Things Up

Positioning Power BI dataflows Dataflows are for Power Query users Easy to build reusable data entities Easier to compose and orchestrate Multi-stage, multi-user data prep workflows Dataflows fill a self-service gap in the end-to-end story Without dataflows, users will Be blocked on IT involvement Use Excel and manual processes Require 3rd party data preparation tools like Alteryx, Datameer, Trifacta, etc. Dataflows bridge Power BI and Azure CDM folders and BYOSA enable simple collaboration between business and IT CDM and CDM folders are strategic technologies beyond Power BI

Production Customer Scenario Sources Dataflows Datasets Metric Base Tables Ingest PostgreSQL Link Ingest Metric Final calculated cleansed data Spark (ODBC) Workspace Link Metric – specific business line Filtered to the specific line of business Metric – specific business line Filtered to the specific line of business Metric – specific business line Metric – specific business line Workspace Workspace Workspace Workspace © Microsoft Corporation

Session resources Dataflows on public Microsoft sites: Tech Ready 15 5/5/2019 Session resources Dataflows on public Microsoft sites: Dataflows documentation: https://docs.microsoft.com/en-us/power-bi/service-dataflows-overview Dataflows roadmap / release notes: https://docs.microsoft.com/en-us/business-applications-release-notes/April19/business-intelligence/power-bi-service/power-bi-dataflows/self-service-data-prep-with-dataflows Dataflows on Power BI Ideas: https://ideas.powerbi.com/forums/265200-power-bi-ideas?category_id=341638 Dataflows on Power BI community forum: link Common Data Model on public Microsoft sites: : Common Data Model documentation: https://docs.microsoft.com/en-us/common-data-model/ CDM Folder model metadata: https://docs.microsoft.com/en-us/common-data-model/model-json Common Data Model on GitHub: https://github.com/Microsoft/CDM End to end CDM partner sample: https://aka.ms/cdmadsblog Matthew’s blog: Dataflows landing page: https://ssbipolar.com/2018/10/23/dataflows-in-power-bi/ Dataflows FAQ: https://ssbipolar.com/2018/11/27/power-bi-dataflows-faq/ © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Thank you!