1. Matthew Roche
Senior Program Manager
Microsoft
@SQLAllFather
STOCKHOLM
Integrating Power BI and Azure Data Lake with dataflows and CDM Folders

2. Our sponsors

3. I really wanted to come to Stockholm… 

4. Ask me questions on Twitter!

5. Introducing Power BI dataflows, three ways

6. Power BI dataflows are part of the evolution of BI
Self-Service BI
Data Warehouse
Reports and dashboards
OLTP systems
Data Lake
Data Preparation / ETL
OLAP / Analytics Models

7. Power BI dataflows are part of the evolution of BI
Self-Service BI
Data Warehouse
Reports and dashboards
OLTP systems
Data Lake
Data Preparation / ETL
OLAP / Analytics Models

8. Power BI dataflows are another object/artifact type
5/9/2019 8:17 AM
Power BI dataflows are another object/artifact type
Reports & dashboards
Datasets
Dataflows
Azure Data Lake Storage Gen2
CDM folder
CDM folder
CDM folder
Business analysts
Low/no code
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Power BI dataflows are like Excel
5/9/2019 8:17 AM
Power BI dataflows are like Excel
Sources 20
Ingest from Dynamics Sales
22 entities
Clean and enrich sales data
10 entities 8
CRM – Production
Dynamics 365 10
Final Business View
11 entities 4 1 1
IoT Signal
Azure Data Lake Storage
Product Telemetry in Azure
5 entities
Add Telemetry Customer Attributes
6 entities 4 1
Product Usage Dataset
Sales/Telemetry Reference Data
External Dataflow
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. Demo: Power BI dataflows end to end

11. Self-Service BI Data Warehouse Reports and dashboards OLTP systems
Data Lake
Data Preparation / ETL
OLAP / Analytics Models

12. Integrating Power BI dataflows with Azure Data Lake Storage Gen2

13. Deliver ready-made insights to Power BI users from Azure
Business analysts
No code, low code
IT professionals, Data scientists
Low to high code
Visualize and report
Power BI
dataflows
Ingest
Model & serve
Azure SQL Data Warehouse
Train & predict
Azure Machine Learning
Advance data prep
Azure Databricks
Orchestrate & move
Azure Data Factory
Ingest
*Business Analysts read the diagram from left to right, while IT reads the diagram from right to left (to stay consistent with our Modern Data Warehouse pattern)*
*use presentation mode to trigger animations on click*
And here’s how everything I’ve showed you fits all together. Data ingested into Azure Data Lake Storage can be consumed by business analysts using Power BI or *Click* further enriched and leveraged by IT pros and data scientists using Azure.
At any point, data processed by any Azure Data Service can be written back to new CDM folders or *Click* connected directly to Power BI, making the insights created in Azure accessible to Power BI and other CDM-enabled apps or tools.
These integrations enable Power BI and Azure Data Services to be better when used together. For your organization, this means freeing valuable time and resources previously spent extracting and unifying data from different sources, and fully harnessing your data with more powerful analytics than ever.
CDM folders
Azure Data Lake Storage

14. Assign workspace to ADLS account
Enabling ADLSg2 Integration in Power BI
What “done” means:
Workspace Admins can now configure workspace to store its dataflows in ADLS account Global Admin configured
Dataflow CDM folder is stored in ADLSg2 and is accessible by its creator ONLY.
Other people in workspace can get data if they are authorized to the CDM folder in ADLSg2
Now data can be read from CDM Folder via Azure Data Services or LOB solutions that are CDM folder aware
In Power BI
Create and configure Storage account
Attach storage account to Power BI
Enable people to use it
Done
Global Admin
Assign workspace to ADLS account
Create and refresh dataflow
Create v2 workspace
Done
Workspace Admins
In Azure
Find dataflow CDM folder storage location
Get authorized to storage location
Attach to CDM folder
from Azure data services
Done
Developers
and
Data scientists

15. Demo: Azure Integration, two ways

16. What is the Common Data Model?

17. The Common Data Model (1 of 2)
The Common Data Model is a metadata system that simplifies data management and application development by unifying data into a known form and applying structural and semantic consistency across multiple apps and deployments.
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. The Common Data Model (2 of 2)
In addition to the metadata system, the Common Data Model includes a set of standardized, extensible data schemas that Microsoft and its partners have published. This collection of predefined schemas includes entities, attributes, semantic metadata, and relationships.
The schemas represent commonly used concepts and activities, such as Account and Campaign, to simplify the creation, aggregation, and analysis of data.
Industry accelerators docs:
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. What are Common Data Model folders?

20. Common Data Model folders (CDM folders)
A CDM folder is a folder in a data lake that conforms to specific, well-defined, and standardized metadata structures and self-describing data. These folders facilitate metadata discovery and interoperability between data producers and data consumers.
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. CDM folders include two types of content
5/9/2019 8:17 AM
CDM folders include two types of content
model.json
A metadata file in a folder in an Azure Data Lake Storage Gen2 instance that follows the Common Data Model metadata format.
Data files
CSV data files in a Common Data Model folder have a well-defined structure and format and are referenced in the model.json file.
model.json docs:
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Common Data Model folders (CDM folders)
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Common Data Model folders (CDM folders)
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. Why am I asking these questions?

25. CDM folders are the “magic glue” between Azure and Power BI
Business analysts
No code, low code
IT professionals, Data scientists
Low to high code
Visualize and report
Power BI
dataflows
Ingest
Model & serve
Azure SQL Data Warehouse
Train & predict
Azure Machine Learning
Advance data prep
Azure Databricks
Orchestrate & move
Azure Data Factory
Ingest
*Business Analysts read the diagram from left to right, while IT reads the diagram from right to left (to stay consistent with our Modern Data Warehouse pattern)*
*use presentation mode to trigger animations on click*
And here’s how everything I’ve showed you fits all together. Data ingested into Azure Data Lake Storage can be consumed by business analysts using Power BI or *Click* further enriched and leveraged by IT pros and data scientists using Azure.
At any point, data processed by any Azure Data Service can be written back to new CDM folders or *Click* connected directly to Power BI, making the insights created in Azure accessible to Power BI and other CDM-enabled apps or tools.
These integrations enable Power BI and Azure Data Services to be better when used together. For your organization, this means freeing valuable time and resources previously spent extracting and unifying data from different sources, and fully harnessing your data with more powerful analytics than ever.
CDM folders
Azure Data Lake Storage

26. Power BI dataflows Positioning and use cases

27. Power BI dataflows sort-of-FAQ
5/9/2019 8:17 AM
Power BI dataflows sort-of-FAQ
A new capability for self-service data preparation in Power BI
Delivered in a familiar Power Query experience
Built on the foundation of Azure Data Lake Storage gen2
Utilize the CDM folder format for data storage
A tool for business users to drive data reuse without requiring IT involvement
Enable Excel-like data lineage and orchestration
NOT a replacement for datasets
NOT a replacement for a data warehouse
NOT a replacement for Azure Data Factory or SSIS
NOT a Premium-only feature
NOT an additional cost or fee
NOT spelled with a space or any capital letters
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Positioning Power BI dataflows
Dataflows are for Power Query users
Easy to build reusable data entities
Easier to compose and orchestrate
Multi-stage, multi-user data prep workflows
Dataflows fill a self-service gap in the end-to-end story
Without dataflows, users will
Be blocked on IT involvement
Use Excel and manual processes
Require 3rd party data preparation tools like Alteryx, Datameer, Trifacta, etc.
Dataflows are a key bridge between Power BI and Azure
CDM folders and integration with ADLSg2 enable simple collaboration between business and IT
CDM and CDM folders are strategic technologies beyond Power BI

29. Canonical Production Customer Scenario
Sources
Dataflows
Datasets
Metric Base Tables
Ingest
PostgreSQL
Link
Ingest
Metric
Final calculated cleansed data
Spark (ODBC)
Workspace
Link
See
Metric – specific business line
Filtered to the specific line of business
Metric – specific business line
Filtered to the specific line of business
Metric – specific business line
Metric – specific business line
Workspace
Workspace
Workspace
Workspace
See also:
© Microsoft Corporation

30. Session resources Dataflows on public Microsoft sites:
Tech Ready 15
5/9/2019
Session resources
Dataflows on public Microsoft sites:
Dataflows documentation:
Dataflows roadmap / release notes:
Dataflows on Power BI Ideas:
Dataflows on Power BI community forum: link
Common Data Model on public Microsoft sites: :
Common Data Model documentation:
CDM Folder model metadata:
Common Data Model on GitHub:
End to end CDM partner sample:
Matthew’s blog:
Dataflows landing page:
Dataflows FAQ:
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

31. Thank you!