1. Get your data flowing with Data Flows! and...umm...dataflows.
Nicholas Schafer
Learner of Business Intelligence

2. A famous quote
“Microsoft recently launched Data Flows in Azure Data Factory (ADF) to move the service beyond the pure orchestration space into the world of extract, transform and load (ETL). At roughly the same time, Microsoft also introduced dataflows into the Power BI service world, allowing users to leverage common query models across an enterprise. So, what do we do with these new offerings? What are they, really? Is Microsoft just trying to confuse us? This session untangles and explores ADF Data Flows and Power BI dataflows to help you to get a better understanding of how you may be able to apply them in your world.”
-Nicholas Schafer

3. A confession(s)
Earlier this week, I was not very prepared because, well…reasons. Now it’s a bit better.
I stole content from a presentation by Miguel Llopis and Matthew Roche from Ignite on Power BI dataflows (Ignite 2018 presentation titled Microsoft Power BI: Unify all your data and deliver powerful insights with self-service data prep capabilities for big data - BRK2061)
I stole content from a presentation by Mark Kromer from SQL Saturday in Redmond on Azure Data Factor Data Flows (
I am learning about Data Flows and dataflows – I am not an expert. : )

4. I have an agenda Overview | Data Flows and dataflows
Azure Data Factory | Data Flows
Power BI | dataflows
Conclusion | Data Flows and dataflows

5. Overview | Data Flows and dataflows

6. Modern Business Intelligence (BI) Challenge
Fragmented, incomplete data
Pulling together data from traditional and cloud data sources and figuring out how to enrich it is extremely difficult.
Requires a team of specialists
Creating E2E BI solutions requires multiple BI tools. This requires specific knowledge of each of the tools and complex integration to build and maintain an E2E BI solution.
Business data has no structural or semantic consistency 
Different applications, departments, and analysts define data in different ways, which makes data exploration, and reuse of data and apps extremely challenging.
Complex system integration
Traditional BI solutions span multiple applications and services. Sharing data across systems requires each system to understand the location, structure and meaning of the data.
Source: Ignite 2018 presentation titled Microsoft Power BI: Unify all your data and deliver powerful insights with self-service data prep capabilities for big data - BRK2061

7. Title

8. Azure Data Factory | Data Flows

9. What is Azure Data Factory (ADF)?

10. How do Data Flows fit into ADF?
Applications
Dashboards
Business/custom apps (structured) r
Ingest storage
Azure Storage/ Data Lake Store
Data Loading
Azure Data Factory
Load files into data lake on a schedule
Data Flow Data Transformation
Extract and transform relational data
Serving storage
Azure SQL DW
Load processed data into tables optimized for analytics
Clean and join disparate data
Scheduled & orchestrated by ADF
Azure Databricks
Source:

11. A simpler explanation

12. So, what’s so cool about it? It’s a visual t.
Design code-free ETL workflows
Copy data from on-prem, other clouds and Azure
Stage data for transformation
Build visual data transformations
Schedule triggers for your pipeline execution
Monitor processes and configure alerts
All within ADF
Transform Data, At Scale, in the Cloud, Zero-Code
Cloud-first, scale-out ELT
Code-free dataflow pipelines
Serverless scale-out transformation execution engine
Maximum Productivity for Data Engineers
Does NOT require understanding of Spark / Scala / Python / Java
Resilient Data Transformation Flows
Built for big data scenarios with unstructured data requirements
Operationalize with Data Factory scheduling, control flow and monitoring
Source:

13. Code free transformations at scale
Does not require understanding of Spark, Big Data Execution Engines, Clusters, Scala, Python …so you can focus on building business logic and data transformation
What can you do code free?
Data cleansing
Aggregation
Data conversions
Data prep
Data exploration
… not …
Source:

14. Build your logical data flows adding data transformations in a guided experience
Source:

15. Debug mode provides row-level context and visible results in inspector pane
Source:

16. Debug Data Flows with Data Preview and Data Sampling with Inspect Pane
Source:

17. Deep Monitoring Introspection of Data Transformations
Source:

18. Microsoft Azure Data Factory Continues to Extend Data Flow Library with a Rich Set of Transformations and Expression Functions
Source:

19. Interactive Expression Builder – Build data transform expressions, not Spark code
Source:

20. Let’s have a look

21. Power BI | dataflows

22. What is Power BI?

23. What is Power BI? Power BI Desktop Power BI Desktop Power BI Service
Get Data
Build Reports
Collaborate
Share
Power BI Data Sources
Adding Visuals
Power BI Workspaces
Power BI Apps
Data Set
Data Set
Reports
Data Set
Reports
Dashboards
Worksheets
Data Set
Reports
Dashboards
Worksheets

24. Assembled into data model
What is Power BI?
Source data
Power BI Desktop
Power BI Desktop
Power BI Service
Power BI Service
Get Data
Build Reports
Collaborate
Share
Power BI Data Sources
Adding Visuals
Power BI Workspaces
Power BI Apps
Transformed to tables
Assembled into data model
Data Set
Data Set
Reports
Data Set
Reports
Dashboards
Worksheets
Data Set
Reports
Dashboards
Worksheets

25. What is a Power BI dataflow?
Dynamics 365 for Finance & Operations
Azure Data Factory
Azure Databricks
Azure SQL DW
Self service customizations
in Power BI
Azure ML
Dataflow
Dynamics 365 data
Azure Data Lake Storage Gen2
CDM folder

26. A simpler explanation

27. What is Power Query With Power Query, you can:
Connect to data to create a new Query
Create multiple queries from multiple data sources to support your data model
Preview your data and write formulas in the formula bar to transform it
Modify the query properties and apply transformation steps

28. What is the Common Data Model?
The Common Data Model is a standardized, modular, extensible collection of data schemas published by Microsoft that are designed to make it easier for you to build, use, and analyze data.
Power BI Dataflows allows you to ingest data into the Common Data Model form from a variety of sources such as Dynamics 365, Salesforce, Azure SQL Database, Excel, or SharePoint. Once you've connected and prepared your data, you can choose to map it to a Common Data Model standard entity or load it as a custom entity in Common Data Model form in Azure Data Lake Storage Gen2.
Azure Data Lake Storage Gen2 helps speed your transition from proof of concept to production by combining the power of a file system that's compatible with Hadoop, an integrated hierarchical namespace, and the massive scale and economy of Azure Blob Storage. The Common Data Model brings semantic consistency to data within the lake so that applications and services can interoperate more easily when data is stored in Common Data Model form.
Common Data Service, which supports Dynamics and PowerApps, stores data in conformance with the Common Data Model definition. In fact, many of the original business entities in the Common Data Model came from Dynamics offerings, such as Dynamics 365 for Sales and Dynamics 365 for Marketing.
Industries such as healthcare are working closely with Microsoft to extend the Common Data Model to their specific business concepts, such as Patient and Care Plan. This extends the benefit of the Common Data Model standard entities to these verticals so that industry solutions interoperate more easily.

29. What is the Common Data Model

30. Create and use dataflows
There are three primary steps to using a dataflow:
1. Author the dataflow, using Microsoft tools that are designed to make doing so straightforward
2. Schedule the refresh frequency of the data you want to bring into your dataflow
3. Build the dataset using your dataflow, using Power BI Desktop

31. Data Sources

32. Connecting to dataflows in Power BI Desktop

33. Let’s have a look

34. Conclusion | Data Flows and dataflows

35. Conclusion

36. Resources Azure Data Factory Data Flows
Microsoft Docs Mapping Data Flows Concepts (
Microsoft Docs Create data flow Quickstart (
Azure Data Factor Data Flows - Mark Kromer from SQL Saturday in Redmond on (
Power BI dataflows
Microsoft Docs Power BI Self-Service dataflows (
Microsoft Power BI: Unify all your data and deliver powerful insights with self-service data prep capabilities for big data - BRK2061 Ignite 2018 presentation by Miguel Llopis and Matthew Roche
Sample Data
Kaggle Lego Database (