Data Integration with Power Query

Let’s talk about me Nick Chobotar Business Intelligence (BI) Analyst, I am out of Philadelphia office. I specialize in development and implementation of business intelligence solutions using Microsoft BI stack in combination with dimensional data modelling techniques. My experience includes design and development of data warehouses and data marts, ETL and standardization of data from various conventional and unconventional data sources, design and development of tabular SSAS semantic solutions as well as dashboards and end user reporting. Contact: nick.chobotar@nihilentinc.com

Let’s talk about Nihilent Technologies, Inc. BI & Advanced Analytics Cloud Transformation Services Enterprise Transformation Consulting Customer Relationship Management Application Development Portals & Collaboration 1,500+ Employees 14,000+ Migrated Cloud Users & devices Healthcare Manufacturing Financial Services Retail

Proven Delivery Healthcare Manufacturing Financial Services Retail

Agenda Power Query introduction Power Query M Language Basics Type system Joins Parameters Model Overview and Design Data extraction from various sources: SQL Server (Product Subcategory) Odata (Product) OneDrive (Product Category) List.Generate (Sales fact – mockup) DimDate (Calendar) dimension Data transformation Query grouping

Power Query Evolution Early 2013 Data Explorer excel add-in Late 2013 Power Query excel add-in 2016 Excel (native) 2015 part of Power BI Desktop 2017 Power Apps 2017 SQL Server 2018 Power BI Web service (Datapools in preview)

Power Query SDK The Power Query SDK provides an M language service for Visual Studio, and a framework for building Data Connectors for Microsoft Power BI. This extension adds support for the .mproj project type as well as syntax highlighting and intellisense for .pq files

Power Query M Language Basics

Standard Library Functions M function 25 Categories 1) Accessing data functions 2) Action functions 3) Binary functions 4) Combiner functions 5) Comparer functions 6) Date functions 7) DateTime functions 8) DateTimeZone functions 9) Duration functions 10) Error handling 11) Expression functions 12) Function values 13) List functions 14) Lines functions 15) Logical functions 16) Number functions 17) Record functions 18) Replacer functions 19) Splitter functions 20) Table functions 21) Text functions 22) Time functions 23) Type functions 24) Uri functions 25) Value functions =#shared is my best friend powerpivotpro.com NOTE: All functions are prefixed with its category e.g. Text.Upper(), Number.Round(), List.Max(), Date.Year()

Power Query Types Built in Types: Complex Types: binary, date, datetime, datetimezone, duration, logical, null, number, text, time Complex Types: Tables Lists Records Functions

Types Demo

Joins Merge Queries with UI Create a Join with M

Joins Demo

Parameters Parameter is a query that returns a single value that could be used by multiple queries. Three ways of working with parameters: Home -> Manage Parameters button -> Click New New Query -> Blank Query (right click Covert To Parameter) New Query -> Blank Query ( Parameter as a query name) Common uses: Elimination of hardcoding Data filtering Changing data sources

Parameters Demo

Model Overview Data Sources Transformations Finished Model OneDrive Generated Fact SQL Server OData Generated Calendar Data Sources Transformations Finished Model

Model Diagram

Fact Table with List.Generate Generates a list of values given four functions that generate the initial value initial, test against a condition condition, and if successful select the result and generate the next value next. List.Generate( A starting condition, When to stop, How to make the next record, What to put into the list ) Excellent technique for mocking data Fact table fields: Date ProductID, Quantity, UnitCost, UnitSalesPrice

Date (Calendar) Dimension Year QuarterNumber QuarterIndex Quarter MonthNumber MonthIndex Month MonthName MonthNameShort WeekNumber WeekIndex Week DayOfYear DayOfQuarter DayOfMonth DayOfWeek DayName DayNameShort Weekday WeekdayFlag WorkdayFlag CurrentYear CurrentQuarter CurrentMonth CurrentWeek Today CurrentYTD CurrentQTD CurrentMTD CurrentWTD QuarterVisual

SQL Server DimProductSubcategory Dimension Query Folding - pushing certain data transformations back to the data source. List of the most common transformations Query folding Filtering (on rows or columns) Joins Aggregates and GROUP BY Pivot and unpivot Numeric calculations Simple transformations, such as UPPER Do not write custom M before query folding steps !!!! Audit steps or Groups SQL Server Profiler Power Query (right click applied step - “View Native Query”)

OneDrive DimProductCategory Dimension let sharingUrl = "insert URL here", base64Value = Binary.ToText(Text.ToBinary(sharingUrl, TextEncoding.Utf8), BinaryEncoding.Base64), encodedUrl = "https://api.onedrive.com/v1.0/shares/u!" & base64Value & "/root/content", Source = Web.Contents(encodedUrl) in Source https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/shares_get

Odata DimProduct Dimension AdventureWorks OData Feed sample CompanySales Documents ManufacturingInstructions ProductCatalog TerritorySalesDrilldown WorkOrderRouting Source: https://blogs.msdn.microsoft.com/derrick_vanarnams_blog/2012/09/05/announcing-the-adventureworks-odata-feed-sample/

Questions

References: Power Query SDK https://marketplace.visualstudio.com/items?itemName=Dakahn.PowerQuerySDK Power Apps https://powerapps.microsoft.com/ro-ro/blog/load-data-from-external-data-sources-to-the-common-data-service-using-power-query-2/ Searchable Power Query M reference https://www.powerquery.io/