sqlbi.com

sqlbi.com When Many To Many Are Really Too Many Alberto Ferrari Senior Consultant SQLBI.COM

Who’s Speaking? BI Expert and Consultant  Problem Solving  Complex Project Assistance  DataWarehouse Assesments and Development  Courses, Trainings and Workshops SQLBI: Branch of a Microsoft BI Gold Partner Book Author, SSAS Maestro «Spaghetti English»  Get prepared, I can’t help it

Agenda  Many To Many Relationships Data Model DAX Formula pattern  Classical Many To Many  Cascading Many To Many  Survey Data Model  Basket Analysis

Prerequisites  We will use, but not describe: Tabular basics DAX basics Evaluation contexts and interactions with relationships Data modeling with Many To Many  After all … this is a 400 session

Current Accounts Model M2M

No support for M2M in Tabular  Facts Multidimensional handles M2M relationships Tabular does not  Thus, my fastest ever session ends here  Or … we can dive into DAX and play with it

Demo – Classical M2M  We will always start looking at the final result  Then, we dive into the DAX code

The M2M DAX Pattern  Leverages CALCULATE Row Contexts Filter Contexts Automatic transformation of Row Context into Filter Context using CALCULATE  Next slide: the formula. Keep it in mind It will appear quite often from now on

The DAX Formula of M2M Pattern AmountM2M := CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, CALCULATE ( COUNTROWS (Bridge_AccountCustomer) ) > 0 ) )

What the formula should perform Filter on Dim_CustomerFilter on Dim_Account

How the formula works AmountM2M := CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, CALCULATE ( COUNTROWS (Bridge_AccountCustomer) ) > 0 ) ) The filter is applied by the Tabular data model

How the formula works AmountM2M := CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, CALCULATE ( COUNTROWS (Bridge_AccountCustomer) ) > 0 ) ) The filter is applied by the Tabular data model FILTER: For each row in Dim_Account CALCULATE: filters the bridge

AmountM2M := CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, CALCULATE ( COUNTROWS (Bridge_AccountCustomer) ) > 0 ) ) How the formula works The filter is applied by the Tabular data model Only rows in Dim_Account where COUNTROWS () > 0 survive the FILTER FILTER: For each row in Dim_Account CALCULATE: filters the bridge

AmountM2M := CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, CALCULATE ( COUNTROWS (Bridge_AccountCustomer) ) > 0 ) ) How the formula works This filter is applied by the Tabular data model FILTER: For each row in Dim_Account CALCULATE: filters the bridge Only rows in Dim_Account where COUNTROWS () > 0 survive the FILTER SUM is evaluated only for the accounts filtered from the customers

The Karma of CALCULATE BI123|many-to-many In DAX16 AmountM2M_Wrong := CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, COUNTROWS (Bridge_AccountCustomer) > 0 ) ) AmountM2M_Correct := CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, CALCULATE ( COUNTROWS (Bridge_AccountCustomer) ) > 0 ) )

AmountM2M := CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, COUNTROWS (Bridge_AccountCustomer) > 0 ) ) Wrong formula in action All the rows in the Account table survived the FILTER

AmountM2M := CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, CALCULATE ( COUNTROWS (Bridge_AccountCustomer) ) > 0 ) ) Many-to-many in action Two rows in the Account table survived the FILTER

The Karma of CALCULATE BI123|many-to-many In DAX19 AmountM2M_Wrong := CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, COUNTROWS (Bridge_AccountCustomer) > 0 ) ) AmountM2M_Correct := CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, CALCULATE ( COUNTROWS (Bridge_AccountCustomer) ) > 0 ) )

Many to Many - Conclusions  Complexity Not in the data model Only in the formula Good understanding of CALCULATE and of relationships in Tabular needed This is just the beginning of our journey

Cascading Many To Many

Filter on Dim_CategoryFilter on Dim_Account The pattern is the same, but this time we need to jump two steps to complete our task

Cascading M2M- Demo

Cascading Many To Many CALCULATE ( CALCULATE ( SUM (Fact_Transaction[Amount]), FILTER ( Dim_Account, CALCULATE ( COUNTROWS (Bridge_AccountCustomer) ) > 0 ) ), FILTER ( Dim_Customer, CALCULATE ( COUNTROWS (Bridge_CustomerCategory) ) > 0 ) )

Cascading Many To Many  Generic Formula  Works with any number of steps  Be careful Always start with the farthest table And move one step at a time in the direction of the fact table One CALCULATE for each step  Complexity: M x N (geometric … )

Cascading Alternative Need for some ETL here

Cascading Alternative The bridge now «feels» three different filters, but the formula becomes a classical many to many and runs faster

Flattened Cascading Many To Many  Flattened Data Model  Faster than the cascading one  Simpler formula  Needs ETL  Worth a try in Multidimensional too …

Survey

Survey Data Model  Facts: One customer answers many questions One question is answered by many customers We want to use a PivotTable to query this

Survey Scenario  Question: « What is the yearly income of consultants? » In other words …  Take the customers who answered «Consultant» at the «Job» question  Slice them using the answer to the question «Yearly Income»

Survey - Demo

Survey Analytical Data Model  Two «Filter» Dimensions One for «Job» = «Consultants» One for «Yearly Income» Question1 = «Job» Answer1 = «Consultant» Question2 = «Yearly Income»

Survey: The Final Result  Value = Customer Count  The fact table, this time, acts as the bridge

Survey Analytical Data Model No relationship between the fact table and the two filter dimensions Because here is only one value for ID_Answer

Survey Analytical Data Model  The two filters Are applied on the Customers table Use separate instances of the Answers table

Survey: The DAX Formula IF ( COUNTROWS (VALUES (Filter1[ID_Answer])) = 1 && COUNTROWS (VALUES (Filter2[ID_Answer])) = 1, CALCULATE ( COUNTROWS (Customers), FILTER ( Customers, CALCULATE ( COUNTROWS (Answers), Answers[ID_Answer] = VALUES (Filter2[ID_Answer])) > 0 ) ), FILTER ( Customers, CALCULATE ( COUNTROWS (Answers), Answers[ID_Answer] = VALUES (Filter1[ID_Answer])) > 0 ) Additional conditions to set the relationships with two tables in DAX only

Survey: The DAX Formula The two references to «Answers» work on separate instances of the same table IF ( COUNTROWS (VALUES (Filter1[ID_Answer])) = 1 && COUNTROWS (VALUES (Filter2[ID_Answer])) = 1, CALCULATE ( COUNTROWS (Customers), FILTER ( Customers, CALCULATE ( COUNTROWS (Answers), Answers[ID_Answer] = VALUES (Filter2[ID_Answer])) > 0 ) ), FILTER ( Customers, CALCULATE ( COUNTROWS (Answers), Answers[ID_Answer] = VALUES (Filter1[ID_Answer])) > 0 )

Survey - Conclusions  Very powerful data model  Very compact Duplicates only dimensions Different from the same pattern in Multidimensional  Handles any number of questions We have shown two But it is not a limit  Interesting topics Fact table as the bridge Relationships set in DAX Can be queried with a simple PivotTable

Basket Analysis

Basket Analysis: The Scenario Of all the customers who have bought a Mountain Bike, how many have never bought a mountain tire tube?

Basket Analysis in SQL Two iterations over the fact table needed SELECT COUNT (DISTINCT A.CustomerID) FROM FactSales A INNER JOIN FactSales B ON A.CustomerID = B.CustomerID WHERE A.ProductModel = ‘MOUNTAIN TIRE TUBE' AND A.Year <= 2004 AND B.ProductModel = ‘MOUNTAIN-100' AND B.Year <= 2004

Look the query plan … This is the fact table … Do you really like to self-join it? This is the fact table … Do you really like to self-join it?

Basket Analysis: The Data Model Of all the customers who have bought a Mountain Bike, how many have never bought a mountain tire tube? We can filter «Mountain Tire Tube» with this table but… where do we filter «Mountain Bike»?

Basket Analysis: The Data Model 1° filter: «Mountain Tire Tube» 2° filter: «Mountain Bike» No relationships

Basket Analysis- Demo

The Final Result 1° filter: «Mountain Bike» 2° filter: «Mountain Tire Tube» HavingProduct = Bought Both NotHavingProduct = Bought Bike, No Tire HavingProduct = Bought Both NotHavingProduct = Bought Bike, No Tire

HavingProduct := CALCULATE ( COUNTROWS (DISTINCT (FactSales[CustomerKey])), FILTER ( ALL (DimTime), DimTime[TimeKey] 0 ) )

Count the number of customers…

HavingProduct := CALCULATE ( COUNTROWS (DISTINCT (FactSales[CustomerKey])), FILTER ( ALL (DimTime), DimTime[TimeKey] 0 ) ) In the period of time before the end of the currently selected filter

HavingProduct := CALCULATE ( COUNTROWS (DISTINCT (FactSales[CustomerKey])), FILTER ( ALL (DimTime), DimTime[TimeKey] 0 ) ) This filter needs a bit of attention…

FILTER ( DimCustomer, SUMX ( ProductFilter, CALCULATE ( COUNTROWS (FactSales), ALL (FactSales), FactSales[CustomerKey] = EARLIER (DimCustomer[CustomerKey]), FactSales[ProductKey] = EARLIER (ProductFilter[ProductKey]), FILTER ( ALL (DimTime), DimTime[TimeKey] 0 ) ) Only the customers

FILTER ( DimCustomer, SUMX ( ProductFilter, CALCULATE ( COUNTROWS (FactSales), ALL (FactSales), FactSales[CustomerKey] = EARLIER (DimCustomer[CustomerKey]), FactSales[ProductKey] = EARLIER (ProductFilter[ProductKey]), FILTER ( ALL (DimTime), DimTime[TimeKey] 0 ) ) Only the customers Where, for each filtered product

FILTER ( DimCustomer, SUMX ( ProductFilter, CALCULATE ( COUNTROWS (FactSales), ALL (FactSales), FactSales[CustomerKey] = EARLIER (DimCustomer[CustomerKey]), FactSales[ProductKey] = EARLIER (ProductFilter[ProductKey]), FILTER ( ALL (DimTime), DimTime[TimeKey] 0 ) ) Only the customers Where, for each filtered product The number of rows in the fact table, for that product The number of rows in the fact table, for that product

FILTER ( DimCustomer, SUMX ( ProductFilter, CALCULATE ( COUNTROWS (FactSales), ALL (FactSales), FactSales[CustomerKey] = EARLIER (DimCustomer[CustomerKey]), FactSales[ProductKey] = EARLIER (ProductFilter[ProductKey]), FILTER ( ALL (DimTime), DimTime[TimeKey] 0 ) ) Only the customers Where, for each filtered product The number of rows in the fact table, for that product The number of rows in the fact table, for that product Is greater than zero i.e. there exists at least one sale

FILTER ( DimCustomer, SUMX ( ProductFilter, CALCULATE ( COUNTROWS (FactSales), ALL (FactSales), FactSales[CustomerKey] = EARLIER (DimCustomer[CustomerKey]), FactSales[ProductKey] = EARLIER (ProductFilter[ProductKey]), FILTER ( ALL (DimTime), DimTime[TimeKey] 0 ) ) Only the customers Where, for each filtered product The number of rows in the fact table, for that product The number of rows in the fact table, for that product Is greater than zero i.e. there exists at least one sale These conditions are needed because there aren’t relationships between Filter and the fact table. Thus, we add them manually in DAX

Ok, but «NotHavingProduct»? NotHavingProduct := CALCULATE ( COUNTROWS ( FILTER ( DimCustomer, CALCULATE (COUNTROWS (FactSales)) = 0 ) ), FILTER ( ALL (DimTime), DimTime[TimeKey] 0 ) ) Instead of: COUNTROWS ( DISTINCT (FactSales[CustomerKey]) ) Instead of: COUNTROWS ( DISTINCT (FactSales[CustomerKey]) )

Nice, but where are the M2M? NotHavingProduct := CALCULATE ( COUNTROWS ( FILTER ( DimCustomer, CALCULATE (COUNTROWS (FactSales)) = 0 ) ), FILTER ( ALL (DimTime), DimTime[TimeKey] 0 ) ) 1° M2M Pattern 2° M2M Pattern

Denali new features  Multiple relationships between tables Role Dimensions Role Keys  Active / Inactive relationships  USERELATIONSHIP New filter function Selects a model relationship to use in calculations  These additions make the data model much easier

Inactive Relationships in Denali Inactive Relationships

The Formula with Denali COUNTROWS ( FILTER ( DimCustomer, CALCULATE ( COUNTROWS (FactSales), USERELATIONSHIP ( Factsales[ProductKey], ProductFilter[ProductKey]) ) > 0 && CALCULATE ( COUNTROWS (FactSales), USERELATIONSHIP ( Factsales[ProductKey], DimProduct[ProductKey]) ) = 0 ) )

The formula with Denali  It is not Shorter Faster (but it is too early to evaluate)  It is Much clearer Shows the real intentions of the author  Key Point It can be written by Excel Users It is straightforward, no complex and geeky logic  In this example, Denali moves toward the users

Basket Analysis: Conclusions  Modeled with the many to many pattern Fact table as a bridge one Relationships set in DAX  Very Powerful  Search for missing values «Easy» as searching for existing ones

No support for M2M in Tabular  Facts Multidimensional handles M2M relationships. Tabular does not (apparently) DAX is the key to handle them  Many interesting models can be created leveraging DAX and the many to many patterns

Conclusions  Many To Many Relationships Can be handled in Tabular Even if not very easily  Very Powerful Cascading Many To Many Survey Data Model Basket Analysis  Are they worth some hours of study?

sqlbi.com

Links  SQLBI Website  PowerPivot Workshop  Alberto Ferrari blog  Marco Russo blog For any question contact us at

Coming up… # SQLBITS SpeakerTitleRoom Quest Trivia Quiz: Test Your SQL Server and IT Knowledge and Win Prizes Aintree SQL Server Community SQL Server Community presents : The A to Z of SQL NuggetsLancaster SQLSentry Real Time and Historical Performance Troubleshooting with SQL Sentry Pearce Attunity Data Replication Redefined – best practices for replicating data to SQL Server Empire Fusion-ioIs it a bird? Is it a plane?Derby