2. 2 Optimizing Query Performance in Microsoft® SQL Server® 2008 Analysis Services Rob Hawthorne – rob@prophesy.co.nz Managing Director Prophesy Ltd Session Code: BIN 401

3. Specialist BI and Data Management Company Focused solutions within the entire Microsoft BI stack Kimball Methodology for building Data Warehouse/Business Intelligence solutions Working with Government, Finance and ISV’s in NZ Specialist consultants who are recognised in their field True work-life balance Introduction Prophesy Limited

4. Managing Director for Prophesy – BI & Database Been working with SQL Server since early versions 6.0 and above Focused on high availability and scalability Author of “Microsoft SQL Server 2000 Database Development from Scratch” Blatant Product Placement Introduction Rob Hawthorne & wife carrying champ in 2004 & 2005!

5. Agenda Query Execution Overview Analysing Query Performance Optimization Techniques Conclusion

6. 6 Query Execution Phases Query Execution Overview Parsing the Query Axis Population Serialize Axes Cell Data Computation Serialisation of Cells

7. 7 Example Query Execution Overview Consider the Query How is the Query Evaluated? WITH MEMBER Measures.ContributionToParent AS ‘measures.Sales/(measures.Sales, Product.Currentmember.parent)’ SELECT Product.[Product Family].members ON COLUMNS, Customer.Country.members ON ROWS FROM Sales WHEREmeasures.ContributionToParent

8. 8 Query Execution Architecture Client Application Analysis Services MDX query Query Parser Formula Engine Populate axes Compute cell data s FE Caches Subcube operations Calculation Engine Storage Engine s SE Caches Partition Data Query SE Evaluation Engine Serialize results

9. 9 Visualising the Query Execution Rob Hawthorne – rob@prophesy.co.nz Managing Director Prophesy Ltd

10. 10 DrinkFoodNon-Consumable Canada Mexico USA WITH MEMBER Measures.ContributionToParent AS ‘measures.Sales/(measures.Sales, Product.Currentmember.parent)’ SELECT Product.[Product Family].members ON COLUMNS, Customer.Country.members ON ROWS FROM Sales WHERE measures.ContributionToParent WITH MEMBER Measures.ContributionToParent AS ‘measures.Sales/(measures.Sales, Product.Currentmember.parent)’ SELECT Product.[Product Family].members ON COLUMNS, Customer.Country.members ON ROWS FROM Sales WHERE measures.ContributionToParent WITH MEMBER Measures.ContributionToParent AS ‘measures.Sales/(measures.Sales, Product.Currentmember.parent)’ SELECT Product.[Product Family].members ON COLUMNS, Customer.Country.members ON ROWS FROM Sales WHERE measures.ContributionToParent Axis Population Query Execution Overview

11. 11 DrinkFoodNon-Consumable Canada(null)(null)(null) Mexico(null)(null)(null) USA9.22%71.95%18.83% Cell Data Computation Query Execution Overview

12. 12 DrinkFoodNon-Consumable Canada(null) Mexico(null) USA9.22%71.95%18.83% DrinkFoodNon-Consumable Canada(null) Mexico(null) USA $24,597.00 $191,940.00 $50,236.00 All Products Canada (null) Mexico (null) USA $266,773.00 Measures.ContributionToParent measures.[Unit Sales](Measures.Sales, Product.Currentmember.Parent) / = Cell-By-Cell Computation Query Execution Overview

13. 13 Disadvantages Query Execution Overview Calculations have to be analyzed for each cell The same navigation for each cell repeated Calculating null values when we should know ahead of time they are null

14. 14 Solution: Subspace Computation Query Execution Overview Only calculate non-null* values Perform the cell navigation (e.g.,.prevmember,.parent) just once instead of for every cell Navigate for the entire space * Null is the most common default (sparse) value

15. 15 Reconsider the query: WITH MEMBER Measures.ContributionToParent AS ‘measures.Sales/(measures.Sales, Product.Currentmember.parent)’ SELECT Product.[Product Family].members ON COLUMNS, Customer.Country.members ON ROWS FROM Sales WHEREmeasures.ContributionToParent We know that: When we divide a/b, the result is only non-null when a is not null so we only need to compute this expression where a is not null The parent of every [Product].[Product Family] is [Product].[All] - a constant value. We don’t have to navigate to this cell over and over for each member in Product.[Product Family].members only to retrieve the same value. We also know that storage engine requests only return non-null data Subspace Computation Query Execution Overview

16. 16 CountryProductMeasureValue USAAll ProductsSales $266,773.00 CountryProductMeasureValue USADrinkSales $24,597.00 USAFoodSales $191,940.00 USA Non- ConsumableSales $50,236.00 CountryProductMeasureValue USADrinkContribution to Parent 9.22% USAFoodContribution to Parent 71.95% USANon-ConsumableContribution to Parent 18.83% DrinkFoodNon-Consumable Canada(null) Mexico(null) USA9.22%71.95%18.83% 2) Perform the computation for the non-null values: only 3 computations instead of 9… 3) …and everything else is null 1. Retrieve non-null values from storage engine Subspace Computation Query Execution Overview

17. 17 Block Computation Illustration Evaluate an expression when it needs to Squeeze out the “empty” space vs Qty * Sales Price Qty * Sales Price

18. 18 Sample query: with member Customer.Country.USAVsCanada as [Customer].[Country].[USA] - [Customer].[Country].[Canada] select [Measures].[Store Sales] on 0, [Product].[Product Category].[Product Category].members on 1 from sales where USAVsCanada cell properties value Scalar Evaluation Query Execution Overview

19. 19 - USAVsCanada USACanada Scalar Evaluation Query Execution Overview

20. 20 [Country].[USAVsCanada] [Product Category].* [Measures].[Store Sales] [Country].[USA] [Product Category].* [Measures].[Store Sales] [Country].[Canada] [Product Category].* [Measures].[Store Sales] - USAVsCanada USACanada Scalar Evaluation Query Execution Overview

21. 21 CountryCategoryStore Sales USAVsCanadaBeer and Wine27917.96 USAVsCanadaBread30465.94 USAVsCanadaDrinks34394.19 USAVsCanadaHot Beverages-7734.99 …….... CountryCategoryStore Sales CanadaDrinks2908.36 CanadaBread2572.84 CanadaHot Beverages7734.99 CanadaCandy2455.34 …….... CountryCategoryStore Sales USABeer and Wine27917.96 USABread33038.78 USADrinks37302.55 USACandy27816.8 …….... - [Country].[USAVsCanada] [Product Category].* [Measures].[Store Sales] [Country].[USA] [Product Category].* [Measures].[Store Sales] [Country].[Canada] [Product Category].* [Measures].[Store Sales] Might be out of order (no duplicate rows, though * ) Correspondig rows could be in different positions New (non-corresponding) rows might appear… … while others might disappear (no null valued rows either * ) Scalar Evaluation Query Execution Overview

22. 22 CountryCategoryStore Sales USABeer and Wine27917.96 USABread33038.78 USADrinks37302.55 USACandy27816.8 …….... CountryCategoryStore Sales USAVsCanadaBeer and Wine27917.96 USAVsCanadaBread30465.94 USAVsCanadaDrinks34394.19 USAVsCanadaHot Beverages-7734.99 …….... CountryCategoryStore Sales CanadaDrinks2908.36 CanadaBread2572.84 CanadaHot Beverages7734.99 CanadaCandy2455.34 …….... [Country].[USA] [Product Category].* [Measures].[Store Sales] [Country].[Canada] [Product Category].* [Measures].[Store Sales] key iterate lookup & mark - [Country].[USAVsCanada] [Product Category].* [Measures].[Store Sales] Iterate non-marked and output negatives Scalar Evaluation Query Execution Overview

23. 23 CountryCategoryStore Sales USABeer and Wine27917.96 USABread33038.78 USADrinks37302.55 USACandy27816.8 …….... CountryCategoryStore Sales USAVsCanadaBeer and Wine27917.96 USAVsCanadaBread30465.94 USAVsCanadaDrinks34394.19 USAVsCanadaHot Beverages-7734.99 …….... CountryCategoryStore Sales CanadaDrinks2908.36 CanadaBread2572.84 CanadaHot Beverages7734.99 CanadaCandy2455.34 …….... - [Country].[USAVsCanada] [Product Category].* [Measures].[Store Sales] [Country].[USA] [Product Category].* [Measures].[Store Sales] [Country].[Canada] [Product Category].* [Measures].[Store Sales] key iterate lookup & mark Iterate non-marked and output negatives ? Scalar Evaluation Query Execution Overview

24. 24 CountryCategoryStore Sales USAVsCanadaBeer and Wine27917.96 USAVsCanadaBread33038.78 USAVsCanadaDrinks37302.55 USAVsCanadaCandy27816.8 …….... CountryCategoryStore Sales USAVsCanadaBeer and Wine27917.96 USAVsCanadaBread30465.94 USAVsCanadaDrinks34394.19 USAVsCanadaHot Beverages-7734.99 …….... CountryCategoryStore Sales USAVsCanadaDrinks2908.36 USAVsCanadaBread2572.84 USAVsCanadaHot Beverages7734.99 USAVsCanadaCandy2455.34 …….... - [Country].[USAVsCanada] [Product Category].* [Measures].[Store Sales] [Country].[USAVsCanada] [Product Category].* [Measures].[Store Sales] [Country].[USAVsCanada] [Product Category].* [Measures].[Store Sales] ? ? Scalar Evaluation Query Execution Overview

25. 25 CountryCategoryStore Sales USAVsCanadaBeer and Wine27917.96 USAVsCanadaBread33038.78 USAVsCanadaDrinks37302.55 USAVsCanadaCandy27816.8 …….... CountryCategoryStore Sales USAVsCanadaBeer and Wine27917.96 USAVsCanadaBread30465.94 USAVsCanadaDrinks34394.19 USAVsCanadaHot Beverages-7734.99 …….... CountryCategoryStore Sales USAVsCanadaDrinks2908.36 USAVsCanadaBread2572.84 USAVsCanadaHot Beverages7734.99 USAVsCanadaCandy2455.34 …….... - [Country].[USAVsCanada] [Product Category].* [Measures].[Store Sales] USA  USAVsCanada Canada  USAVsCanada Space Transforms Query Execution Overview

26. 26 CountryCategoryStore Sales CanadaDrinks2908.36 CanadaBread2572.84 CanadaHot Beverages7734.99 CanadaCandy2455.34 …….... CountryCategoryStore Sales USABeer and Wine27917.96 USABread33038.78 USADrinks37302.55 USACandy27816.8 …….... CountryCategoryStore Sales USAVsCanadaBeer and Wine27917.96 USAVsCanadaBread33038.78 USAVsCanadaDrinks37302.55 USAVsCanadaCandy27816.8 …….... CountryCategoryStore Sales USAVsCanadaDrinks2908.36 USAVsCanadaBread2572.84 USAVsCanadaHot Beverages7734.99 USAVsCanadaCandy2455.34 …….... - [Country].[USAVsCanada] [Product Category].* [Measures].[Store Sales] USA  USAVsCanada Canada  USAVsCanada [Country].[USA] [Product Category].* [Measures].[Store Sales] [Country].[Canada] [Product Category].* [Measures].[Store Sales] Space Transforms Query Execution Overview

27. 27 MDX Constructs And Subspace Computation Analysis Services 2005 Service Pack 2 Basic arithmetic operators: *, /, +, - Basic relational operators:, =, <>, = Static member references: e.g. [Customers].[Seattle] Static tuple references: e.g. ([Customers].[Seattle], [Time].[2007]) Basic unary operators: +, -, ~, numeric weights Semi-additive measures Basic aggregate functions (Sum, Min, Max, Aggregate) on: Static sets: e.g. Sum({USA, Canada}, Measures.Sales) Sets built from.members, PeriodsToDate, YTD/QTD/MTD, Crossjoin, Cousin, CurrentMember, range (“:” operator) and static sets Measures.CurrentMember Not CurrentMember on other dimensions IIF(cond, expr, NULL) IsEmpty Constant scalars: e.g. NULL, “Coke”, 5, 3.14

28. 28 MDX Constructs And Subspace Computation Analysis Services 2008 – Big Improvements! Extends architecture of the calculation engine to support a logical plan and physical plan Allows analysis and optimization of the logical plan More scalar operations: IS, MemberValue, Properties, Name, IsAncestor, IsLeaf, IsSibling CalculationPassValue, IIF, CASE Member functions: PrevMember, NextMember, Lag, Lead, FirstChild, LastChild, Parent, Ancestor, etc. Just this category enables subspace computation for a large number of calculation patterns! Set functions: Support subspace computation in limited situations – e.g. inside Aggregate Static sets and sets built from static sets can be detected to enable subspace computation for scalar functions: E.g. Aggregate( [Geography].[Country].Members ) or Aggregate( Descendants( [USA], 3 ) ) Lots of set functions have this type of support: Descendants, Children, Hierarchize, Members, etc. Some examples where subspace computation won’t work: Named sets – e.g. Aggregate( [NamedSet] ) will fall back to cell-by-cell mode “Dynamic” operations – e.g. [Date].[Fiscal].Lag( [Measures].[Count] ) Recursion – detecting possible recursion may revert to cell-by-cell

29. 29 NON EMPTY Improvements Analysis Services 2005 supported two modes for executing NON EMPTY queries: Naïve and Express Naïve iterates over all cells and checks if they are empty Express iterates over data and looks up tuples on the axes – sparseness of data makes this much faster Express algorithm worked only in limited cases: No calculations involved, only Storage Engine Calculated Measures with NON_EMPTY_BEHAVIOR defined properly In Analysis Services 2008, the Express algorithm will (usually) be used even if calculations are present – exceptions: Recursion in calcs Complex overlapping calculations

30. 30 Calculation Hints Analysis Services 2008 also adds hints to the MDX language: IIF(, [HINT EAGER | STRICT], [HINT EAGER | STRICT] ) Helps tune the execution plan to be: Eager – evaluate the then/else for the entire subspace Strict – evaluate the expression only for the cells that match the condition Significant performance issues with choosing the wrong plan Unfortunately, it is difficult to explain which hint to use

31. Agenda Query Execution Overview Analysing Query Performance Optimization Techniques Conclusion

32. 32 Three common scenarios Large SE request Several SE requests FE-intensive query Depends on execution plan chosen by FE Large SE request “Query Subcube” event with large duration “Reading data from partition” event  agg miss! Design more aggregations (UBO, agg manager) Effective partitioning Common Solutions for Slow Queries Analysing Query Performance

33. 33 Several SE requests Lots of “Query Subcube” events (cache miss) Use CREATE CACHE to prefetch data FE Intensive Query Single CPU pegged at 100% Use block functions/operators and avoid functions that cause cell-by-cell Not surprisingly, simpler MDX is often faster Query optimization techniques (next section) ScenarioLarge SE Request Several SE Requests FE Intensive Query SolutionPartitioning, aggregations CREATE CACHE Cell-by-cell  Block Optimizations: NEB, Auto-exists, Scope, MemberValue … Some Solutions for Slow Queries Analysing Query Performance

34. Agenda Query Execution Overview Analysing Query Performance Optimization Techniques Conclusion

35. 35 Most client tools (Excel, ProClarity) display non-empty results – eliminate members with no data With no calculations, non-empty is fast – just checks fact data With calculations, non-empty can be slow – requires evaluating formula for each cell Non_Empty_Behavior allows non-empty on calculations to evaluate calculations on a reduced space Essentially, it is telling the engine: This calculation is guaranteed to be null if the NEB expression is NULL Note: query processing hint – use with care! It is often used incorrectly  Create Member [Measures].[Internet Gross Profit] As [Internet Sales Amount] - [Internet Total Cost], Format_String = "Currency", Non_Empty_Behavior = [Internet Sales Amount]; Non_Empty_Behaviour Optimisation Techniques

36. 36 Use scopes instead of conditions such as Iif/Case Calculations inside scope apply only to the subspace Scopes are evaluated once statically Conditions are evaluated dynamically for each cell Create Member Measures.Sales As Iif(Currency.CurrentMember Is Currency.USD, Iif(Currency.CurrentMember Is Currency.USD, Measures.SalesUSD, Measures.SalesUSD * Measures.XRate); Measures.SalesUSD, Measures.SalesUSD * Measures.XRate); Create Member Measures.Sales As Null; Scope(Measures.Sales, Currency.Members); This = Measures.SalesUSD * Measures.XRate; This = Measures.SalesUSD * Measures.XRate; Scope(Currency.USA); Scope(Currency.USA); This = Measures.SalesUSD; This = Measures.SalesUSD; End Scope; End Scope; End Scope; Scope vs. Iif & Case Optimisation Techniques

37. 37 Attributes/hierarchies within a dimension are always existed together City.Seattle * State.Members returns {(Seattle, WA)} (Seattle, OR), (Seattle, CA) do not “exist” Exploit the power of auto-exists Use Exists/CrossJoin instead of.Properties—faster Requires attribute hierarchy enabled on member property Filter(Customer.Members, Customer.CurrentMember.Properties(“Gender”) = “Male”) Customer.CurrentMember.Properties(“Gender”) = “Male”) Exists(Customer.Members, Gender.[Male]) Auto-Exists vs..Properties Optimisation Techniques

38. 38 MemberValue – intrinsic member property introduced in Analysis Services 2005 Every member has a value (just like key and name) You can use MemberValue function to get a “typed” result Create Set RichCustomers As Filter(Customer.Members, Filter(Customer.Members, CInt(Customer.CurrentMember.Properties(“Salary”)) CInt(Customer.CurrentMember.Properties(“Salary”)) > 100,000); > 100,000); Create Set RichCustomers As Filter(Customer.Members, Filter(Customer.Members, Salary.MemberValue > 100,000); Salary.MemberValue > 100,000); MemberValue vs..Properties Optimisation Techniques

39. 39 Analysis Services 2005/2008 has built-in features for advanced scenarios Many-to-many dimensions Measure expressions Semi-additive measures Unary operators, custom rollups Use built-in features instead of custom MDX Measure expressions for M:1 currency conversion Semi-additive measures for snapshots, inventory Unary operators for complex rollups Built-In Features vs. MDX Calcs Optimisation Techniques

40. 40 Avoid using CalculationPassValue and Solve_Order Rely on auto recursion resolution using scopes and assignments Avoid LinkMember, StrToSet, StrToMember, StrToValue Avoid assigning values like 0, Null, “N/A”, “-” to empty cells Use Format_String instead for custom UI formatting of cells For an expression like “expr1 * expr2”, the expression spanning the larger cube space should be on the left Sales * ExchangeRate instead of ExchangeRate * Sales Sales * 1.15 instead of 1.15 * Sales Use Aggregate MDX function instead of Sum, Min, Max More at: http://sqljunkies.com/weblog/mosha http://sqlserveranalysisservices.com Miscellanous Optimisation Techniques

41. 41 NON_EMPTY_BEHAVIOUR Rob Hawthorne – rob@prophesy.co.nz Managing Director Prophesy Ltd

42. Agenda Query Execution Overview Analysing Query Performance Optimization Techniques Conclusion

43. 43 Conclusion Query evaluation is still largely a black box Profiler & Perfmon help Simplifying calculations/queries is often the best approach to investigate performance Analyzing & optimizing query performance Requires lots of hands on experience Understand and experiment with optimization techniques Lots of shared knowledge in the community, blogs, etc.

44. 44

45. 45 Resources www.microsoft.com/teched Tech·TalksTech·Ed Bloggers Live SimulcastsVirtual Labs http://microsoft.com/technet Evaluation licenses, pre-released products, and MORE! http://microsoft.com/msdn Developer’s Kit, Licenses, and MORE!

46. 46 Please complete an evaluation BIN401 Rob Hawthorne Rob Rocks! Tech.Ed is Awesome!

47. 47 © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.