1. Load Testing Analysis Services 30 th march 2012 Bob Duffy Database Architect Prodata SQL Centre of Excellence

2. Speaker Profile – Bob Duffy Database Architect at Prodata SQL Centre Excellence, DublinDatabase Architect at Prodata SQL Centre Excellence, Dublin One of about 25 MCA for SQL Server globally (aka SQL Ranger)One of about 25 MCA for SQL Server globally (aka SQL Ranger) SQL Server MVP, 2009+SQL Server MVP, 2009+ MCM on SQL 2005 and 2008MCM on SQL 2005 and 2008 20 years in database sector, 250+ projects20 years in database sector, 250+ projects Senior SQL Consultant with Microsoft 2005-2008Senior SQL Consultant with Microsoft 2005-2008 Worked on many 1-5TB data warehouse and SSAS projectsWorked on many 1-5TB data warehouse and SSAS projects Blog http://blogs.prodata.ie/bobBlog http://blogs.prodata.ie/bobhttp://blogs.prodata.ie/bob

3. Agenda The Basics and Approach The Basics and Approach Tools Available Tools Available Creating Unit Tests Creating Unit Tests Creating Load Tests Creating Load Tests Case Study Case Study Lessons Learned Lessons Learned

4. What’s a Load Test A test event to verify scalability goals Typically Comprises of Unit Test – one or more MDX queries Load Test harness – execute unit tests to simulate users Analysis Tools – capture and compare performance Often you have a “baseline” and subsequent tests

5. Typical Goals / Reasons for Load Tests Throughput (not speed) Peak Users Queries Serviced Average Query Time Typical Comparison Scenarios Upgrade from 2005 New HW Platform must be 4 x faster No Scenario – Performance Tuning MDX Tuning Regression Testing (Performance and Functional) Usually Constraints Query Response Time must be under 3 seconds CPU must be no more than 80% for our 50 users

6. Case Study – Retail We have 500 stores reporting each first of the week/month Need to support 300 concurrent users Need response time for operational product report under 1 minute We currently have “black Monday” How many users can we support now ? How much faster will 2008 make things ? What hardware would we need to support 2,000 users ? How much faster would some mdx tuning make things?

7. Creating Unit Tests Can be Single Statement or Scenario Free Tools XMLA from command prompt Visual Studio Unit Test (AS Sim) Performance Workbench

8. Introducing VS 2010 Test Suite Requires VS 2010 Test Edition or Ultimate Facilitates Creation and running of Unit Tests

9. The Dreaded Parameter Issue Reporting Services et al use OLEDB Parameters Profiler Exposes a ReportParameter Column with values Often we just simulate parameters. Eg use a set of test resellers and a test date

10. How to Capture a Workload If you know your workload Create Unit Tests and Scenarios Otherwise you may need to profile workload Eg Capture a Trace Use XMLA and ASCMD.exe to start trace Requires a XML trace definition QueryBegin and QueryEnd events Specific Columns, especially TextData Be careful of “Parameters” Use the “RequestParameter” column on QueryBegin to get this Use Profiler to import into data table or replay

11. Load Tools – What do They do Replay Scenarios or Unit Tests Usually Multi-Threaded Sometimes distributed options Capture perfmon counters (for tuning) Chart Interesting Performance Counters Simulate Users using think times Store Results Report on Results Regression Analysis Functional Testing See http://blogs.prodata.ie/post/Load-Test-Tools-for-Analysis-Services.aspx http://blogs.prodata.ie/post/Load-Test-Tools-for-Analysis-Services.aspx

12. Introducing Visual Studio Load Test

13. Determining Max user Load

14. Case Studio. Retail Sales Tests CPU Bound. Maxes Out at only 15 users. Only 8 old cores - need more CPU or tuning to reduce CPU

15. Test 1 – Scale Up to 48 Core Without “Engine Tuning” worse performance! See “Memory Thrashing” in Pages/Sec 1050=>904 queries in 5 minutes

16. Tuning for Load Tests – INI Settings Only Adjust until CPU is 80% or more on single NUMA Node SettingsDefaultWhen to Tune MemoryHeapType HeapTypeForObjects 1111 2 - always 0 – always ThreadPool\Query\MaxThreads10*When you get Queuing, and Idle threads are 0 ThreadPool\Process\MaxThreads64*When you get Queuing, and Idle threads are 0 Co-OrdinatorExecuteMode-4When Queuing, and Idle threads > 0 CoordinatorQueryBalancingFactor CoordinatorQueryBoostPriorityLevel 3 1 Only adjust if blocking of small queries 0 PreAllocate / Lock Pages in Memory0Mostly on 2003 Prefetch10 (most of the time) 2008 settings

17. 2. Engine Tuning – Heap Type, MaxThreads After adjusting Heap and MaxThreads 1050=>2,951 queries BUT still CPU only 23% avg on SSAS Server ;-(

18. Case Study Part II Unanswered Questions How much faster was SQL 2008 ? Why can we only get 23% of CPU on new hardware ?

19. Initial 2005 v 2008 Comparison Same Hardware (24 core) 2005 was 8% faster on same HW. Or is it ? Chart below is on ONE thread

20. 2005 v 2008 on 12 threads With 12 threads 2008R2 was slower 30% slower for our customer and 3% in my sample

21. A Thread Abuse Query

22. 2005 Sample Query

23. 2008r2 – sample Query

24. 2012 Profile

25. Few Weeks later COD Hotfix 2008 Now 10% faster for test case

26. What we should have looked for Monitor Processing Thread “Busy”

27. Why only 23% CPU on a 48 core box? 2 Reasons: VS2010 configuration and NUMA 1) VS 2012 Feature Pack and Scale Out VS 2010 Agent throttled to only 1 CPU by default !! http://msdn.microsoft.com/en-us/library/ms243155.aspx

28. NUMA Issues Spotting NUMA Issues Use coreinfo from sysinternals or just plain test it:

29. Lessons Learned (1/3) Make sure you know your goals Tuning of the Engine / HW Determine Max users 2005 =>2008=>2012 performance or functional testing Benchmarking environment Don’t be greedy in tests. One scenario at a time. On 2005/2008 you will need to tune ini file Watch for key concurrency issues Memory thrashing Thread starvation CPU saturation Test with and without NUMA Clear both database and file system cache Find which queries are slower on 2008

30. More Lessons Learned (2/3) Scale out may be require sooner than you think To end the case study An 80 core server achieved 500 query throughput How much did we achieve with a four node NLB? 3,250 (over 6x)

31. Final Lesson. Pick your hardware 80 core servers a no-no with SSAS! My Blog with Suggestions http://blogs.prodata.ie/post/Selecting-Hardware-for-Analysis- Services-(10GB-1TB-size).aspx http://blogs.prodata.ie/post/Selecting-Hardware-for-Analysis- Services-(10GB-1TB-size).aspx

32. Thank You!

33. Bob Duffy Blogs Scaling AdventureWorks for Load Testing http://blogs.prodata.ie/post/Scaling-AdventureworksDW-Cube-for- BI-Testing-(2-of-2).aspx http://blogs.prodata.ie/post/Scaling-AdventureworksDW-Cube-for- BI-Testing-(2-of-2).aspx Selecting HW for Analysis Services http://blogs.prodata.ie/post/Selecting-Hardware-for-Analysis- Services-(10GB-1TB-size).aspx http://blogs.prodata.ie/post/Selecting-Hardware-for-Analysis- Services-(10GB-1TB-size).aspx Load Testing Tools for Analysis Services http://blogs.prodata.ie/post/Load-Test-Tools-for-Analysis- Services.aspx http://blogs.prodata.ie/post/Load-Test-Tools-for-Analysis- Services.aspx

34. References SSAS Load Testing Best Practices http://sqlsrvanalysissrvcs.codeplex.com/releases/view/3505 ASLoadSim, ASQueryGen, LoadTestReports, ASCMD stress testing scripts http://sqlsrvanalysissrvcs.codeplex.com/releases/view/22769 AS Performance Workbench http://asperfwb.codeplex.com/ Creating a in SSAS Creating a Server Side Trace in SSAS http://www.bp-msbi.com/2012/02/counting-number-of-queries-executed-in-ssas/ SSAS 2008 Performance tuning http://www.microsoft.com/download/en/details.aspx?id=17303

35. Bonus– Tuning of TCP/IP Stack When the number of connections to the server was 90 or greater, we experienced stability issues and “thrashing” of Processor utilisation. This was resolved by application of the following KB Article http://support.microsoft.com/kb/951037 http://support.microsoft.com/kb/951037