Load Testing Analysis Services 30 th march 2012 Bob Duffy Database Architect Prodata SQL Centre of Excellence

Speaker Profile – Bob Duffy Database Architect at Prodata SQL Centre Excellence, DublinDatabase Architect at Prodata SQL Centre Excellence, Dublin One of about 25 MCA for SQL Server globally (aka SQL Ranger)One of about 25 MCA for SQL Server globally (aka SQL Ranger) SQL Server MVP, 2009+SQL Server MVP, MCM on SQL 2005 and 2008MCM on SQL 2005 and years in database sector, 250+ projects20 years in database sector, 250+ projects Senior SQL Consultant with Microsoft Senior SQL Consultant with Microsoft Worked on many 1-5TB data warehouse and SSAS projectsWorked on many 1-5TB data warehouse and SSAS projects Blog

Agenda The Basics and Approach The Basics and Approach Tools Available Tools Available Creating Unit Tests Creating Unit Tests Creating Load Tests Creating Load Tests Case Study Case Study Lessons Learned Lessons Learned

What’s a Load Test A test event to verify scalability goals Typically Comprises of Unit Test – one or more MDX queries Load Test harness – execute unit tests to simulate users Analysis Tools – capture and compare performance Often you have a “baseline” and subsequent tests

Typical Goals / Reasons for Load Tests Throughput (not speed) Peak Users Queries Serviced Average Query Time Typical Comparison Scenarios Upgrade from 2005 New HW Platform must be 4 x faster No Scenario – Performance Tuning MDX Tuning Regression Testing (Performance and Functional) Usually Constraints Query Response Time must be under 3 seconds CPU must be no more than 80% for our 50 users

Case Study – Retail We have 500 stores reporting each first of the week/month Need to support 300 concurrent users Need response time for operational product report under 1 minute We currently have “black Monday” How many users can we support now ? How much faster will 2008 make things ? What hardware would we need to support 2,000 users ? How much faster would some mdx tuning make things?

Creating Unit Tests Can be Single Statement or Scenario Free Tools XMLA from command prompt Visual Studio Unit Test (AS Sim) Performance Workbench

Introducing VS 2010 Test Suite Requires VS 2010 Test Edition or Ultimate Facilitates Creation and running of Unit Tests

The Dreaded Parameter Issue Reporting Services et al use OLEDB Parameters Profiler Exposes a ReportParameter Column with values Often we just simulate parameters. Eg use a set of test resellers and a test date

How to Capture a Workload If you know your workload Create Unit Tests and Scenarios Otherwise you may need to profile workload Eg Capture a Trace Use XMLA and ASCMD.exe to start trace Requires a XML trace definition QueryBegin and QueryEnd events Specific Columns, especially TextData Be careful of “Parameters” Use the “RequestParameter” column on QueryBegin to get this Use Profiler to import into data table or replay

Load Tools – What do They do Replay Scenarios or Unit Tests Usually Multi-Threaded Sometimes distributed options Capture perfmon counters (for tuning) Chart Interesting Performance Counters Simulate Users using think times Store Results Report on Results Regression Analysis Functional Testing See

Introducing Visual Studio Load Test

Determining Max user Load

Case Studio. Retail Sales Tests CPU Bound. Maxes Out at only 15 users. Only 8 old cores - need more CPU or tuning to reduce CPU

Test 1 – Scale Up to 48 Core Without “Engine Tuning” worse performance! See “Memory Thrashing” in Pages/Sec 1050=>904 queries in 5 minutes

Tuning for Load Tests – INI Settings Only Adjust until CPU is 80% or more on single NUMA Node SettingsDefaultWhen to Tune MemoryHeapType HeapTypeForObjects always 0 – always ThreadPool\Query\MaxThreads10*When you get Queuing, and Idle threads are 0 ThreadPool\Process\MaxThreads64*When you get Queuing, and Idle threads are 0 Co-OrdinatorExecuteMode-4When Queuing, and Idle threads > 0 CoordinatorQueryBalancingFactor CoordinatorQueryBoostPriorityLevel 3 1 Only adjust if blocking of small queries 0 PreAllocate / Lock Pages in Memory0Mostly on 2003 Prefetch10 (most of the time) 2008 settings

2. Engine Tuning – Heap Type, MaxThreads After adjusting Heap and MaxThreads 1050=>2,951 queries BUT still CPU only 23% avg on SSAS Server ;-(

Case Study Part II Unanswered Questions How much faster was SQL 2008 ? Why can we only get 23% of CPU on new hardware ?

Initial 2005 v 2008 Comparison Same Hardware (24 core) 2005 was 8% faster on same HW. Or is it ? Chart below is on ONE thread

2005 v 2008 on 12 threads With 12 threads 2008R2 was slower 30% slower for our customer and 3% in my sample

A Thread Abuse Query

2005 Sample Query

2008r2 – sample Query

2012 Profile

Few Weeks later COD Hotfix 2008 Now 10% faster for test case

What we should have looked for Monitor Processing Thread “Busy”

Why only 23% CPU on a 48 core box? 2 Reasons: VS2010 configuration and NUMA 1) VS 2012 Feature Pack and Scale Out VS 2010 Agent throttled to only 1 CPU by default !!

NUMA Issues Spotting NUMA Issues Use coreinfo from sysinternals or just plain test it:

Lessons Learned (1/3) Make sure you know your goals Tuning of the Engine / HW Determine Max users 2005 =>2008=>2012 performance or functional testing Benchmarking environment Don’t be greedy in tests. One scenario at a time. On 2005/2008 you will need to tune ini file Watch for key concurrency issues Memory thrashing Thread starvation CPU saturation Test with and without NUMA Clear both database and file system cache Find which queries are slower on 2008

More Lessons Learned (2/3) Scale out may be require sooner than you think To end the case study An 80 core server achieved 500 query throughput How much did we achieve with a four node NLB? 3,250 (over 6x)

Final Lesson. Pick your hardware 80 core servers a no-no with SSAS! My Blog with Suggestions Services-(10GB-1TB-size).aspx Services-(10GB-1TB-size).aspx

Thank You!

Bob Duffy Blogs Scaling AdventureWorks for Load Testing BI-Testing-(2-of-2).aspx BI-Testing-(2-of-2).aspx Selecting HW for Analysis Services Services-(10GB-1TB-size).aspx Services-(10GB-1TB-size).aspx Load Testing Tools for Analysis Services Services.aspx Services.aspx

References SSAS Load Testing Best Practices ASLoadSim, ASQueryGen, LoadTestReports, ASCMD stress testing scripts AS Performance Workbench Creating a in SSAS Creating a Server Side Trace in SSAS SSAS 2008 Performance tuning

Bonus– Tuning of TCP/IP Stack When the number of connections to the server was 90 or greater, we experienced stability issues and “thrashing” of Processor utilisation. This was resolved by application of the following KB Article