Too Slow?: Use VS2010 Profiling & Load Testing to Manage Performance Issues Benjamin Day

Consultant, Coach, Trainer Scrum.org Classes  Professional Scrum Developer (PSD)  Professional Scrum Foundations (PSF) TechEd, VSLive, DevTeach, O’Reilly OSCON Visual Studio Magazine, Redmond Developer News Microsoft MVP for Visual Studio ALM Team Foundation Server, TDD, Testing Best Practices, Silverlight, Windows Azure

Thanks. To my content reviewers  Jerri Chiu, Microsoft  Dennis Bass, Microsoft

The Agenda Basics of  Web Performance Tests  Load Tests  Load Test Rigs  Performance Explorer Existing application  Identify performance problems  Go from bad performance to good  ASP.NET & WCF

Disclaimer: I’m going to use the term “load testing” imprecisely…a lot.

Here we go.

The Purpose of Load Testing Identify the capabilities of your application Possible goals  Explore  Verify  Find the limits  Crush

Best Practice: The Obligatory “Eat Your Vegetables” Load test throughout the development cycle. Catch problems early Establish performance baseline Watch for performance trends Cheaper to fix early Plan for performance

Cheaper To Fix Early

Dose Of Reality You know you should work that way. You probably don’t. This talk is for you. You have problems. How do you figure out what’s going wrong?

Inspiration for this talk Real customer engagement Spent years developing a web application Released it Seriously grumpy customers Huge hardware ~20 simultaneous users Fixed it  simultaneous users  Reduced hardware

The Tools Visual Studio Ultimate Web Performance Tests Load Tests Load Test Rigs Visual Studio Premium Profiling Tools Performance Wizard

TOUR OF THE APPLICATION demo

Disaster Strikes. The application is deployed to production. Users are losing their minds.  Intermittent errors on the site.  Site is slow. Your boss is seriously angry.  “Fix it!”

The Problem. “No Repro”  “Works on my box.”  You don’t know what the errors are.  The site seems to work fine. Wild guess: it dies under load. You need to create some load.

The Plan. 1.Script user activity with Web Performance Tests 2.Simulate a mix of users using Load Tests 3.Simulate lots of users with a Load Test Rig 4.Reproduce the errors...hopefully 5.Find code problems with the Performance Wizard 6.Fix the code

Web Performance Tests.

The Plan. 1.Script user activity with Web Performance Tests 2.Simulate a mix of users using Load Tests 3.Simulate lots of users with a Load Test Rig 4.Reproduce the errors...hopefully 5.Find code problems with the Performance Wizard 6.Fix the code

What is a Web Performance Test? Test type in Visual Studio 2010 Ultimate Simulate a user using an ASP.NET application  HTTP traffic Building block of Load Tests

Tip: Web Tests & User Stories Web Tests should simulate a User Story Example:  As an administrator, I need to search for a person by username so that I can deactivate that person’s account.  As a customer, I want to order a pizza and pay with a credit card. Helps to organize your suite

CREATE A BASIC WEB TEST demo

More about Web Tests Collection of web requests Not for Test-First Development Recorded or coded Can be data-driven Make them do real stuff with  Validation Rules  Extraction Rules

Tip: Use Web Tests as a Quality Gate to QA Web Tests are great for “smoke testing” Run against a build to learn if the build is worth QA-ing Save QA a lot of time  (They like this.)

Parameterization Minimize hard coded values Use test context values in requests and validations Web server addresses for requests Double “squiggle bracket” syntax  {{variable_name}} Essential for data-driven web tests

Data-driven Web Tests Run web test once per row in data source Simulate multiple users doing *similar* actions Data source types  OLE DB, CSV, XML Parameterization Better tests, better coverage

WEB TESTS IN THE UNDERPERFORMING APPLICATION demo

Tip: Wait for stability Wait until your screens are reasonably stable Control name changes  broken web tests Control hierarchy changes  name changes  broken web tests

Load Tests.

The Plan. 1.Script user activity with Web Performance Tests 2.Simulate a mix of users using Load Tests 3.Simulate lots of users with a Load Test Rig 4.Reproduce the errors...hopefully 5.Find code problems with the Performance Wizard 6.Fix the code

What are Load Tests? Mix of Web Performance and Unit Tests Simulates a mix of users doing a mix of things Simulate  Network speeds  Different browsers  Varying user activity loads

Types of “Load Tests” Performance Test  Response time  Time lapses  Duration Load Test  Behavior under normal/peak workload  Actual workload Stress Test  Surfacing issues under extreme conditions and resource failures Source: J.D. Meier,

Why would you put a Unit Test in a Load Test? Helpful for testing WCF Cumbersome to simulate WCF via Web Test  (You’d have to re-invent the wheel.) Unit tests can re-use the WCF libraries  (Done and done.) Important for Silverlight and SOA

HEY KIDS, LET’S CREATE SOME LOAD TESTS! demo (…and then you all say, “Hooray!”)

Running Load Tests.

The Plan. 1.Script user activity with Web Performance Tests 2.Simulate a mix of users using Load Tests 3.Simulate lots of users with a Load Test Rig 4.Reproduce the errors...hopefully 5.Find code problems with the Performance Wizard 6.Fix the code

Two ways to run load tests. Generate load from Visual Studio 2010  Uses only 1 core/processor  Limited to 250 simulated users Generate load from a Test Rig  Far better test  Simulate lots more users

Load Test Agent Licensing Microsoft Visual Studio Team System 2010 Test Load Virtual User Pack Simulate up to 1000 virtual users Used to cost $$$

Visual Studio 2010 Load Test Feature Pack Used to cost ~$ per 1000 simulated users Free starting March 8, 2011 Visual Studio 2010 Ultimate with MSDN Simulate unlimited amount of load  (Limited only by your hardware.) en-us/products/2010-editions/load-test-virtual-user-pack

Load Test Rig & The Application Controller  Orchestrates  Licensing  Perf Mon Counters Agent (1..n)  Simulates users

Configure the Controller Install SQL Express  Optional  Stores collected test run data Install Visual Studio 2010 Test Controller Run the process as an ActiveDirectory domain user  Example: LoadTestController Add a Virtual User License Pack

Configure the Agent(s) Install the Agent Installation will ask which Controller Tip: Run the Agent service(s) using an Active Directory user  Example: LoadAgent

Permissions Performance Monitor Users  Controller user needs to be in this group  Typical servers: Controller, Agent(s), Web Server, Database server TeamTestControllerAdmins  Run tests on the rig  View results  Clean up the results database TeamTestControllerUsers  Run tests on the rig  View Results

The Plan. 1.Script user activity with Web Performance Tests 2.Simulate a mix of users using Load Tests 3.Simulate lots of users with a Load Test Rig 4.Reproduce the errors...hopefully 5.Find code problems with the Performance Wizard 6.Fix the code

RUN LOAD TESTS ON THE RIG demo

Hopefully, I remembered to mention the following… The “Manage Test Controllers…” dialog *.TestRunConfig settings Running the load tests Could spot the exceptions that were happening Fix the data access problems Demonstrated improved performance Compare two Load Test Runs

The Core Problem.

‘sp_who2’ says lots of open connections.

Performance Profiling

The Plan. 1.Script user activity with Web Performance Tests 2.Simulate a mix of users using Load Tests 3.Simulate lots of users with a Load Test Rig 4.Reproduce the errors...hopefully 5.Find code problems with the Performance Wizard 6.Fix the code

Profiling Tools Load Tests help you find issues in a system  “Surface” issues Profiling helps you find issues in the code Unit Tests, Web Tests, Load Tests  Repeatable  Good for targeted tuning System under “natural” load  Not repeatable  Looking for rough patterns

Profiling Methods Gathers information at intervals (clock cycles) No code modifications Use this for first pass explorations Sampling Modifies your code Gathers detailed timing and count information Instrumentation Multi-threaded code profiling Concurrency Object creation Garbage collection.NET Memory Interaction between your application and SQL Server via ADO.NET Tier Interaction

Load Test Performance Sessions Via Wizard Must be only Web Performance Tests Can’t profile via Load Test Rig ASP.NET must be running in x86 mode

LET’S GO PROFILE SOME STUFF. demo

The Plan. 1.Script user activity with Web Performance Tests 2.Simulate a mix of users using Load Tests 3.Simulate lots of users with a Load Test Rig 4.Reproduce the errors...hopefully 5.Find code problems with the Performance Wizard 6.Fix the code

Any last questions?

For More Information: Articles tinyurl.com/2ulvvvr

For More Information: Microsoft Links Visual Studio Performance Testing Quick Reference Guide Content Index for Web Tests & Load Tests

For More Information: Microsoft Bloggers VS Team Test Blog Site Ed Glas - Bill Barnett - Sean Lumley - Dennis Stone - Mike Taute - Neelesh Kamkolkar -

Thank you. blog.benday.com | |

Additional information that I practically never have time to cover.

Miscellaneous Tips.

Include Load Tests In Your Nightly Build Create a Test List (*.vsmdi) Include the Test List in the build Captures trend data at 24 hours intervals Compare the load test data with Excel

Static Code Analysis / FxCop Help you to find issues in your code Show you IDispose errors Rules are configurable Use your judgment about which are relevant for your team

Avoid Static Methods & CA1822 Static Code Analysis Rule CA1822 is evil. Sacrifices maintainability for perfomance. Blog post: “Static Methods Are A Code Smell”

Only optimize known performance problems. You’d be surprised by what *ISN’T* a performance problem Don’t spend a ton of time coding fancy solutions to performance problems you *THINK* you might have Find and fix *REAL* problems

Tip: WCF Message Size Data structure formatting can make a huge difference Example: Banks and the States they operate in Option 1: Dictionary  Key = Bank Id  Value = State Abbreviation Option 2: Dictionary  Key = State Abbreviation  Value = Array of Bank Id’s Option 2 was approximately 90% smaller

Tip: Add Performance Counters Create Performance Counters (perfmon) in your app Do this early in the development cycle More directed profiling of your app during Load Tests Good for managing your app when it’s in production  (Operations people love this.)

Using Performance Sessions on x64 You can’t profile x64 IIS processes from the IDE Option #1: Profile from the command line  Start ASP.NET profiling using VSPerfAspNetCmd.exe  Run the web test Option #2: Run the ASP.NET process in x86 mode

How To Run ASP.NET in x86 Mode Set “Enable 32-bit Applications” to true on your application’s AppPool