Low Cost Load and Performance Testing

Example Test

Why Test? Performance is a direct reflection on your company Your client does not care about the ‘cause’

Whose Fault Is It? Cause can come from many sources ◦ Poor architecture ◦ Poor configuration ◦ Inadequate hardware ◦ Insufficient network bandwidth

Why Isn’t Performance Testing Done Takes time Takes production or production- like environment Takes a special skill set Takes specialized tools

Why Isn’t Performance Testing Done Can take lots of $$$$$! ◦ Proprietary tools – can cost 100’s of thousands of dollars to buy and maintain

So What Are We Supposed to Do? Option 1 – Do nothing Option 2 – Keep buying hardware Option 3 – Buy an expensive load and performance tool. Hire a tool expert to use it. Option 4 – Allocate in-house personnel to build a custom solution

So What Are We Supposed to Do? Option 5 – Pray

Low Cost Options Big tool… little price ◦ Utilize an industry recognized best-of- breed tool at a greatly reduced cost ◦ Pros  Access to functionality that normally costs lots of $$  Good for client server or web applications ◦ Cons  Proprietary languages. Need ‘tool guru’  Need hardware, bandwidth to run it

Low Cost Options Open Source ◦ 40+ options - JMeter, OpenSTA, WebLoad, Grinder ◦ Pros  Good price  Can extend and adapt to your needs  Good for simple tests ◦ Cons  Not as easy to use. Lacks functionality  Only for web based products  Need hardware, bandwidth to generate load

Low Cost Options Cloud Based ◦ Rapidly growing offering  BrowserMob, LoadStorm ◦ Pros  Low price  Can easily scale to thousands of users  Can completely eliminate the need for hardware ◦ Cons  Early in the development phase  Only for web-based products  Analysis is slowly coming up to speed

Cost Comparison

How Do I Choose? How do I know what is best for me?

Understand Your True Needs Performance, Load, Stress, Reliability ◦ All unique areas of testing Performance - not to find functionality bugs, but to eliminate system bottlenecks and establish a baseline for future regression tests Takes a white-box approach. The system is inspected and monitored "from the inside out" and from a variety of angles. Measurements are taken and analyzed and tuning is done accordingly.

Understand Your True Needs Load - process of testing the system by feeding it the largest tasks it can operate with Goals expose bugs such as memory management bugs, memory leaks, buffer overflows, etc Ensure that the application meets the performance baseline established during performance testing.

Understand Your True Needs Stress - one primary objective, and that is to determine the load under which a system fails, and how it fails. The main purpose is to make sure that when the system fails, it recovers gracefully. i.e. – no locked sessions, no corrupted data, if load is removed does it return to normal working condition

Understand Your True Needs Reliability – determine application performance over a prolonged period of time. Load can vary between very high and very low The main purpose is to make sure that the system performance does not degrade over time. i.e. – slow memory leaks, stale sessions, other processes interfering with off hour activities

Understand Your True Needs Understand your user scenarios ◦ Analyze traffic patterns  Google Analytics, application logs ◦ User types  Administrative, buyer, seller, poster, responder, browser, etc… ◦ User activity  Downloading, uploading, purchasing, saving cart, submitting, ◦ User ‘load’  Mix of user type and activity. How many of each

Don’t Go It Alone May not do load and performance testing for months/years ◦ Difficult for most companies to have SME Reach out to colleagues, user groups, consultants ◦ Can help ‘reduce the noise’ ◦ Guide you through the process

Keep Customers Happy Performance is something customers notice when it is poor Don’t give customers a reason to look at other options

Questions/Discussion Points Questions? Lawrence Nuanez – Senior Consultant – ProtoTest – ext 109