Establishing an effective performance testing environment. Gordon McKeown TMF 2010

Agenda  Introductory tour of the issues.  Share our experiences & identify best practice.  In the spirit of the TMF... You will drive the the session! TMF "Creating an effective performance testing environment" 2 27/1/2010

What do we mean by "Environment“ ?  A wide scope. But not too wide!  In  "Systems"  Network  Tool  Load Injectors  System Under Test (SUT)  Software installation & configuration  Test data  Out  People (except for responsibilities / roles )  Tool selection TMF "Creating an effective performance testing environment" 3 27/1/2010

Performance test elements 27/1/2010 TMF "Creating an effective performance testing environment" 4

Desirable characteristics  Few impediments to tester activities.  Low management & bureaucratic overheads.  “Realistic tests” are possible.  SUT as close as possible to production.  Experimentation is possible.  Isolation  Reproducible tests  Reduced risk of impact on corporate ICT 27/1/2010 TMF "Creating an effective performance testing environment" 5

Some of the issues  Who controls the test infrastructure?  Security standards.  Test data.  Relationship to "production“.  Resource sharing.  Technical innovations.  Any other major areas? 27/1/2010 TMF "Creating an effective performance testing environment" 6

Should the test team declare independence? 27/1/2010 TMF "Creating an effective performance testing environment" 7

Who should control the environment?  " Lack of environmental control by testers is a common, serious impediment to testing efficiency." That is my experience. What is yours?  Testing is an exploratory activity, we often don't know what the next change should be.  Is it easier if load testing is integrated with development?  Are things worse in organisations conducting infrequent tests? E.g. medium sized organisation with limited IT capability who have commissioned a new system. 27/1/2010 TMF "Creating an effective performance testing environment" 8

Corporate security standards  We have the characteristics of hackers!  Denial of Service Attack.  Recording interactions.  "Man in the middle."  Looking inside systems.  Reverse engineering in extremis. 27/1/2010 TMF "Creating an effective performance testing environment" 9

Test data  Quantity & realism  Depending on DB design too little data can give unrealistically fast or slow timings.  An "unrealistic" database can distort results  Managing "real" data extracted from a live system.  Legal & ethical issues  Anonymizing 27/1/2010 TMF "Creating an effective performance testing environment" 10

Test's relationship to “Production"  Naive scaling  But who can afford full duplication?  Are sizing models fit for purpose?  Systems are not linear.  The "unknown unknowns" are a risk.  An example: licensing  One of our client's load tested against a near replica system. We recommended a subset of tests should be run against actual system (with test db) before going live. Discovered default server license for 50 users had not been upgraded to 300.  Significant but unforeseen configuration issues.  Changing configuration settings from scaled down replica to production: should this be tested? 27/1/2010 TMF "Creating an effective performance testing environment" 11

Resource sharing  Functional testing & UAT sharing resources with performance testing.  Mutual interference & frustration  Reproducible tests may not be possible.  Switching between configurations & databases.  Out of hours testing: a solution or another set of problems? 27/1/2010 TMF "Creating an effective performance testing environment" 12

Technical innovation  Cloud computing.  Software as service.  Virtualization.  What else might change the game? 27/1/2010 TMF "Creating an effective performance testing environment" 13