UKTMF 27 th January 2010 Non-Functional Testing1 Non-Functional Testing Non-Functional Testing Why is this so often done badly or not done at all? Can we build a Test Managers toolkit to help? What would it contain? Paul Godsafe

UKTMF 27 th January 2010 Non-Functional Testing2 Reliability - The capability of software to maintain its level of performance under stated conditions for a stated period of time. –Maturity, Recoverability, Fault Tolerance Usability - The effort needed for use, and the individual assessment of such use, by a stated or implied set of users. –Learnability, Understandability, Operability Efficiency - The relationship between the level of performance of the software and the amount of resources used, under stated conditions. –Time Behaviour, Resource Behaviour Maintainability - The effort needed to make specified modifications. –Stability, Analyzability, Changeability, Testability Portability - The ability of software to be transferred from one environment to another. –Installability, Replaceability, Adaptability, Conformance (e.g. to a particular database standard) Ref. ISO 9126

UKTMF 27 th January 2010 Non-Functional Testing3 Why is Non-Functional Testing so often done badly or not at all? Need not clearly defined or understood Creates problems, not solutions –Need to define requirements –Where do you start (or end)? –Often hard to understand and/or interpret results Expensive to do –People, kit, software, time Risk/reward is hard to quantify in advance –Are you an optimist or pessimist? Solutions rarely available on demand –Cost, lead times, reliability, access to skills……. –Intermittent need

UKTMF 27 th January 2010 Non-Functional Testing4 Oh what to do, what to dooooo? what to dooooo?

UKTMF 27 th January 2010 Non-Functional Testing5 Well, we could……… Maintain an idealistic view Just complain Do nothing and accept the status quo Talk to other TMs and agree that nobody loves us Develop some pragmatic solutions based on……….

UKTMF 27 th January 2010 Non-Functional Testing6 Genuine Need, Realistic Demand and Access to solutions (tools/skills/services) Attitude to risk Commercial/operational drivers Attitude towards testing Centralised/devolved organisation What can be realistically delivered Attitude may be influenced by: –Past experience –The way you provide advice, guidance & solutions

UKTMF 27 th January 2010 Non-Functional Testing7 So can we build a Toolkit to Help? Possibly but………. A toolkit can only be used on a defined problem. Are you looking for, analysing or solving non-functional defects? In addition you may need: A strategic objective to drive the scale/scope Demand to use your toolkit –Where will the non-functional requirements come from? An interface to development projects A strategy for delivering your toolkit - self-service, internal or 3 rd party –Scalability and demand management –A delivery policy/mandate A means of payment

UKTMF 27 th January 2010 Non-Functional Testing8 Does your toolkit address the causes of inertia…. Does it define need and improve understanding? Does it provide a complete solution? Is it cost-effective? Does it deliver value-add? Is it available when needed? Virtual NFT – the power of suggestion

UKTMF 27 th January 2010 Non-Functional Testing9 The End of the Beginning