Exit, Cry Tears Dealing with Testing Review Boards Paul Gerrard Gerrard Consulting PO Box 347 Maidenhead Berkshire SL6 2GU UK e: w: t:

Paul Gerrard Paul is the founder and Principal of Gerrard Consulting, a services company focused on increasing the success rate of IT-based projects for clients. He has conducted assignments in all aspects of Software Testing and Quality Assurance. Previously, he has worked as a developer, designer, project manager and consultant for small and large developments using all major technologies and is the webmaster of gerrardconsulting.com and several other websites. Paul has degrees from the Universities of Oxford and London, is Web Secretary for the BCS SIG in Software Testing (SIGIST), Founding Chair of the ISEB Tester Qualification Board and the host/organiser of the UK Test Management Forum conferences. He is a regular speaker at seminars and conferences in the UK, continental Europe and the USA and was recently awarded the “Best Presentation of the Year” prize by the BCS SIGIST. Paul has written many papers and articles, most of which are on the Evolutif website. With Neil Thompson, Paul wrote “Risk-Based E-Business Testing” – the standard text for risk-based testing. In his spare time, Paul is a coach for Maidenhead Rowing club.

Slide 3 Are Test Exit Reviews, and Testing Review Boards a Challenge?

Exit Criteria Textbook approach is to set exit (or acceptance) criteria e.g… - All tests executed - All tests passed - All incidents resolved, re-tested, signed-off - All outstanding incidents waived This isn’t enough… - Need some expression of coverage as a target - Theoretically, we can use test design techniques

Coverage Targets Unlikely to use white box targets for system, acceptance tests Black box targets: - Equivalence partitions - Boundary values - State transitions etc. But how often are these enforced? How do you know it’s adequate? What if the TRB ask, “How do you know?”

Coverage targets 2 Often, we have to use ad-hoc targets - 100% Branches in the business process - 100% Transactions in end-to-end scenarios - 100% Data variations in fields that are significant in processing flows  (OK 100% of SOME equivalence partitions) But how do you know these targets are met unless you do an audit?

The pressure to accept, and press- on regardless Later test phases are on critical path Overrunning a test phase slips the whole project but… - Is overlapping test phases an attractive option? - We could de-scope tests - Demote high severity defects to lower severity? More likely… - Just start the next phase anyway…

Demoting high severity defects to lower severity Severity classification should be based on business viewpoint “Does this failure adversely affect the acceptability of the system?” Business users may feel pressurised into relaxing their original assessment - Their decision may cause a project slippage Can you keep the project “honest”? Do you have to remind the team of the meaning of severity?

Starting the next phase anyway Faults found (and fixed) in one phase may invalidate tests in the overlapping phase Known faults may block tests in the later phase Resource limits, (in dev or test) slows progress – so no advantage may be gained Environmental availability may block approach and the required compromises may not be safe Again, can you keep the project “honest”.

The challenges of ‘exit’ Are your test execution metrics misleading? Do you reports tests, not steps? Where’s the bottleneck – execution, checking, dev? Do managers focus on the good news or bad news? Does environment downtime make it look worse? - Do testers find other things to do or just sit around? - Or do they work late to compensate for other folk? Are your managers over zealous, bullying? Are your managers pessimistic, over cautious? Are you asked to make judgements (on their behalf)?