Slide 1 Test Assurance – Ensuring Stakeholders get What They Want Paul Gerrard Gerrard Consulting PO Box 347 Maidenhead Berkshire SL6 2GU UK e: w: t:

Slide 2 Why Test Assurance? Most projects need to be kept ‘honest’ Sponsors and business - Need someone ‘on the inside’ on their side - Need translations from technical, evasive spin to lay persons’ language Testing is rarely done well - project needs support from someone capable and bias- free Risks arising need someone to prod the project into action (who else will do this?)

Slide 3 Objectives of the role Primary objective: Support - To provide advice, leadership and direction to the project team to improve the effectiveness of testing and in particular test reporting Secondary objective: Assurance - To provide independent assurance to the management board on the thoroughness, completeness and quality of testing Optional  Excluded from scope is any responsibility for the delivery of testing - specifically excluded to avoid any conflicts of interest

Slide 4 Support Information Provision - Including specific points of expertise or experience Decision Support - To support the decision making process, where points of technique, documentation, reporting or appropriate (not always 'best') practice may be suggested to the team Advisory - To advise on process for traceability, coverage, sign-off for assurance purposes Risk Awareness - To identify risks that must be addressed by test phases Review support - Critical reviews of coverage, documentation, plans, results while under development.

Slide 5 Assurance Test Audit - Independent audit of test records and process to include checks for traceability, coverage, consistency, accuracy etc. Test Phase report - Phase report to document the audit, identify areas for correction, interpretation Sign-Off - See later.

Slide 6 Assessment of Testing as a whole Can the system can be used in a realistic business environment, processing realistic data? Does the system meets performance, availability and reliability objectives? Can the system can be supported by operations staff so that exceptions can be handled? Can the system can be installed, started, stopped, backed up and restored from a range of failure scenarios? Have all outstanding product risks of concern have been addressed Have each of the test phases achieved their target level of coverage so that the project can be confident in their assessment?

Slide 7 Process and Coverage Assurance Is the process sound; will it achieve the test phase objectives? Was the process followed, exit criteria met, waived appropriately? Did the appropriate, responsibility people perform sign-offs? What process was used to design tests; were coverage targets set? Were the targets met in test planning and execution?

Slide 8 Product Risk Assurance What risks drove the test planning, prioritisation? Were risks addressed through the testing? Are there risks, not tested that are outstanding?

Slide 9 Intervention types Support interventions - Intended to assist the project teams - To give direction, advice, information Assurance - To review test activities from scoping, planning through to execution and sign-off - To issue challenges to the project

Slide 10 Interventions - Assurance Test Assurance Notes: Where anomalies or uncertainties in the planning, scope or approach to testing are detected, or where a new risk to be addressed by testing appears, an Assurance Note is a challenge to the project to clarify the purpose of that aspect of the project. Test Audit: Independent audit of test records and process to include checks for traceability, coverage, consistency, accuracy etc. Test Phase report: Phase report to document the audit, identify areas for correction, interpretation. Sign-Off: see slides that define precisely what Assurance Sign-off means (and doesn't mean).

Slide 11 The Meaning of Test Assurance Sign-Off

Slide 12 Sign-Off by stakeholders and other participants Indicates that the signatories: - Have read and understood it what they have signed - Have approved a deliverable, outcome or process - Made their decision in a transparent fashion - Commit to a defined course of action – usually to ‘accept’ and/or implement - Have reached a consensus view - Have based their decision on their (or delegated) judgements - Agree that their views have been taken into consideration Signatories must be recognised authorities Sign-off is usually irreversible – once signed, cannot be undone Not granted lightly.

Slide 13 Test Assurance Sign-Off (is different) The test approach, plan, specifications, scripts, logs, incident reports have been reviewed and test lead has been interviewed/consulted The following have been ‘assured’: - Objectives and acceptance/exit criteria - The test design approach enables objectives to be met - Coverage target(s) ensure that enough information will be gathered to make the acceptance/exit decision - The exit criteria have been met or…  The remaining anomalies have been waived by the business  And… the right people have been involved - The test records indicate that the phase process has been followed and are a true reflection.

Slide 14 The politics of assurance Who will pay for the assurance function? - Would your department employ a critic? Can the Assurance Manager stay independent? - Or will he go native with the development group? Can the Assurance Manager speak the language of business? - Are you a techy at heart – unable to see beyond code, bugs, incident reports? If you made a recommendation, would the development organisation attack you, or the problem? Assurance Managers sign-off – do they take responsibility for the release? Do Test Assurance Management skills exist yet? If you ‘did’ assurance, would it be a career limiting move?