1. Wholesale Market Reform
V0.1
31/10/2016
Wholesale Market Reform
CMOS 2.3 Exit Criteria
10th January 2017

2. Executive Summary
The purpose of this document is to set out for review a set of Exit Criteria to be applied to the MOSL CMOS 2.3 release.
This criteria only covers MOSL and it excludes Market Participant criteria.
The Test Exit Criteria will be used as the gateway between the transition of the CMOS 2.3 release from the Market Participant Sandpit (MPS) to the Production Environment. This Test Exit Criteria is designed to capture the current risk position and it will be used to determine when the testing is complete. This may include the acceptance of concessions.
Note: although this criteria covers the defects uncovered by the Market Participant and MOSL’s testing, it isn’t intended to replace the criteria defined by MPs or MOSL set out within their own test strategies. Specifically, MOSL retain the responsibility of ensuring CMOS is fit for purpose, and each MP has a responsibility of ensuring CMOS integrates successfully with their own internal systems and processes, through their own internal quality assurance processes.

3. Exit Criteria (MOSL)
In the context of the following criteria, the set of "tests" refers to the testing scope covered by both MPs (covering their internal systems and processes) and MOSL (covering the CMOS product), including and not limited to SIT and UAT. The criteria is as follows:
A confirmed baselined version of the application and any associated data and configuration items was delivered into the Market Participant Sandpit;
100% of all in-scope tests have been attempted and all defects encountered have been logged and triaged with MOSL, complete with the agreed assigned priority and severity;
All defects delivered as part of the CMOS 2.3 Release Note have been retested;
All fixed defects delivered during the testing as hotfixes have been retested;
The impact of de-scoped tests (including de-scoped defect fixes) is understood and captured;
No defects exist with a priority of P1 or business severity of S1 (see, “Appendix A” for definitions);
The impacts of any P2 or S2 defects is understood and accepted;
Outstanding P2 or S2 defects have an agreed (CIO Forum) release for delivery;
The impact of P3 and P4 defects are understood and accepted;
If test cases or defects are outstanding, an acceptable technical or procedural workaround has been identified with an acceptable agreed residual risk;

4. Exit Criteria (MOSL)
Test Completion Report is updated with the results of the testing and is signed-off within MOSL;
A tested back-out plan is available and in place;
Release Notes, deployment, and back-out guides have been updated for use in live;
Any test constraints (e.g. missing end-points requiring the use of stubs and / or “test only” configuration changes) are captured and shared;
It is agreed any P1 or S1 defects occurring in production after the release date are resolved within 10 days of go-live.

5. Appendix A - Definitions
Note: the Market Participant User Forum will need to agree how to convert observations to defects.
Defect priorities and severities:
Priority
Description
P1 - Urgent
Blocking defects (i.e. technical showstoppers) experienced by more than one Market Participant. No testing is possible.
P2 - High
Highest priority for non-blocking defects. This indicates areas of the testing cannot be performed or the functional area is significantly degraded; however, progress can continue in other areas.
P3 - Medium
Indicates a defect which should be fixed.
P4 - Low
Indicates little urgency is needed for a defect fix
Severity
Description
S1 - Business Critical
Blocking defects (i.e. Business showstoppers) experienced by more than one Market Participant. The Business cannot operate, can operate at considerable risk (e.g. reputational, financial) or cost, there is impractical effort required.
S2 – Serious
Areas of the Business cannot operate or can operate only for days at considerable risk or cost with significant effort required.
S3 - Degraded Service
Business workarounds are required at risk and cost. Note increased volumes of minor defects collectively may become a serious issue.
S4 - No Service Impact
Business workaround available at an acceptable risk and cost.