Technical Program Delivery Group international Engineering Safety Management Independent Assessment Accreditation Paul Cheeseman Technical Program Delivery Group paul.cheeseman@tpd.uk.com © TPD 2017

iESM - Aim www.intesm.org To assist the international railway industry in delivering products/systems with acceptable levels of safety by developing & sharing good practice in railway Engineering Safety Management worldwide. Developed as part of the TPD internal research activities, for the good of the rail industry, supported by MTR Corporation. TPD, having unique access to the Yellow Book originators, former YB Steering Group members and the editor, decided to address the international need with a new handbook. With the kind support of MTR Corporation we have been able to establish a new Working Group to steward and take forward ESM for a wider audience. The primary purpose of international Engineering Safety Management (iESM) is to help people who lead and undertake railway engineering make sure that their work contributes efficiently to improved safety and helps new railways and changes to be accepted more efficiently. The key strengths are “all in one place” and “good practice, not mandated standard”. www.intesm.org © TPD 2017

AN 4 Independent Assessment v “Your organization must ensure that engineering safety management activities are reviewed by competent people who are not involved with the activities concerned.” Provides more detailed guidance on iESM Guidance, Volume 2, Chapter 17 – how to do it Aimed at Independent Assessment organizations / Independent Assessors Useful for railway administrations, Project Managers and Project Safety Managers iESM Principle These reviews may be divided between process-focused reviews (often referred to as safety audits) to check how things are being done and product- focused reviews (often referred to as independent assessments) to check what is being produced. Audits provide evidence that you are following your plans for safety. Assessments provide evidence that you are meeting your safety requirements. How often and how thoroughly each type of review is carried out, and the degree of independence of the reviewer, will depend on the extent of the risk and novelty and on how complicated the work is. If an ESM activity is done many times, it may be better to specify it precisely and review the specification rather than the activities themselves. You will still need to confirm that you comply with the specification each time you carry out the activity but this compliance will not be subject to independent assessment. © TPD 2017

“Quis custodiet ipsos custodes? ” “Who will guard the guards themselves?“ – Juvenal (Satires) “Who watches the watchmen“ – Star Trek “There are checks and balances in science. There’s somebody checking the people doing the science and then there is somebody who checks the checkers and somebody who checks the checker’s checkers.” – Michael Shermer EXCEPT THERE ISN’ T - Who assesses the independent assessors? – as far as we can tell no one! Surely a gap in the safety assurance arrangements for clients and suppliers alike? © TPD 2017

Existing safety-related schemes Mandatory accreditation schemes for: Assessment Bodies (under the Common Safety Methods for Risk Assessment) Notified Bodies (under Interoperability Regulations) Design Bodes (under National Notified Technical Rules) Client specified schemes such as IRSE Signaling licensing But nothing for Independent Assessment (apart from generic audit training) Whilst similar, none of these covers the risk-based independent safety assessment role and all of them bring a significant overhead of accreditation assurance. © TPD 2017

Assessment or Certification? Risk-based: “In this situation, with these assumptions, caveats and dependencies this is adequate” A judgement based on professional opinion supported by objective evidence of process, inspection of output and compliance with standards Compliance-based: “This complies with the specified standard” A statement of fact where no judgement, risk assessment or test of reasonable practicability is necessary © TPD 2017

Life saving second opinions Primary purpose of independent assessment is provide confidence that the product or application under consideration that the risk is controlled to an acceptable level. A secondary purpose is to improve the ESM evidence and / or its presentation. Not about seeking a single, elusive “right answer” © TPD 2017

Emerging CENELEC requirement The emerging draft of new CENELEC EN50126-2 includes a set of key competences for an Assessor including having an “acceptance/license” from a recognized safety authority. The standard offers no further guidance on what is expected or how it may be achieved…… © TPD 2017

CENELEC Requirements for an Assessor No Requirement 1 Be competent in the domain/technologies where independent assessment is carried out 2 Have acceptance / license from a recognized safety authority 3 Have / strive to continually gain sufficient levels of experience in the safety principles and the application of the principles within the application domain 4 Be competent to check that a suitable method or combination of methods in a given context have been applied 5 Be competent in understanding the relevant safety, human resource, technical and quality management processes in fulfilling the requirements of the EN 50126 6 Be competent in independent assessment approaches/methodologies 7 Have analytical thinking ability and good observation skills 8 Be capable of combining different sources and types of evidence and synthesize an overall view about fitness for purpose or constraints and limitations on application 9 Have an understanding of the overall system including its application environment 10 Understand the requirements of EN 50126 © TPD 2017

Additional iESM requirements 11 Professional status in an engineering or scientific discipline relevant to the system or equipment 12 Prior experience as an independent assessor or safety engineer for a minimum of 5 years in areas relevant to the system or equipment 13 A commitment to safety 14 The flexibility to adapt to changing circumstances and the perform assessment tasks efficiently and to minimize wastage of physical and virtual resources © TPD 2017

iESM Approach Professional Registration Behavioural competence – including for example maintaining independence Technical competence - e.g. safety engineering and risk acceptance techniques, assessment and audit, general consultancy skills Knowledge – legal requirements, domain, system, technology, specialist areas iESM training, self assessment with “Responsible Manager” The intent here is not to repeat other schemes and existing competence assessments CV, company processes © TPD 2017

Independent Assessment Scope & Content Appreciation Assessment Strategy Selection and Planning Reviewing Safety Documentation Assessing Safety Analysis Producing assessment reports including formation of an overall judgment on the safety of a product or system, or process used. Managing Outcomes (excluding audit) © TPD 2017

Evidence of competence “Have you done it?” Or if not: A demonstration that the Assessor would be able to apply the competency correctly in a hypothetical situation (can be applied); The ability to answer questions pertaining to the competency based on past experience (can be tested); Evidence of having been trained in that particular competency (has been shown). © TPD 2017

Levels of competence Level 1 is the base level which indicates that the Assessor has sufficient knowledge and understanding of good practice to be able to work on assessment tasks under supervision. Their competencies are likely to have been developed through targeted training and work on non-assessment projects. At Level 2, the Assessor has sufficient knowledge and understanding of good practice and has sufficiently demonstrable experience, to be able to work on the tasks associated with the overall activity without the need for detailed supervision. At Level 3, the Assessor has sufficient understanding of why things are done in certain ways, and sufficiently demonstrable skills, to be able to undertake overall responsibility for leading a task or project. A Level Three Assessor will be familiar with the ways in which systems or products have failed in the past. They are able to deal with a multiplicity of problems under pressure without jeopardising safety or quality issues. © TPD 2017

Summary New iESM Accreditation scheme is available for individuals and independent assessment service providers. Aims to be “light-touch” but give substantial credibility to the Assessors who achieve the mark through a combination of training, experience and demonstrable competence. Piloted in Australia and should fill a gap in the assurance arrangements for clients and suppliers. Supported by iESM and IA training courses from TPD and their partners Refinements will be made as experience grows and in response to emerging requirements from the international standards. www.intesm.org © TPD 2017