Presentation is loading. Please wait.

Presentation is loading. Please wait.

Cyril Legrand – October 22-25, 2017 IRSC 2017 – Hong Kong

Published byMelvyn Moody Modified over 6 years ago

Similar presentations

Presentation on theme: "Cyril Legrand – October 22-25, 2017 IRSC 2017 – Hong Kong"— Presentation transcript:

1 Cyril Legrand – October 22-25, 2017 IRSC 2017 – Hong Kong
Performance Evaluation Model of Railway Operators Safety Management Systems Cyril Legrand – October 22-25, 2017 IRSC 2017 – Hong Kong

2 Introduction All operators must submit their SMS to NSA
Support Operation Leadership Continual improvement Plan Check Do Act Org. Safety Management System (SMS) Main themes topics: Leadership Support Continual improvement Operation Aim: to guarantee a coverage of all risk inherent to operator activities and to improve continuously the safety management Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

3 SMS performances evaluation
Outlines Part 1 Part 2 Part 3 Part 4 SMS performances evaluation Conclusions and prospects Context Tool proposed Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

4 Context SMS Authorisation Fact: SMS must be checked and controlled during authorisation validity (5 years max.) Question: How to evaluate the performances of railways operator SMS from all along its life? A conformed SMS is it robust? How to prove it? On the basis of which criteria? Is pure conformity sufficient ? How to assess safety culture through a set of procedures ? Human factors? How to consider it in evaluation phase? Investigation of evaluation means (RESYGESS) Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

5 Context SMS can be seen as: Point of view Tool/method Methodology SMS
Petri networks UML « Stress-tests » … Evaluation 1) Complex system (Systemics) Modelisation 2) Management system (ISO standards) SMS Evaluation Deming wheel SMS submitted Risk Treatments « 5 why ? » method Controls Safety-related events Maturity models … Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

6 Focus on 2nd point of view
Issues Issues about 1st point of view (SMS modelling) Important number (several thousands for a national IM) of procedures within SMS Interfaces and dependences between procedures Different SMS structure for each operator Human factors complex to modelise Inexistence of direct modelling tool for SMS Long-term research topic Focus on 2nd point of view Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

7 Questioning about SMS evaluation
Suitability Efficiency Maturity Maturity Conformity Conformity Effectiveness Effectiveness SMS SMS SMS Robustness Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

8 Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong
Conformity Evaluation based on lists of criteria (EU regulations 1158/2010 et 1169/2010) IM A B C D E F G H I J K L M N O P Q R S T U V W RU A B C D E F G H I J K L M N O P Q R S 93 criteria 80 criteria Example of criteria: “S. PROVISIONS FOR RECURRENT INTERNAL AUDITING OF THE SAFETY MANAGEMENT SYSTEM S.1 There is an internal auditing system which is independent and impartial and which acts in a transparent way” Remaining question: The conformity with these requirements does not prove that the process set up is effective, or how it is effective? Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

9 Effectiveness Noticed deviations Supervision activities
Authorisation granted Conformed SMS Existing tool in EPSF « Performance matrix » (presented in IRSC 2015) Effectiveness evaluation Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

10 Maturity Maturity model coming from EUAR
Make easier information exchange between each NSAs and NSA / EUAR Based on PDCA model and EUAR SMS Wheel Provide NSA with a simple model which allow them to make an opinion Model for RU/IM self-assessment 5 levels (from « Basic » to « Excellence ») Example: internal audits Radar Plot Audits planned, coordinated, prioritised considering previous audits and compared with best practices No evidence of audits being carried out Maturity scale Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

11 Overview of SMS evaluation
SMS performance evaluation Safety-related events evaluation SMS performance attributes Conformity Effectiveness Maturity Indicator(s) Number of criteria to satisfy (requirements) Non conformities detected during supervision activities - Deviations from submitted SMS - Properties coming from EUAR maturity model Occurrence Gravity Tendency on 5 years Possible adjustments: - Revision of conformity, effectiveness and maturity (on a part or on entire SMS) - A warning point placed on concerned part of SMS Tendency on 5 years Model Real life Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

12 Evaluation tool proposed
Safety railway authorisation not granted At the SMS submission No Safety railway certificate (RU) or authorisation (IM) granted with restrictions 100 % ? 90 % ? Yes Application Conformity evaluation No Maturity evaluation2 Yes Safety railway certificate (RU) or authorisation (IM) granted without restrictions Impossible in new authorisation case (no preliminary data available) Maturity evaluation1 1based on previous authorisation and supervision activities performed during the validity of previous authorisation 2based on restrictions After issuing authorisation Supervision activities results Effectiveness evaluation Conformity evaluation Maturity evaluation Immediate data flux Data flux for next renewal demands Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

13 Tool presentation (1/2) Part 1: « Identity card » of operator
Operator information Supervision information Safety-related events information Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

14 Tool presentation (2/2) Part 2: Conformity / Effectiveness / Maturity
The second part is the most important. It shows the notation of conformity, maturity and effectiveness for each SMS process and for each year in the validity period of authorisation. Some macros were developed for level determination of each attributes (check-lists of criteria for conformity, questionnaires for maturity and the synthesis of noticed deviations for effectiveness). The third part is an illustrative one with a radar plot of results for each year and each process. Theory Real life Development of actions: Conformity, maturity and effectiveness results about a RU (freight and dangerous goods transport) Possible notation adjustments (no safety related events => revise upwards CEM notations?) Consideration of safety-related events (gravity 3 to 6) Future audits on concerned part of SMS (leadership) Assessors Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

15 Another point of view of SMS performance evaluation
EUAR guideline Maturity Effectiveness criteria Operator criteria (specified in its SMS) SMS Performance Effectiveness Effectiveness deviation noticed in supervision activities Non-conformity criteria Deviation with CSM Conformity Non-conformity deviation noticed in supervision activities Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

16 Conclusions and prospects
SMS = the safety core of an organisation Procedures within SMS are the common assessment basis between operator and NSA SMS → complex to apprehend, model and assess because of its « organisational » nature SMS → Human factors are omnipresent PURE CONFORMITY IS NOT ENOUGH: NEED TO GO DEEPER Model based on the combination of effectiveness and maturity in addition to conformity A model remains… a model, so safety events must be considered in mirror The tool helps the assessors by rationalizing data but can’t replace assessor’s brain BUT… Future state: Current state: tool is in validation phase (real examples) Shared tool with other NSAs and EUAR Self-assessment tool for RU/IM ? Development of a formal tool

17 Thank you for your attention
Cyril Legrand – October 22-27, 2017 IRSC 2017 – Hong Kong

Download ppt "Cyril Legrand – October 22-25, 2017 IRSC 2017 – Hong Kong"

Similar presentations

1 Future strategy for e-submission as seen by industry Dr Michael Colmorgen, IFAH-Europe 2nd Veterinary Workshop on E-submission 4 Dec 2009, EMEA, London.

1 Future strategy for e-submission as seen by industry Dr Michael Colmorgen, IFAH-Europe 2nd Veterinary Workshop on E-submission 4 Dec 2009, EMEA, London.
Risk Management in Regulatory Systems An Introductory Webinar.

Risk Management in Regulatory Systems An Introductory Webinar.
30. Conference of Directors of EU Paying Agencies Workshop1: The possibilities for optimizing the processes of implementation of direct payments Agency.

30. Conference of Directors of EU Paying Agencies Workshop1: The possibilities for optimizing the processes of implementation of direct payments Agency.
1 Certification Chapter 14, Storey. 2 Topics  What is certification?  Various forms of certification  The process of system certification (the planning.

1 Certification Chapter 14, Storey. 2 Topics  What is certification?  Various forms of certification  The process of system certification (the planning.
TENCompetence Assessment Model, Related Tools and their Evaluation Milen Petrov, Adelina Aleksieva-Petrova, Krassen Stefanov, Judith Schoonenboom, Yongwu.

TENCompetence Assessment Model, Related Tools and their Evaluation Milen Petrov, Adelina Aleksieva-Petrova, Krassen Stefanov, Judith Schoonenboom, Yongwu.
Auditing Standards IFTA\IRP Audit Guidance Government Auditing Standards (GAO) Generally Accepted Auditing Standards (GAAS) International Standards on.

Auditing Standards IFTA\IRP Audit Guidance Government Auditing Standards (GAO) Generally Accepted Auditing Standards (GAAS) International Standards on.
Welcome ISO9001:2000 Foundation Workshop.

Welcome ISO9001:2000 Foundation Workshop.
1 Accreditation and Certification: Definition  Certification: Procedures by which a third party gives written assurance that a product, process or service.

1 Accreditation and Certification: Definition  Certification: Procedures by which a third party gives written assurance that a product, process or service.
BEWAG (UIP) General assembly European Railway Agency Jean-Marie DECHAMPS, Head of Sector (Safety) Brussels, 27 April 2015.

BEWAG (UIP) General assembly European Railway Agency Jean-Marie DECHAMPS, Head of Sector (Safety) Brussels, 27 April 2015.
Development and application of guidance documents – industry view Dr Martin Schaefer ECCA-ECPA Conference March 2014.

Development and application of guidance documents – industry view Dr Martin Schaefer ECCA-ECPA Conference March 2014.
Fundamental Auditing Concepts. Materiality Evidence Independence Audit risk IS and general audit responsibilities for fraud Assurance.

Fundamental Auditing Concepts. Materiality Evidence Independence Audit risk IS and general audit responsibilities for fraud Assurance.
Workshop BeWag UIP Implementation of certification of entities in charge of maintenance Jean-Marie DECHAMPS-SAFETY Unit Brussels 28/04/2014 9.1.

Workshop BeWag UIP Implementation of certification of entities in charge of maintenance Jean-Marie DECHAMPS-SAFETY Unit Brussels 28/04/
Conformance Mark Skall Lynne S. Rosenthal National Institute of Standards and Technology

Conformance Mark Skall Lynne S. Rosenthal National Institute of Standards and Technology
© 2015 BELGORAIL 27/04/2015 – BEWAG « Staff (human) in the ECM regulation 445/20111/EU» Staff (human) in the ECM regulation.

© 2015 BELGORAIL 27/04/2015 – BEWAG « Staff (human) in the ECM regulation 445/20111/EU» Staff (human) in the ECM regulation.
Overview of Formal Methods. Topics Introduction and terminology FM and Software Engineering Applications of FM Propositional and Predicate Logic Program.

Overview of Formal Methods. Topics Introduction and terminology FM and Software Engineering Applications of FM Propositional and Predicate Logic Program.
ISSAI 400 Compliance Auditing

ISSAI 400 Compliance Auditing
RE-E / IRSC / 27/09/09 - 02/10/09 1 The necessary evolution of railway safety regulations in Europe A challenge – Risks to be controlled.

RE-E / IRSC / 27/09/ /10/09 1 The necessary evolution of railway safety regulations in Europe A challenge – Risks to be controlled.
IAEA International Atomic Energy Agency Methodology and Responsibilities for Periodic Safety Review for Research Reactors William Kennedy Research Reactor.

IAEA International Atomic Energy Agency Methodology and Responsibilities for Periodic Safety Review for Research Reactors William Kennedy Research Reactor.
Тренировъчна сесия София 23 – 24 февруари 2006 Specific measures in support of international co-operation Overall presentation of the Work programme Karina.

Тренировъчна сесия София 23 – 24 февруари 2006 Specific measures in support of international co-operation Overall presentation of the Work programme Karina.
The New Legislative Framework

The New Legislative Framework

Similar presentations

Ads by Google