1. Practical steps to proactively measure safety performance Canada Chapter System Safety Society April 19, 2012 Presented by Terry Kelly & Tom Moir SMS Aviation Safety Inc.

2. Outline  Introduction and assumptions  Steps in developing a measurement protocol  A case study  Concluding thoughts

3. Assumptions  We are dealing with complexly inter- related systems People, equipment, organization (s)  Continuous & dynamic state(s) of change  Goal is to explicitly manage risks to ALARP (performance-based)  ALARP is achieved through a blend of prescribed and performance-based requirements/standards

4. Achieving ALARP Five forms of functionality: 1.Proactive & comprehensive 2.Reliability 3.Explicit safety-risk management 4.Safety culture 5.Safety measurement

5. Requirements (1)  Measures (system) performance, not just processes  Measures positive performance, as well deficiencies in performance  Is common across programs and functions in a company/organization  Focuses mainly on proactive indicators of performance, as well as reactive measures  Focuses on hazards as well as risks

6. Requirements (2)  Accounts for human and organizational factors as well as technical and operational factors  Evaluates the results of day-to-day decision-making, as well as strategic decisions  Discriminates in terms of safety significance  Information is easily/efficiently collected and applied  The results can be replicated, and have validity over time

7. Building a Measurement Tool FOUR EASY STEPS! 1.Define performance requirements 2.Identify individual & organizational behaviours/outcomes 3.Choose a measurement scale 4.Determine collection methods

8. 1. Define Performance Requirements  Research: Industry standards & regulations Industry association literature (e.g., good practices) Texts on risk management in complex systems (e.g., Reason) Management and leadership texts  Use knowledge and experience

9. Safety Culture Everyone in the company understands their role and is committed to sharing information so that the risks related to organizational, human and operational factors are actively managed. Proactive safety management is embraced throughout the organization. There is a free flow of safety-related information vertically and laterally, within and outside the organization. The positive safety culture provides organisational resiliency, which enables the organisation’s SMS to remain appropriate and effective during times of change.

10. 2. Identify Behaviours and Outcomes  What are you going to look for? How people behave e.g., pre-job briefings, raise hazard reports, follow procedures How people think e.g., company priorities, risk How the companies manage operations e.g., metrics, documents up to date and accurate

11. Function 4 - Safety Culture A. Safety information is valued and used at all levels of the organisation. It is obtained from consistent, unfettered and active (3-way) communication throughout the organisation. B. Safety information is actively sought and communicated with stakeholders, including the regulator. C. Staff have confidence in non-punitive reporting. D. Managers trust and employ validated, safety-related information from all sources E. Everyone from all levels “walks the talk” and are actively engaged in proactive safety management.

12. Caution  You don’t necessarily want to spell out too precisely the actions or outcomes you will see.  Why not?

13. 3. Choose a Measurement Scale  Rating scale Allows you to assess change in performance over time Allows evaluator to translate and solidify an otherwise ambiguous perception  Rating scale needs to be accompanied by guidance

14. Evaluation Table Function Not Present Exists in Part Exists Completely Observations / Evidence (cite sources) Proactive & comprehensive safety management 1234567 a. Performance-based safety policy (e.g. strategic safety goal) and principles (e.g. sharing and using safety-related information) guide the organisation’s direction, and its staff’s behaviours ☐☐☐☐☐☐☐ b. Safety goals & objectives are explicitly set (safety planning) and measured, and progress reported (accountability), to continuously improve safety management and performance. ☐☐☐☐☐☐☐ Function 1 - Proactive & Comprehensive Safety-risk Management. All components of the organisation are actively engaged in or support the proactive management of safety-risks (e.g. operational, technical, financial, HR components, etc.). Safety-risk management is top-down, and integrated with strategic, business and HR planning….

15. Caution  Watch for prescriptive tools masquerading as performance- based tools!

16. What’s Wrong with This Picture?

17. 4. Determine Collection Method  Choose from a combination of: Document & Data review Interviews (management & staff) Observation (of work & meetings)  Choose sample sizes to ensure validity  Multiple lines of evidence

18. Report  Performance ratings supported by evidence  High-level strengths  System Safety Deficiencies (prioritised in terms of risk)

19. Case Study  Offshore helicopter operator  SMS (not regulated) in place since 2009  3-day SMS Evaluation

20. Function 4 C c. Staff have confidence in non- punitive reporting Score 1234567 Supporting evidence: There were indications that staff are comfortable raising items in the event reporting program when they do not feel they are getting results through other avenues. Several interviewees expressed that they did not feel their careers would be jeopardized in any way by their raising safety related concerns. Several persons interviewed expressed the opinion that AMEs were less trusting of the non-punitive reporting program than other people, and that they were also less likely to raise event reports. This is in contrast to operations, where reporting was felt to be strong. Management expressed the opinion that people ’ s attitudes were shifting (for the better) since the introduction of the SMS.

21. Function 5 C c. System safety deficiencies are assessed in terms of safety significance to guide priorities to improve safety management/performance Score 1234567 Supporting evidence: Company practices do not rigorously identify and address organizational factors. This weakness has been recognized by company management. See also 2.3.2.b. and 2.3.2.f.

22. System Safety Deficiencies Robust Analysis(B – 2) Presently, all events reported through the electronic system receive some form of analysis. This normally consists of company management discussing the event, either in person or over the Internet, and developing action plans. More significant events receive more in-depth analysis based on Tap-root terminology or the S-H-E-L(L) model. However, these analyses do not thoroughly assess HF or OF. This consistently results in safety actions that target the behaviours of frontline workers, often by calling for increased vigilance.

23. Key Points  Measures (system) performance, focussing explicitly on an organization’s capability to manage risk  Focuses mainly on proactive measures, using multiple lines of evidence drawn from different types of data

24. Conclusion Our objective has been to demonstrate how a simple tool can be developed and employed with relative ease to proactively measure the safety performance of even a large and complex organization