1. Trial SMS Assessment 4 th ARAST Meeting Bangkok, Thailand 21 to 25 February 2011

2. Objective  To Report on the Trial SMS Assessment completed by COSCAP in 2010 COSCAP 4th ARAST 2011 2 Trial SMS Assessment

3. Background  Effective implementation of SMS is key to achieving the desired safety benefits  Service Providers will implement SMS to meet State requirements  Civil Aviation Authorities will conduct oversight to verify effective implementation  Traditional oversight audit and inspections methods measure compliance (i.e., implementation and use)  Traditional oversight audit and inspections methods not well- suited to measure effectiveness of SMS processes COSCAP 4th ARAST 2011 3 Trial SMS Assessment

4. Assessment Method  Based on ICAO Safety Management Manual SMS Gap Analysis  Preparation ~~ Development of method and assessment tools  Conduct of Assessment ~~ Four Parts: Completion of a Documentation Review Guide by the Air Operator Review of the completed Review Guide and Company documentation by the Assessment Team On-site assessment Report Completion COSCAP 4th ARAST 2011 Trial SMS Assessment 4

5. ICAO Safety Management Manual (DOC 9859)  Provides guidance on the features of an SMS  SMM includes a “Gap Analysis” table to assist implementation  “Gap Analysis” table identifies features that an SMS should have  COSCAP developed SMS “Expectations” from the Gap Analysis COSCAP 4th ARAST 2011 Trial SMS Assessment 5

6. Gap Analysis ~~ SMS Assessment Expectations Chapters 3, 9 &10 Does the [organization] have reactive processes that provide for the capture of information relevant to safety and risk management? Chapters 9 &10 Has the service provider developed training relevant to reactive methods of safety data collection? Chapters 9 &10 Has the service provider developed communication relevant to reactive methods of safety data collection? COSCAP 4th ARAST 2011 Trial SMS Assessment 6 Chapters 3, 9 & 10 3.5.3; 3.5.12; 9.9.5 10.4.1 3) The organization has reactive processes that provide for the capture of information relevant to safety and risk management Chapters 9 & 10 4.4.6; 9.11.6; 9.11.7 10.4.2 4) The service provider has developed training relevant to reactive methods of safety data collection Chapters 9 & 10 7.2.8; 9.11.2; 9.11.7 9.12.2 5) The service provider has developed communication relevant to reactive methods of safety data collection

7. Assessment Documentation  Outline of Trial Assessment (pdf)(pdf)  Expectations (pdf)(pdf)  Documentation Review Guide (pdf)(pdf)  Assessment Worksheet (sample)sample COSCAP 4th ARAST 2011 Trial SMS Assessment 7

8. Assessment Conduct  Review of Company safety management documents & records  Review of actual processes  Interviews with personnel (President, middle management & staff)  Observation of actual work practices COSCAP 4th ARAST 2011 Trial SMS Assessment 8

9. Evaluation of Assessment Method  Generally good  Expectations Some revision of expectations required for clarity Eliminate some duplicate expectations  Worksheets Reduce number of sample questions Clarify sample questions Improve alignment of questions to their expectation COSCAP 4th ARAST 2011 Trial SMS Assessment 9

10. Evaluation of Assessment Method  Model report required  Assessment Team training required  Guidance for Assessment timeline required Formal initiation by CAA Notification of Service Provider Service Provider requires time to prepare Review Guide (minimum 30 days) Completed Review guide and Service provider documentation needed by Assessment Team 30 days prior to on-site review Assessment Team face-to-face meetings to complete final review preparations COSCAP 4th ARAST 2011 Trial SMS Assessment 10

11. SMS Assessment ~~ Next Steps  Revise COSCAP SMS Assessment tools  Conduct second Trial Assessment  Refine models  Develop related COSCAP Guidance material  Distribute to COSCAP members  Provide support and training as requested COSCAP 4th ARAST 2011 Trial SMS Assessment 11

12. Assessment Team COSCAPKim Trethewey Len Cormier Nie Junjian AAPAMartin Eran-Tasker AirbusBruno Delprat BoeingDr. Bill Rankin CAA SingaporeHuei Wang Kang Capt Michael Hoh Michael Pang Center Aviation Safety Technology, ChinaZhang Yuan CAA State of OperatorFour CAA Inspectors IATATony Houston Korean Air LinesPeter Blake SR TechnicsEduard Schweizer COSCAP 4th ARAST 2011 Trial SMS Assessment 12

13. Appreciation Tremendous support from COSCAP Safety Partners  Airbus  Association of Asia Pacific Airlines (AAPA)  Boeing Commercial Airplanes  International Air Transport Association (IATA)  Korean Air Lines  SR Technics COSCAP 4th ARAST 2011 Trial SMS Assessment 13

14. Comments ~~ Questions Kim Trethewey Chief Technical Advisor COSCAP North Asia kim.trethewey@coscap-icao.org COSCAP 4th ARAST 2011 Trial SMS Assessment 14