1. ICAO EUR HLSC Preparatory Seminar
9-11 February 2010
Baku, Azerbaijan
Theme 1: The foundation for global aviation safety
Topic 1.2:The evolution of the safety oversight audit process by ICAO

2. Theme 1-The foundations for global aviation safety
Topic 1.2 – The evolution of the safety oversight audit process by ICAO
Objective
Present a report on the evolution of the safety oversight audit process by ICAO, highlighting the transition from periodic compliance audits to safety risk-based continuous monitoring
There is a working paper put forward by ICAO for the upcoming HLSC next month and the purpose of that paper to present a report on the evolution of the ICAO USOAP process and its transition from periodic compliance checks to a continuous and risk-management based monitoring. The transition to this new approach will begin in 2011 and involves a period of adjustment and testing, in order to create a system that will provide the best service to ICAO Member States while also generating valuable data and contributing to the improvement of global aviation safety. This presentation provides an outline of the new CMA and a description of a transition phase and seeks for the support of the CMA and to the transition plan by States,
ICAO EUR High Level Safety Conference Preparatory Seminar

3. What is safety oversight ?
Safety oversight is a function by which Contracting States ensure the effective implementation of:
ICAO Standards and Recommended Practices (SARPs);
the critical elements of a safety oversight system; and
relevant safety practices and procedures.
To step aside a little bit and remind about objectives of the ICAO USOAP and the meaning of the Safety Oversight function of a State, By employing this function a State ensures that ICAO SARPs and relevant safety practices and procedures are effectively implemented. There 8 critical elements that must be established and implemented by a State in order to ensure an effective oversight function in place.
Safety oversight is a State’s responsibility
ICAO EUR High Level Safety Conference Preparatory Seminar

4. Critical elements of a State’s safety oversight system1
Legislation 2
Operating
Regulations 3
Organization,
SO Functions 5
Guidance,
Procedures
& Info. 4
Technical
Experts
Trg.
ESTABLISH 6
Licensing &
Certification
Obligations
IMPLEMENT 7
Surveillance
& Inspection
Obligations
Primary aviation legislation-the provision of a aviation law –comprehensive and effective and commensurate the environment and complexity of the State’s aviation activity.
Specific operating regulations; this term is used in a generic sense and includes but is not limited to, orders, directives, instructions, rules, requirements, policies etc that provide adequate provisions to meet requirements emanating from the primary legislation and provide for a standardised procedure, equipment or infrastructure.
State civil aviation system and safety oversight function- there must be stae civil aviation authority established supported with qualified technical personnel, safety oversight function, safety objectives and policies stated, enforceable and empowered.
Qualification and training of Technical personnel – establish a minimum knowledge and experience requirements for a personnel involved in a safety related activity. Appropriate training is provided to maintain and enhance the competencies.
Technical guidance (standardise), tools and provision of safety information to enable technical personnel to perform their safety oversight related duties.
Licensing, certification, authorization and approval of personnel and organisations performing aviation activity.
Surveillance and performing inspections and audits to ensure that the holders continue meeting the requirements
Implement processes and procedures to ensure that safety concerns not only identified but analysed and resolved 8
Resolution
of Safety
Concerns
DOC 9734 Part A
ICAO EUR High Level Safety Conference Preparatory Seminar

5. Why an ICAO Programme?
Various reports in the early 1990s on the lack of implementation by States of ICAO SARPs.
Accidents indicating insufficient safety oversight by States.
Increased concern over the level of aviation safety worldwide.
Need to reduce accident rates to off-set the rapid increase in traffic.
The ICAO USOAP history has its roots in 1990s when various reports received various aviation community stakeholders and analysis of accidents and incidents had revealed the lack of implementation of ICAO SARPs and indicated insufficient level of safety oversight by States.
This coupled with rapid and continued despite of some temporary disruption, growth in air traffic, led to concerns over the level of aviation safety worldwide.
Certainly, the growth in traffic /density is a factor. Statistically the more movements there are the more errors/failures happen and we need to act proactively to counter balance this.
ICAO EUR High Level Safety Conference Preparatory Seminar

6. Objective of the Programme
The objective of USOAP is to promote global aviation safety through auditing Contracting States, on a regular basis, to determine States’ capability for safety oversight by assessing the effective implementation of the critical elements of a safety oversight system and the status of States’ implementation of safety-relevant ICAO Standards and Recommended Practices (SARPs), associated procedures, guidance material and safety-related practices.
Therefore the USOAP was conceived with an objective to promote global safety by regular auditing to determine safety oversight capability by assessing the level of effective implementation of the critical elements of a safety oversight system and also the status of implementation of safety-related SARPs
ICAO EUR High Level Safety Conference Preparatory Seminar

7. Evolution of USOAP D 35TH G Mandatory A Voluntary C Audit Programme S
Assessment
Programme
1995 – 1998
Annexes 1, 6, 8
88 Requests
67 Assessments
Annex-by-Annex
Approach D G C A O N F
’97
Mandatory
Audit Programme
(USOAP)
A-32/11
1999 – 2004
181 Audits
162 Follow-ups
Annexes 1, 6, 8
Annex-by-Annex
Approach
35TH A S E M B L Y
’04    
The programme had started with its first phase of voluntary assessment based on safety related requirements of Annex 1, 6 and 8. that further evolved into its second phase of mandatory, regular, systematic and harmonised safety audit programme but still annex compliance- centric,
ICAO EUR High Level Safety Conference Preparatory Seminar

8. Annex by Annex approach
It was applied from 1996 – start of the voluntary assessments until 2004 – end of first cycle of mandatory audits.
Assessed the implementation of specific annexes.
Experience shows that it was successfully implemented and served the intended purpose:
Raised State awareness on their safety oversight responsibilities.
Provided reliable information on the status of implementation of ICAO SARPs.
Enabled the identification of deficiencies in the global safety of aviation activities in the three areas.
These audits had proved to be successful substantiated by a fact that is had provided reliable information on the status of SARPs implementation and helped to identify deficiencies
ICAO EUR High Level Safety Conference Preparatory Seminar

9. Evolution of USOAP D Comprehensive G Comprehensive Systems C Systems
Approach
A-35/6
2005 +
Safety-related
Provisions in all
Annexes
(All except
9 and 17) D G C A O N F
’06
Comprehensive
Systems
Approach
Greater
Transparency –
Public
information
Procedure for
Significant
Safety Concerns    
However, as we mentioned earlier the evolving complexity of aviation activities demand more proactive and comprehensive approach, therefore a decision was taken at the 35 Assembly to embark on a comprehensive systems approach in conducting the USOAP where compliance against all safety related provisions of all Annexes(except Annex 9 and 17) was included as well assessment of the effective level of safety oversight function implementation.
Later on, it was decided at the DGCA in 2006 to augment this approach by adding greater transparency , procedure to tackle with specifica safety concerns identified in the course of the audits. etc
ICAO EUR High Level Safety Conference Preparatory Seminar

10. Comprehensive systems approach
Endorsed by Assembly Resolution A35-6 (2004):
USOAP to be further expanded to include the safety-related provisions in all safety-related Annexes.
USOAP to adopt a comprehensive systems approach in conducting safety oversight audits.
PEL, OPS, AIR, ATM, AGA and AIG to constitute the core elements (subjects) of the audit Programme.
This slide summarise the outcome o the 35th Assembly
ICAO EUR High Level Safety Conference Preparatory Seminar

11. Comprehensive systems approach
Aircraft
Operations
Annexes 6, 18
PANS-OPS
Personnel
Licensing
Annex 1
Airworthiness
of Aircraft
Annexes
6, 7, 8, 16
Effective
State Safety Oversight
System
Organization &
Safety Oversight
Functions
SAAQ
Legislation
and
Regulations
Convention
Aerodromes
Annex 14
Accident &
Incident
Investigation
Annex 13
Air
Navigation
System
Annexes 2,3,4,5
10,11,12,15 &
PANS-ATM
The elements the CSA are depicted on this slide. The CSA assess the level of effective SO implementation and compliance against the following standards.

12. Results of the DGCA Conference 2006
Greater transparency – audit information to be made public.
Procedure to deal in a timely manner with significant safety concerns identified during audits.
Classification of audit findings under the critical elements.
Consequential amendments to the MOU on safety oversight audits.
This slides adds more information on the outcome of the DGCA where an agreement for a greater transparency was achieved and as a consequence amendments to MoUs were made.
ICAO EUR High Level Safety Conference Preparatory Seminar

13. USOAP CSA Audits Completed as of December 2008
The CSA cycle will be completed by the end of The second edition of the analysis of audit results for a reporting period until December 2008 is made available on ICAO website.
The purpose of this analysis is to provide information on the degree of implementation of the 8 Ces and on the status of SARPs compliance.The analysis identifies the areas of concern fpr indovidual States or group of States, allows to evaluate the impact on global aviation safety, consider the various options available to improveconformance and priritize actions and provide assistance where required.
Herewith we provide some results of the analysis.
This slide shows the geographical distribution by ICAO region of the 190 contracting staets and 113 of the audited. ICAO is on shcedule to complete all 190 by the end of this year.
ICAO EUR High Level Safety Conference Preparatory Seminar

14. Release of audit information to the public
Information released to the public is posted in the Flight Safety Information Exchange (FSIX) website.
For audits conducted during the first audit cycle:
States were requested to sign a release consent form authorizing ICAO to release either a summary of the audit report and a graph, or the entire report.
For audits under the comprehensive systems approach:
Graph depicting implementation of the critical elements; or full audit report.
MOU amended to allow for the release to the public
The audit results are released to public on the ICAO FSIX website.
For the audits of the first cycle the States were requested to sign up to a consent to release either a summary or the entire report.
For the CSA the MoUs were amended to allow for the release.
ICAO EUR High Level Safety Conference Preparatory Seminar

15. Global Audit Results
This figure shows the average percentage lack of effective implementation for each of the 8 Ces of a ssafety oversight system. The global average is 42 %.
The percentage in the graph is the ratio of not satisfactory responses to protocol questions over the total number of applicable questions.
The CE with the highest lack of effective implementation is CE-4 on qualification and training. The analysis cntains a detailed analysis per CE and identifies the prime causes of non-compliance.
ICAO EUR High Level Safety Conference Preparatory Seminar

16. Lack of CEs effective implementation in the EUR Region
The slide exhibits the level of non-compliance in the EUR per CE
ICAO EUR High Level Safety Conference Preparatory Seminar

17. Accident rate (global) analysis
The following figure depicts the accident rates for the period ranging from 1990 to A trend line is also provided display a continuous reduction in the accidents number despite of continuous growth of traffic.
Note that the graph depicts all reported accidents involving aircraft of a MCTOM of 2250kg for scheduled passenger services with fatalities reported through the ADREP and is related to the State of registry of the aircraft involved
ICAO EUR High Level Safety Conference Preparatory Seminar

18. Accident rate by ICAO Region 1994-2008
Accident rate per 10 million departures
Of the 113 states audited 46% have had fatal accidents and these state represent around 67% of global capacity. The remaining 54 % have had a relatively accident-free record and represent 8% of global traffic capacity. It should be noted that audit results indicate there are several states have had difficulties in establishing a sound accident and incident reporting system
ICAO EUR High Level Safety Conference Preparatory Seminar

19. Lack of implementation vs Accident rates
Accident rate per 10 million departures
When plotting the results by ICAO region there is apparently no strong corelation between accident rates and overall audit results. However, the coreelation is stronger between accident rates and specific CEs
ICAO EUR High Level Safety Conference Preparatory Seminar

20. Relationship between accident rates and individual CEs
The R is the coefficient of determination, a statistical measuremnt of the proportion of variability in a set of data. Gives information on the goodness of a model.
As you see there is a very strong relationship with lack of effective implementation in the area of licensing and certification, surveillance. The link is low in legislation and operating procedures areas.
ICAO EUR High Level Safety Conference Preparatory Seminar

21. Evolution of USOAP I 36TH A C A O H L S C USOAP S beyond 2010 E M
’07
USOAP
beyond
2010
A-36/4
Continuous
Monitoring
Mechanism I
C A O H L S C
’10   
So far about the analysis of the audit results and the CSA cycle is almost completed. Now what is next?
ICAO EUR High Level Safety Conference Preparatory Seminar

22. Assembly resolution 36-4
Application of a continuous monitoring approach for the ICAO Universal Safety Oversight Audit Programme (USOAP) beyond 2010
Whereas the primary objective of the Organization continues to be that of ensuring the safety of international civil aviation worldwide;
Whereas promoting the implementation of international standards contributes to this objective;
Recalling that the 32nd Session of the Assembly resolved that a universal safety oversight audit programme be established, comprising regular, mandatory, systematic and harmonized safety oversight audits to be carried out by ICAO;
Whereas the ICAO Universal Safety Oversight Audit Programme (USOAP) has been successful in meeting the mandate given by Resolutions A32-11 and A35-6;
Recalling that the ultimate responsibility for safety oversight rests with Contracting States, who shall continuously review their respective safety oversight capabilities;
Recalling that the 35th Session of the Assembly recognized that the establishment of regional and sub-regional safety oversight organizations has great potential to assist States in complying with their obligations under the Chicago Convention through economies of scale and promotion of uniformity on a larger scale; and
Recognizing the need to address the future nature and direction of the USOAP following the completion of the current audit cycle in 2010;
Recalling the primary ICAO objective of safety,
recalling the role of States in implementing a safety oversight,
Acknowledging the success of the USOAP CSA programme,
The assembly identified the need to address the future nature and direction of the USOAP
ICAO EUR High Level Safety Conference Preparatory Seminar

23. Assembly Resolution 36-4 The Assembly:
1. Expresses its appreciation to the Secretary General on the continuing success in the implementation of the comprehensive systems approach for the USOAP;
2. Directs the Council to make appropriate changes to USOAP to incorporate the analysis of safety risk factors, including the corresponding guidance material, and apply them on a universal basis, following the completion of the current audit cycle in 2010, to assess, on an on-going basis, States’ compliance with their oversight obligations and adapt the audit planning and scope accordingly;
3 Directs the Council to examine the feasibility, among the various options that could be considered, of a new approach based on the concept of continuous monitoring, to be implemented at the end of the current audit cycle in In so doing, the Council shall continue to maintain as core elements the key safety provisions contained in Annex 1 – Personnel Licensing, Annex 6 – Operation of Aircraft, Annex 8 – Airworthiness of Aircraft, Annex 11 – Air Traffic Services, Annex 13 – Aircraft Accident and Incident Investigation, and Annex 14 – Aerodromes; and
4. Requests that the Council report to the next ordinary Session of the Assembly on the overall implementation plan for this new auditing approach to commence after 2010.
Expressed its appreciation
Directed the Council to assess feasibility and make appropriate changes to USOAP to incorporate the analysis of risk factors and apply them universally on an ongoing basis, adopting a more proactive approach and making a more efficient and effective use of the resources, including the role of the Regional Offices.
This has led to a new approach based on the concept of continuous monitoring supported by the necessary tools and methodologies.
The core elements for compliance assessment with the safety related provisions of all Annexes is maintained but the methodology is adapted to the changing environment to become more proactive, interactive, distributed (involving regional offices).
Report on the overall implementation plan.
ICAO EUR High Level Safety Conference Preparatory Seminar

24. Continuous Monitoring-States
On-line framework
Sign a new MoU
Appoint a NCMC
Update CAP
Complete CMA protocols
Update SAAQ
Develop CMA protocols completion plan
Pass a CBT
Update Corrective Action Plans
Update State Aviation activity Questionnaire
Complete CMA protocols
The CMA involves the establishment of a system to monitor the safety oversight capability of member states on on-going basis and with a harmonised and consistent approach towards assessing the safety level of aviation activities and evaluate safety management capabilities.
As interactive on-line framework will need to be established to facilitate State’s participation in the CMA. A centralised database to effectively manage information received from different sources on an on-going basis.
Under the CMA, the USOAP will be capable of identifying when CSA audits, specific purpose audits, or other types of intervention such as operational or technical assistance are required and gain flexibility to allow the best use of resources.
Under the CMA transition plan both ICAO and States are required to complete a number of actions .
States will be first required to sign MoUs with ICAO and nominate their NCMC . Each NCMC will receive NCMC and will be responsible for communicating to ICAO their respective State’s progress in resolving their safety oversighdefficiencies and their level of aviation activity,
The on-line framework allows the states to update their corrective action plans, Update SAAQ and complete CMA protocols on-line. This information will need to be provided prior to launching the CMA.
This information will be shared through the on-line interactive framework with all member States, regional offices and HQ.
ICAO EUR High Level Safety Conference Preparatory Seminar

25. Continuous Monitoring-ICAO
On-line framework
Agreements with International entities
Publish the new Doc9735
Update Corrective Action Plans
Test the CMA on-line framework
Launch CBT auditor training
CMA workshops
Update State Aviation activity Questionnaire
Complete CMA protocols
ICAO will use this period to further develop the comprehensive guidance material and on-line tools that will be needed by States and Regional Offices. A new edition of Doc9735 is scheduled in June 2011.
In the meantime the on-line framework will be tested with selected volunteers. .
New agreements and amendments to existing for the sharing of confidential safety information will also be made between ICAO, EC, IATA,EASA. This will assist in reducing the burden on States caused by repetitive audits or inspections designed to evaluate the level of a State safety oversight capability. Regional training workshops to be conducted and training for auditors to be prepared.
ICAO will conduct targeted ICAO coordinated validation missions when necessary during a transition period. ICVMs wll determine the status of corrective or mitigating measures taken by States on safety defficiencies, including significant safety concerns.. Other intervention activities that would normally be found under a CMA are to be phased in gradually, with pilot projects conduct in select States, including audits on-request. ICAO would be able to conduct audits on request where the scope would be defined by the requesting state on a cost-recovery basis.
Audits on request
ICVMs
ICAO EUR High Level Safety Conference Preparatory Seminar

26. ICAO transition to CMA
A detailed transition plan has been prepared for the 2 year period frpom to 2012 leading to the full scale implementation of the CMA. This transition plan includes timelines of all major components and presented on this slide. .
ICAO EUR High Level Safety Conference Preparatory Seminar

27. Member States transition to CMA
And the project plan for the States.
The CMA launch on 1 Jan 2013 will seethe implementation of a full array of intervention activities, including CSA audits or limited CSAs.
The results of all audits will be displayed on a secure website and kept updated. An overview of the audit results for each member state will continue to be made available to the public using the current methods.
ICAO EUR High Level Safety Conference Preparatory Seminar

28. Conclusions
The USOAP programme CSA audits cycle is scheduled to be completed by the end of 2010
CSA audits have proven to be successful and facilitated to improving civil aviation safety
Changing environment requires continuous improvement and dynamic interaction
Integration of safety-risk based management concept into the USOAP
Transition to be completed by 2013
States are urged to support this process
In conclusion,
Changing environment demands for a more proactive approach to managing safety. Continuous monitoring with safety –risk management concept integrated and dynamic interactive on-line framework is a solution.
However, being a dynamic and interactive network, this requires a full support by States to make it happen.
The successful implementation of this transition plan relies on active cooperation and feedback from States and active participation of Regional Offices and regional organisations.
ICAO EUR High Level Safety Conference Preparatory Seminar

29. Questions and Answers
ICAO EUR High Level Safety Conference Preparatory Seminar

30. Additional slides Guidance material
Seminar/workshop on the Preparation, Conduct, and Reporting of an ICAO Safety Oversight Audit 30

31. Comprehensive systems approach
Additional/Support Tools - Guidance material:
Designed to assist States in the implementation of SARPs and associated Procedures:
Doc 9734 Safety Oversight Manual, Part A – provides guidance on the establishment and management of a State’s safety oversight system (2nd Edition, 2006).
Doc 9734 Safety Oversight Manual, Part B – provides guidance on the establishment and management of a regional safety oversight system (First Edition 2006).
ICAO EUR/NAT

32. Comprehensive systems approach
Additional/Support Tools - Guidance material:
Doc 9735 Safety Oversight Audit Manual– provides detailed information on USOAP and on the conduct of safety oversight audits (2nd Edition, 2006).
Additional guidance material is associated with individual protocol questions, as applicable.
ICAO EUR/NAT