1. USOAP Comprehensive Systems Approach
By COSCAP-NA
USOAP
Comprehensive Systems Approach
Module 3
Introduction to USOAP

2. Contents From an Annex-based to systems-based audits
The Comprehensive Systems Approach
Tools used in the implementation of the comprehensive systems approach
Module 3
Introduction to USOAP

3. An Annex by Annex-based approach
Former auditing practices:
Assesses the implementation of specific Annexes at a time;
A lengthy process which may take decades to address all ICAO Annexes;
Does not give a clear picture of the overall safety level provided by the aviation system in the State (inter-relationship between Annex provisions may not be evident);
Expensive to implement effectively.
Module 3
Introduction to USOAP
Introduction to USOAP

4. An Annex by Annex-based approach
An Annex by Annex-based approach was practiced from 1996 – start of the voluntary assessment to 2004 – end of first cycle of mandatory audits
Experience shows that it was successfully implemented and served the intended purpose:
Raised State awareness of their safety oversight responsibilities;
Provided reliable information on the status of implementation of ICAO SARPs;
Enabled the identification of deficiencies in the global safety of aviation activities in the three areas
Module 3
Introduction to USOAP
Introduction to USOAP

5. Graphic presentation of the results of the first audit cycle
Initial Audit Contracting States (32.62%)
Audit Follow-up – 181 Contracting States (17.46%) 40 30 20 10
Primary Aviation
Legislation
Specific Operating
Regulations
Civil Aviation System
And safety Oversight
Functions
Qualification and
Training of
Technical Staff
Procedures and
Technical Guidance
Licensing and
Certification
Obligations
Surveillance
Obligations
Resolution of
Safety Issues
Module 3
Introduction to USOAP

6. An Annex by Annex-based approach
The challenge represented:
Aviation activities are interrelated. A problem in one area affects the “overall” safety of the whole system;
Safe aircraft operation is not limited to licensing, operations and airworthiness;
State oversight capabilities with respect to other aviation activities have an equal impact on safety;
Module 3
Introduction to USOAP
Introduction to USOAP

7. An Annex by Annex-based approach
The challenge represented:
Continuing on an Annex-by-Annex approach would have been very lengthy and expensive;
It would also represent an administrative problem, replacing experts on a three-year or six year cycle;
Overall picture of the health of the aviation system would have been impossible to assess.
Module 3
Introduction to USOAP
Introduction to USOAP

8. Comprehensive systems approach
A comprehensive systems approach (CSA) refers to:
“the implementation of a structured process and methodology for the planning, preparation, conduct, reporting, follow-up and evaluation of ICAO safety oversight audits.”
Module 3
Introduction to USOAP

9. Comprehensive systems approach
A structured approach enabling an audit process, methodically conducted and documented to ensure:
Standardization,
Traceability, and
Accountability
Compatible with USOAP Principles
Module 3
Introduction to USOAP

10. Comprehensive systems approach
A systems approach:
Looks at States’ safety oversight systems from a process perspective;
Provides a timely picture of the overall health of a State’s safety oversight system and its effectiveness;
Focuses attention on critical safety areas;
Allows for flexibility depending on the size and complexity of the aviation system established in a State.
Module 3
Introduction to USOAP
Introduction to USOAP

11. Comprehensive systems approach
Benefits of a systems approach:
Enables the assessment of a State’s capability for safety oversight;
Enables the assessment of the effective implementation of the critical elements of a safety oversight system;
Evaluates the overall aviation safety system established and avoids a piecemeal approach of evaluating aviation safety Standards;
Module 3
Introduction to USOAP
Introduction to USOAP

12. Comprehensive systems approach
Benefits of a systems approach:
Promotes uniform implementation of international safety Standards;
Promotes the building confidence among States in that each State would be aware of the other States’ capability for safety oversight;
Overall, it will significantly contribute to the safety and efficiency of international air transport system.
Module 3
Introduction to USOAP
Introduction to USOAP

13. Comprehensive systems approach
Implementation required the development of a plan, processes, procedures and specific tools to be used both by ICAO and Contracting States
It also required the establishment of a Programme which is:
Fully transparent; and
Works closely with all Contracting States.
Module 3
Introduction to USOAP
Introduction to USOAP

14. Comprehensive systems approach
Endorsed by Assembly Resolution A35-6 (2004)
USOAP to be further expanded to include the safety-related provisions in all safety-related Annexes;
USOAP to adopt a comprehensive systems approach in conducting safety oversight audits;
PEL, OPS, AIR, ATS, AGA and AIG to constitute the core elements (subjects) of the audit Programme.
Module 3
Introduction to USOAP
Introduction to USOAP

15. Comprehensive systems approach
Endorsed by Assembly Resolution A35-6 (2004)
Audit reports to reflect the critical elements of a safety oversight system;
Final audit reports to be made available to all Contracting States;
USOAP to continuously ensure the maintenance of the quality assurance system.
Module 3
Introduction to USOAP
Introduction to USOAP

16. Comprehensive systems approach
States (Assembly Resolution A35-6 (2004)) to:
Second qualified and experienced technical staff to ICAO on long- or short-term basis;
Submit, on a timely manner, and keep up-to-date, all information and documentation associated with the preparation and conduct of an audit;
Module 3
Introduction to USOAP
Introduction to USOAP

17. Comprehensive systems approach
States (Assembly Resolution A35-6 (2004)) to:
Cooperate with ICAO and, as much as possible, accept audit missions as scheduled by ICAO;
Accept the primacy of USOAP audit results as meeting the established SARPs;
Secretary General to foster coordination and cooperation between USOAP and other audit programmes (IATA, EUROCONTROL, etc.) (A35-7 refers).
Module 3
Introduction to USOAP
Introduction to USOAP

18. Comprehensive systems approach
(core audit areas)
Aircraft
Operations
Annexes 6, 18
PANS-OPS
Personnel
Licensing
Annex 1
Airworthiness
of Aircraft
Annexes
7, 8, 16
Effective
State Safety Oversight
System
Organization &
Safety Oversight
Functions
SAAQ
Legislation
and
Regulations
Convention
Annex 2
Aerodromes
Annex 14
Accident &
Incident
Investigation
Annex 13
Air
Navigation
System
Annexes 3,4,5
10,11,12,15 &
PANS-ATM
Module 3
Introduction to USOAP

19. Comprehensive systems approach
Audit processes and procedures:
A three-phase audit process established (to be discussed phase-by-phase, in Modules 5, 6 and 7);
Process and procedures highly transparent and available to all Contracting States;
Initial planning/scheduling to the publication of a final safety oversight report would take approximately two years to complete.
Module 3
Introduction to USOAP
Introduction to USOAP

20. Comprehensive systems approach
Selection criteria for States to be audited:
Expressed urgency to audit a particular State
Requests by a Contracting State
Size and complexity of aviation activities
Results of previously conducted audits
Regional balance
Aircraft accident and incident rates
Completion of documentations required for the audit (auditors’ tools)
Language requirements
Proximity to a State selected for an audit
Module 3
Introduction to USOAP

21. Comprehensive systems approach
Audit tools:
Audit tools are used throughout the three phases of the audit process starting with the initial planning stage to the publishing of the final audit report;
Audit tools developed include:
SOA Quality Manual;
State Aviation Activities Questionnaire (SAAQ);
Compliance Checklist (CC); and
Audit Protocols
Module 3
Introduction to USOAP
Introduction to USOAP

22. Comprehensive systems approach
State Aviation Activities Questionnaire (SAAQ)
Designed to enable ICAO to collect useful information on the organization and system established by a State to meet its safety-related obligations as a signatory to the Convention on International Civil Aviation.
Module 3
Introduction to USOAP
Introduction to USOAP

23. Comprehensive systems approach
State Aviation Activities Questionnaire (SAAQ)
Completed SAAQs to be submitted to ICAO by 31 May 2005 (extended for some States on request);
States to update information contained in their respective SAAQ whenever there is a change in their civil aviation system;
An updated SAAQ to be submitted at least 90 days prior to the conduct of the on-site phase of the audit;
SAAQ questionnaire available at:
Module 3
Introduction to USOAP
Introduction to USOAP

24. Comprehensive systems approach
Contents of the State Aviation Activity Questionnaire (SAAQ):
Part I ─ General administrative information
Part II ─ Legislation
Part III ─ Organization
Part IV ─ Operational activities
Part V ─ Air navigation services
Part VI ─ Aerodromes
Part VII ─ Aircraft accident and incident investigation
Module 3
Introduction to USOAP
Introduction to USOAP

25. Comprehensive systems approach
SAAQ, Web-based version
Module 3
Introduction to USOAP
Introduction to USOAP

26. Comprehensive systems approach
Compliance Checklist:
A tool which will provide ICAO with information on a State’s level of implementation of international Standards and Recommended Practices (SARPs);
A tool which will enable States to identify any difference which may exist between their own practices and those established by international standards (Article 38 of the Chicago Convention)
Module 3
Introduction to USOAP
Introduction to USOAP

27. Comprehensive systems approach
Compliance Checklist:
States submitting a completed compliance checklist will enable ICAO to maintain a database on a States level of compliance to the ICAO SARPs;
States are required to maintain the compliance checklist up-to-date and/or update the compliance checklist at least 90 days prior to the scheduled audit.
Module 3
Introduction to USOAP
Introduction to USOAP

28. Comprehensive systems approach
Compliance checklist contents
Standards and Recommended Practices contained in all safety-related Annexes (all ICAO Annexes except Annexes 9 and 17);
Each Annexes compliance checklist starts with a Foreword, providing States with instructions on how to complete the document;
The Foreword also defines the categories to be considered in determining differences to be identified and submitted to SOA;
Currently, CCs are used for auditing purposes.
Module 3
Introduction to USOAP
Introduction to USOAP

29. Comprehensive systems approach
Compliance Checklist – an example
Provision for comments
Level of compliance
SARP Identifier
Annex Reference
Annex SARP
Legislation Reference
Text of difference
Module 3
Introduction to USOAP
Introduction to USOAP

30. Comprehensive systems approach
Compliance checklist, Web-based version
Module 3
Introduction to USOAP
Introduction to USOAP

31. Comprehensive systems approach
Audit Protocols (PQ):
Developed by SOA with the assistance of relevant ANB Sections;
Used for the conduct of the on-site audit;
Enables auditing against the critical elements of a safety oversight system;
Provides guidelines to the auditor on what evidence should be reviewed;
Could also be used by States to conduct an internal audit (Doc 9735).
Module 3
Introduction to USOAP
Introduction to USOAP

32. Comprehensive systems approach
Audit Protocols (PQ):
Use of standardized protocols as the primary tool for the conduct of the on-site audit ensures transparency, consistency and standardization;
It enhances confidence and reliability in the conduct of audits;
Designed to reflect the core areas of the audit Programme.
Module 3
Introduction to USOAP
Introduction to USOAP

33. Comprehensive systems approach
Areas covered by the Audit Protocols:
Primary Aviation legislation and civil aviation regulations
State aviation system and safety oversight functions
Personnel licensing system
Operation of aircraft
Airworthiness of aircraft
Aircraft accident and incident investigations
Air navigation services
Aerodromes
Module 3
Introduction to USOAP
Introduction to USOAP

34. Comprehensive systems approach
Audit Protocols – an example
ICAO reference
Aspects to be audited or questions to be answered
Status
Example of evidence to be reviewed
Implementation as applicable
Comments CE
Auditor
Assessment
OPS – Legislation and regulations for aircraft operations
State Response GM
Doc 8335
1.2.1 & Doc 9734 Part A
3.3
OPS Has the State developed and promulgated specific operating regulations applicable to aircraft operations to enable the State to implement the provisions of annex 6?
“ Yes “ No
Title and date of applicability of all regulations related to operation of aircrafts.
Review the compliance checklist. 2
“ Satisfactory
“ Not satisfactory
“ Not applicable
Auditor Notes GM
Doc 8335
1.2.1 & Doc 9734 Part A
3.3
OPS Has the State developed and promulgated specific operating regulations applicable to aircraft operations to enable the State to implement the provisions of annex 18?
“ Yes
“ No
Title and date of applicability of all regulations related to operation of aircrafts.
Review the compliance checklist. 2
“ Satisfactory
“ Not satisfactory
“ Not applicable
Audit Protocol Question
Review Evidence
OPS – Availability of qualified technical personnel and training GM
Doc 8335
9.3.1 &
9.4
Doc 9734 Part A
3.6
OPS Has the CAA established minimum flight and work experience to become operations inspectors?
“ Yes
“ No
Review recruitment requirements. This can include:
1. Five or more years in air transport
flight hours,
3. Training experience, etc. 4
“ Satisfactory
“ Not satisfactory
“ Not applicable
Applicable Critical Element
Source Reference
Module 3
Introduction to USOAP
Introduction to USOAP

35. Comprehensive systems approach
Additional/Support Tools - Guidance material:
Designed to assist States in the implementation of SARPs and associated Procedures:
Doc 9734 Part A – provides guidance on the establishment and management of a State’s safety oversight system (2nd Edition, 2005)
Doc 9734 Part B – provides guidance on the establishment and management of a regional safety oversight system (1st Edition, 2005)
Module 3
Introduction to USOAP
Introduction to USOAP

36. Comprehensive systems approach
Additional/Support Tools - Guidance material:
Designed to assist States in the implementation of SARPs and associated Procedures:
Doc 9735 – provides detailed information on USOAP and on the conduct of safety oversight audits (2nd Edition, 2005)
QMSP-007-Programme Management and Implementation Provisions (SOA Quality Manual, updated on continuous basis)
Module 3
Introduction to USOAP
Introduction to USOAP

37. Comprehensive systems approach
Additional/Support Tools - Seminars and workshops
One of the main tools used to assist States to establish an effective safety oversight system;
A tool for providing information and “how to” to States in their preparation for an ICAO audit;
A means of sharing knowledge and experience.
Module 3
Introduction to USOAP
Introduction to USOAP

38. Comprehensive systems approach
Safety oversight audit process:
Pre-audit
Audit
Post audit
On-site
Location
Montreal
Montreal
Sec Gen letter to State
Opening meeting
Team meeting
Final report
Activities
Update SAAQ & CC
Closing meeting
SPO handover to TL
Time Frame
12 months
days
8 months+
Module 3
Introduction to USOAP

39. Comprehensive systems approach
The audit process feedback loop
Process
Analysis of Differences
Determination of level of SARPs Implementation
Analysis of Audit Findings
Determination of State capability for Safety Oversight
Output
Recommendations
Audit Report
Resolution of identified deficiencies
Annex supplement
Input
SAAQ
Compliance checklist
Protocols
Other sources
The Porter’s competitive force model.
Feedback Loop
Module 3
Introduction to USOAP
Introduction to USOAP

40. Review The Annex-based system The Comprehensive Systems Approach
Tools used in the implementation of the comprehensive systems approach
The overall process
Module 3
Introduction to USOAP

41. Questions ? INTRODUCTION TO USOAP By COSCAP-NA Module 3
Introduction of C-WP/12006 to Council
Thank you Mr. President, and good morning everyone,
C-WP/12006 presents the overall progress made in the implementation of the ICAO Universal Safety Oversight Audit Programme since its inception and in particular during the period from January to March 2003.
Questions ?
Module 3
Introduction to USOAP