1. From Compliance to Performance Introduction of Performance Based Regulation (PBR) into POA Oversight Mark Barker 2 nd March 2016

2. Introduction  Mark Barker  Principal Airworthiness Surveyor  UK Civil Aviation Authority  Airworthiness Division  Civil and Military Approvals Sector  EASA Production Organisation Approvals (POA) Manager  CAA Focal for MAA Design Approved Organisation Scheme (DAOS) 2

3. Subjects  Introduction of Performance Based Regulation into UK POA  Developments in Supplier Oversight  Work on identified risks with the Prime and SME Manufacturing Groups  Findings update …  How do I get less audits ? 3

4. Prior to CAA 4

5. CAA … 5

6. Common Themes in that time  All of these audits were based on verifying compliance  Increasing commonality in requirements within Aviation  Increasing commonality of approach expected from all Regulators  Direction from Compliance to Performance … 6

7. Better Regulation Principles  Legislative and Regulatory Reform Act 2006  Regulators’ Code 2014 Any regulation should be: transparent accountable proportionate consistent targeted 7

8. 8

9. Leaflet C-180 9

10. Amendment 2 to Leaflet C-180 will require the addition of the following paragraph into your Production Exposition:- This is then used to generate the rating to determine the level of CAA Witness:- Risk Based Approach:- Suppliers 10 Approximate Percentage of Production Contracted Approximate Number of Units produced by Subcontractors Approximate Number of UK Subcontractors Number Undertaking Critical Activity (UK) Approximate Number of Overseas Subcontractors Number Undertaking Critical Activity (O’seas) FTE Personnel Undertaking Supplier QA Activity Number of on-site audits undertaken annually Result Score 0 0 25% 1 50% 2 75%+ 3 Result Score 0 100 1 200-400 2 400+ 3 Result Score 0 100 1 200-400 2 400+ 3 Result Score 0 2-5 1 10 2 50 3 Result Score 0 10 1 50 2 100+ 3 Result Score 0 2-5 1 10 2 50 3 Result Score 0 3 2-5 2 10 1 50+ 0 Result Score 0 3 2-5 2 10 1 50+ 0 Green <11 - Desktop review once per 2-year cycle. Amber (12-16) - Desktop + 1 Witnessed Visit per 2-year cycle. Red 17 + = Top 10 - Desktop + 2 Visits per 2-year cycle, prioritise on 1 Critical and 1 Overseas Additional visits may be necessary in case of very large/complex organisations

11. Change Notification (Para 2.4 in Leaflet C-180) 21.A.147 normally requires application to CAA for a change in significant subcontractor or subcontractors undertaking critical parts - Variation. For those organisations where we have established regular interface meetings to review performance, update to C-180 introduces an alternative:- But can we recognise Performance ? 11

12. 12 Surveyor will be able to agree a process where intended product transfers are discussed during regular Interface meetings. Variation then only needed when CAA decides to undertake direct sampling. Process noted in Exposition, Meeting Minutes will act as record of notification and the decision whether formal Variation is needed. 2.4.3 Interface Meetings

13. Subjects  Introduction of Performance Based Regulation into UK POA  Developments in Supplier Oversight Leaflet C180 and IN-2016/008 (Regulatory risk for the sub-contractor or supply chain to a POA course)  Work on identified risks with the Prime and SME Manufacturing Groups  Findings update …  How do I get less audits ? 13

14. Sub-Contractor / Supply Chain Course 14

15. Aviation Manufacturing Groups (AMG)  Expression of Interest from previous POA Seminar  Response was such that we adopted two Groups:-  Prime AMG – Taken from the top Ten Supply Chain Involvement organisations identified under Leaflet C-180. Rolls RoyceAirbus BombardierAgusta Westland GE (Dowty Props)Britten-Norman Aircraft BAE SystemsSafran (Messier Dowty) GKN AerospaceSpirit Aerospace Eaton Aerospace 15

16. PAMG High Top 10 - Action 16 RiskIndustry+ CAAHigh Response 9. Tier 2/Tier 3 Supply77 (1st)106 (1 st ) 9 (1 st) 23. Lack of Risk Management for Supply70 (2 nd )95 (2 nd )7 (2 nd =) 7. Emergent Technology incl. ALM67 (3 rd )94 (3 rd )6 (3 rd ) 18. Incoming Competency levels63 (4 th )86 (4 th )7 (2 nd =) 8. Special processes, NADCAP, Crimp..62 (5 th )83 (5 th )4 (5 th =) 19. NDT Skills, competence/retirement59 (6 th )80 (7 th )5 (4 th =) 12. Common approach on non-EU release59 (6 th )70 (13 th )5 (4 th =) 13.Policing of intent of design data/IPR59 (6 th )77 (8 th )4 (5 th =) 14. PMA +NAA training in acceptance59 (6 th )72 (12 th )4 (5 th =) 26. Lack of Lateral View of Escapes58 (7 th )81 (6 th )4 (5 th =)

17. Aviation Manufacturing Groups (AMG)  Small and Medium Enterprises (SME) AMG  31 initial expressions of interest but inaugural meeting was less well attended:- Britten Norman Saywell Barnbrook Systems ACRO Seating Hybrid Air VehiclesDHSL Airbus Helicopter 17

18. SME Top 10 Areas for Action 18 RiskScore Raw Material Supply74 Subcontractor Control63 Capacity/Availability of Subcontractors46 Production Competence42 Supplier N/C to Purchase Order41 Procurement not Involving Engineering38 Flow down of treatment requirements32 Legacy Support/Obsolescence28 Single Source Supply26 Competency Incoming25 Proliferation of prime requirements24

19. SME Aviation Manufacturing Groups  We are looking to agree common progress areas and hear the voice of all levels of our industry.  Should your name be here as well ? 19

20. Production Overview 20

21. CAA Focus Areas

22. So what are we finding ? 22

23. 2007 23

24. 2013/14 24

25. 25

26. There’s not a lot of difference in the picture …. Traditional regulation model “more audits” is not delivering a good safety return All Regulators are addressing a steadily growing Aviation sector with either static or reducing resource. We need to evolve, and we need organisations to evolve. What did you notice ? 26

27. 27 2015

28. The majority of recent production quality escapes requiring safety action have involved non- conformances that were already known within the organisation via concessions, incomplete FAIRs, internal audits etc. But the root cause was never addressed … Repeated concessions against design data can contribute to a culture of tolerance of “low level” non- conformance. Compliance ►► ►► Performance Common Factors 28

29. Direct involvement of Accountable Manager Regular review based on tailored Metrics appropriate to each major Production Organisation based on a common core:- MORs (Now Mandatory and Voluntary) Quality Escapes (not just those leading to a reportable occurrence) Right First Time/Reject Rate Supplier Performance Concessions/Rejections Engineering Concessions/Query Notes Internal Audit Performance Additional topics relevant to Organisation/Product:- –Significant Changes (rapidly expanding organisations) –First Time Pass Rate (rig testing output) –Those specific to organisation improvements/issues (confidential). CAA Performance Conversation 29

30. Key importance of a ‘word picture’ supporting each KPI to provide meaning to the raw numbers. One of the action Tasks for discussion with members of the Prime Aviation Manufacturing Group (PAMG). –CAA is preparing a list of KPIs from core topics and current models for agreement with the Group. –This will include agreeing meaning of terms and providing room for the narrative. Key Performance Indicator Discussion 30

31. Movement of all areas of the regulations (both EASA and Military) is towards extension of review periods where it is justified by the extent of risk of the product being manufactured and the performance of the organisation. But Rule change is slow … So CAA has recently agreed with EASA that from 2016 onwards, those production organisations which are rated as “Green” in terms of organisational complexity and product risk can receive an interim Desktop Audit instead of an on-site audit based on a performance submission. So how can the CAA recognise Performance ? 31

32. Will not be applied where Organisation:- Is newly approved (first 2-year audit cycle) Manufactures Product Manufactures Critical Parts Has received a Level 1 finding or Level 2 finding of High Safety Significance within the preceding cycle Has an identified Supply Chain complexity rating of ‘Amber’ or ‘Red’ Is audited in support of a co-operative arrangement with another NAA or a bilateral partner or under direct contract to EASA Limitations 32

33. Organisation legal entity registration is unchanged Organisation structure is either unchanged or all changes previously advised to CAA QMS on which the approval is based remains in place and has not been significantly changed Evidence of conduct of internal audit programme (audits planned, audits undertaken, summary of key findings and status) Notification of any external quality escapes where parts have been released and found to be outside design data KPI Criteria 33

34. Where Surveyor is satisfied with overall Organisation performance and the content of the response, the Desktop Audit can replace the normally scheduled Annual on-site Surveillance as recorded on Q-Pulse. So to get less audits … Effect 34

35. Thank you for listening Questions…