1. Safety case development in ATM R&D Safety feedback for decision-makers and concept developers Episode 3 - CAATS II Final Dissemination Event Jelmer J. Scholte NLR-ATSI CAATS II Brussels, 13 & 14 Oct 2009

2. Episode 3 - CAATS II Final Dissemination Event 2 Contents Motivation Safety case contents Practical development of safety case Concluding remarks

3. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 3 History (1/2) Accident statistics of Large Aeroplane flights in commercial aviation Accidents Fatal Accidents Fatalities 1980-1999 period2340 613 15,554 Average per year117 30.7 777.7 Average per flight5.57 E-6 1.46 E-637.0 E-6 Separation related 7.9% 3.75% 5.0% Source: NLR-ATSI’s Air Safety Data Base

4. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 4 History (2/2)

5. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 5 Current picture It is good practice for an ANSP to develop a safety case for implementation of changes to its ATM system  to fulfill its own objectives and responsibilities  to satisfy safety regulations Several safety regulations and methods are in use that were developed for use by an ANSP for changes to its ATM system  ESARR 4  EC regulation 2096/ 2005  EATMP ANS Safety Assessment Methodology (SAM)  Eurocontrol Safety Case Development Manual (SCDM)

6. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 6 A practical example (1/2) Independent parallel departures on SIDs

7. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 7 A practical example (2/2) Key hazards in cockpit and at ATC  Crew makes error in entering the SID in FMS  ATC fails to communicate a late SID change to aircraft  ATC-published SID design entered wrongly in database Resolution of conflicts involves ATCo and pilots  ATCo cannot solve the conflict without pilot  Pilot may correct SID errors independently  Timing of pilot’s R/T frequency change from TWR to APP Challenge:  The role of the airline and the pilots is crucial  Focusing on ANSP is not desired!

8. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 8 Future challenges (1/2)

9. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 9 Future challenges (2/2)

10. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 10 Example solutions proposed Reference business trajectories Functional airspace blocks Flexible use of airspace ASAS applications Reduced separation criteria... R&D required to tackle the major design hurdle faced!

11. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 11 E-OCVM (1/2) E-OCVM to support effective R&D: “... the process whereby the many stakeholders eventually should come to a decision to either:  Continue development to... or  stop or substantially modify developments...” V1 Scope V2 Feasibility V3 Integration V4 Pre-operation V5 Operation V0 ATM Needs Idea Implemented Concept Identify ATM performance needs & constraints Scope operational concepts and create validation strategy Iteratively develop and evaluate concept Integrate concept in wider context And confirm performance Industrialisation and procedure approval Implementation

12. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 12 E-OCVM (2/2) E-OCVM poses specific, new requirements to safety case development Feedback to stakeholders!

13. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 13 Summary of motivation Most safety regulations & methods were developed for use by ANSP for changes to its ATM system Major changes to air traffic operations are needed to maintain an acceptable level of safety  ambitious targets in multiple KPAs  large number of stakeholders involved Major changes require R&D supported by safety analysis E-OCVM is the framework for validation of these major changes E-OCVM poses specific, new requirements to safety case development

14. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 14 Contents Motivation Safety case contents Practical development of safety case Concluding remarks

15. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 15 Safety analysis feedback to design Design Analysis

16. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 16 Safety analysis tailored to maturity  The aim of safety analysis changes from V1 to V5 Safety feedback to design Safety assurance V1 V5

17. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 17 Safety analysis objectives per phase V0: ATM Need Identify ATM need w.r.t. safety Identify barriers V1: Scope Plan & scope, based on evidence Feedback to design V2: Feasibility Determine feasibility Feedback to design V3: Integration Determine system level performance Feedback to design

18. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 18 Safety analysis methods Safety case development in R&D has been subject of a lot of recent research  Experiences with developing a safety case in E- OCVM are just building up  Large design challenges pose several new needs to safety case development in R&D  Several complementary approaches are emerging that aim to address the SESAR- identified emerging needs  Integration so far limited

19. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 19 SESAR-identified emerging needs A.The need for a ‘macro’ safety case B.The need to address safety regulations C.The need to address the multi-stakeholder nature of advancing air traffic operations D.The need to address the success side of a change also E.The need to cover human operators in the ATM system F.The need to identify unknown ‘emergent’ risks G.The need to address E-OCVM requirements H.The need to assess concept maturity I.The need for managing relations between cases

20. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 20 A: The need for a ‘macro’ safety case Motivation:  SESAR consists of multiple local changes by various stakeholders. Example: Functional Airspace Blocks  Includes many smaller changes Identified approaches:  Connect to an overall incident-accident model  Apportioned safety criteria based on statistics  ‘Joint safety analysis’

21. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 21 B: The need to address safety regulation Motivation: “Developing the ATM safety regulatory framework is essential to the success of SESAR” Example: ASAS applications  Responsibilities transfer from ground to cockpit  ESARR 4 applied to airline? Identified approaches:  Early scanning of concepts on fundamental safety issues including existing safety regulations  Address impact of changed regulations in early safety analysis  Safety assessment assuming current regulations, while keeping track of needs for changes

22. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 22 C: The need to address the multi-stakeholder nature Motivation  SESAR will fundamentally change stakeholder roles Example: FABs  Who manages traffic?  Who is responsible?  Who decides on acceptability of risk? Identified approach:  Safety validation framework with active roles to be played by all stakeholders - joint goal oriented approach - joint safety validation

23. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 23 D: The need to address the success side of a change also Motivation:  Safety assessments have often focused on failure  ICAO has always asked to address the success side also Example: TCAS RA downlink  Focus on failure of downlink?  What if downlink successful? Identified approaches:  Integrated safety analysis covering both failures and successes  Complement traditional ‘failure approach’ with dedicated ‘success approach’

24. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 24 H: The need to assess concept maturity Motivation: How to decide whether a concept is ready for next E-OCVM phase? Example: individual SESAR development projects Identified approaches:  Generic SARD criteria (Strategic Assessment of ATM R&D)  Safety case specific set by CAATS II in SARD update  Safety case specific set by EEC (for ‘SAME’)

25. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 25 I: The need for managing relations between cases Motivation: effectiveness and efficiency Example: use of real-time simulations  Can multiple cases benefit? Identified approaches:  Safety & HF: share info where useful, disjoint where needed  Safety & environment: disjoint analyses  Safety providing input to business  Framework for managing relations between cases

26. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 26 Contents Motivation Safety case contents Practical development of safety case Concluding remarks

27. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 27 Basic steps I.Select the phase of E-OCVM’s Concept Lifecycle Model to be tackled II.Determine objective and scope of safety analysis in line with the selected phase III.Determine methods and techniques to be used IV.Document the results

28. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 28 Selection of methods/ techniques Develop expertise and practical experience with emerging methods Work on integration of emerging methods to combine their strong points There are complementary needs of  advanced safety courses and  hands-on safety learning Get an expert aware of these emerging needs, and with experience with emerging approaches!

29. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 29 Documentation ‘Negative’ analysis results have great value as feedback to design In R&D, the value is in the explanation why a concept is not yet valid or safe Validation is most of the time invalidation Only the last cycle is validation!

30. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 30 Contents Motivation Safety case contents Practical development of safety case Concluding remarks

31. Brussels, 13 & 14 Oct 2009 Episode 3 - CAATS II Final Dissemination Event 31 Concluding remarks Experiences with developing a safety case in E-OCVM are just building up Several needs are emerging for safety case development for large design challenges, as traditional approaches fall short Several complementary approaches have been identified that aim to address the SESAR-identified emerging needs Key focus points:  Gain experience with emerging complementary approaches  Integration of emerging complementary approaches

32. Questions? Episode 3 - CAATS II Final Dissemination Event Brussels, 13 & 14 Oct 2009