One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 1 FAA/Eurocontrol Technical Interchange Meeting on Performance Metrics Track 2: Metrics in Validation Validation Strategy Ulrich Borkenhagen, Eurocontrol SDE/SCS

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 2 Action Plan 5: Validation and Verification Strategy Objective To determine a strategy for validating and verifying the performance, reliability, and safety of ATM systems and its possible relations to certification. The term performance includes benefits such as capacity gain, cost effectiveness, and environmental issues. The strategy should allow for the validation and verification during the phases of research and development and implementation of airborne and ground-based ATM sub-systems in relation to the current and future operational context.

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 3 Validation Strategy - History EATMS FAA Operational Validation StrategyConcept Validation Plan FAA/EUROCONTROL Operational Concept Validation Strategy ~ ~

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 4 Operational Concept Validation Strategy Terminology Validation Process during the Concept Development Strategy Implementation Strategy Approach (Example)

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 5 Terminology Verification Certification Validation Concept Validation

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 6 Initial Requirements Development Activity Product User Needs / Expectations Development Activity Product Real live operating Deliverable System Development Activity System Development Activity System Development Activity System Development Process

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 7 Initial Requirements Development Activity Product User Needs / Expectations Development Activity Product Real live operating Deliverable System Development Activity System Development Activity System Development Activity Verification The process of evaluating the products of a given system development activity to determine correctness and consistency with respect to the products and standards provided as input to this activity.

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 8 Initial Requirements Development Activity Product User Needs / Expectations Development Activity Product Real live operating Deliverable System Development Activity System Development Activity System Development Activity Certification The process aiming at the satisfaction of authority that a deliverable complies with a set of regulations, in order to ensure its proper operation.

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 9 Initial Requirements Development Activity Product User Needs / Expectations Development Activity Product Real live operating Deliverable System Development Activity System Development Activity System Development Activity Validation The process through which a desired level of confidence in the ability of a deliverable to operate in real-life environment may be demonstrated against a pre-defined level of functionality, opera- bility and performance.

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 10 Concept Validation The sequence of Validation steps integrated into the concept development pro- cess by which an individual concept instantiation is vali- dated and through which the necessary understanding to mature the concept is gained. Initial Requirements Development Activity Product User Needs / Expectations Development Activity Product Real live operating Deliverable System Development Activity System Development Activity System Development Activity

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 11 Valida- tion Process Life- cycle

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 12 Valida- tion Process Life- cycle Observation, reporting and agreement of new concept

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 13 Valida- tion Process Life- cycle Initial proof of concept through model or early prototype

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 14 Valida- tion Process Life- cycle Full specification of concept, pre-operational demonstration

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 15 Valida- tion Process Life- cycle Factory Acceptance

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 16 Valida- tion Process Life- cycle Sign off for operations

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 17 Valida- tion Process Life- cycle R&D, ATSPs System Provider Local ATSP

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 18 Strategy Implementation Overall Plan Validation Data Repository (VDR) Common Validation Methodology Understanding Validation Objectives Validation Environment Creation of Confidence

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 19 Overall Plan

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 20 Validation Data Repository (VDR)

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 21 Common Validation Methodology Validation Route Map High Level Methodology Approach

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 22 Validation Route Map

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 23

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 24 High Level Methodology Approach

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 25 Understanding Validation Objectives

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 26 Strategy Approach High Level Objective: e.g. capacity model

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 27 Strategy Approach High Level Objective: e.g. capacity model Contribution from OI: e.g. MTCD model Contribution B model Contribution A model

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 28 Strategy Approach High Level Objective: e.g. capacity model Contribution from OI: e.g. MTCD model Contribution B model Contribution A model Metric

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 29 Strategy Approach High Level Objective: e.g. capacity model Contribution from OI: e.g. MTCD model Contribution B model Metric Scenario V1 V2 V3

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 30 Example: FAP

One Sky for Europe EUROCONTROL © 2002 European Organisation for the Safety of Air Navigation (EUROCONTROL) Page 31 Track 2: Metrics in Validation Can operationally relevant metrics also be used in simulation environments to predict the benefits of the concept ? Can these metrics be generated by our simulation and validation tools, what level of confidence can they give ? To what extent are concept dependent and allow comparisons across validations ?