1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK

2 What is Validation? l EATMS Validation Strategy (EVAS) The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance

3 What is Validation? l EATMS Validation Strategy (EVAS) The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance

4 What is Validation? l EATMS Validation Strategy (EVAS) The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance

5 What is Validation? l EATMS Validation Strategy (EVAS) The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance

6 What is Validation? l EATMS Validation Strategy (EVAS) The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance l Validation builds confidence that the system is fit for purpose

7 How to Validate

8 Operations Operational trial Shadow-mode trial Field test Large-scale real-time simulation Small-scale real-time simulation Fast-time simulation Analytic modelling

9 Where to Validate Small-scale real-time simulation

10 Where to Validate Small-scale real-time simulation NATS DLR DLR CENA CENANLR EEC EEC real-life environment real-life environment

11 Validation Methodology l Consistent Application of Validation Methodology Across All Sites

12 Validation Methodology l Consistent Application of Validation Methodology Across All Sites

13 Validation Methodology l Consistent Application of Validation Methodology Across All Sites

14 Validation Methodology l Consistent Application of Validation Methodology Across All Sites

15 Validation Methodology l Consistent Application of Validation Methodology Across All Sites

16 Validation Methodology l Consistent Application of Validation Methodology Across All Sites of functionality, operability and performance pre-defined level of functionality, operability and performance

17 Measures of Merit l MoMs Chosen to Test Hypotheses l Controller Workload l Airspace Capacity l Quality of Service l Usability l Safety l Measurements Taken: l e.g. Workload: ISA, TLX, R/T, ….. l Objective; Subjective; Observed

18 Analysis l Descriptive Statistics

19 Analysis l Descriptive Statistics l Inferential Statistics l significant at 5% confidence level

20 Analysis l Descriptive Statistics l Inferential Statistics l significant at 5% confidence level l Backed up by Controller Subjective Comments lI can still handle the aircraft while doing the crossword!

21 Analysis l Descriptive Statistics l Inferential Statistics l significant at 5% confidence level l Backed up by Controller Subjective Comments lI can still handle the aircraft while doing the crossword! desired level of confidence

22 Effectiveness of PHARE Validation l Core Validation Group with Representation from all Participating Sites l Common Methodology Crucial l Methodology Improved with Experience l Validation Conducted by Individual Sites l Lessons Re-learnt &/or Lost l Effectiveness Diminished l Validation Should be Applied Throughout Project - Not Just at the End

23 Summary l Validation Methodology Major Contributor to Credibility of PHARE l Forms the Backbone for much of the Validation Activities within European ATM R&D

24 Summary l Validation Methodology Major Contributor to Credibility of PHARE l Forms the Backbone for much of the Validation Activities within European ATM R&D