1. 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK

2. 2 What is Validation? l EATMS Validation Strategy (EVAS) The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance

3. 3 What is Validation? l EATMS Validation Strategy (EVAS) The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance

4. 4 What is Validation? l EATMS Validation Strategy (EVAS) The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance

5. 5 What is Validation? l EATMS Validation Strategy (EVAS) The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance

6. 6 What is Validation? l EATMS Validation Strategy (EVAS) The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance l Validation builds confidence that the system is fit for purpose

7. 7 How to Validate

8. 8 Operations Operational trial Shadow-mode trial Field test Large-scale real-time simulation Small-scale real-time simulation Fast-time simulation Analytic modelling

9. 9 Where to Validate Small-scale real-time simulation

10. 10 Where to Validate Small-scale real-time simulation NATS DLR DLR CENA CENANLR EEC EEC real-life environment real-life environment

11. 11 Validation Methodology l Consistent Application of Validation Methodology Across All Sites

12. 12 Validation Methodology l Consistent Application of Validation Methodology Across All Sites

13. 13 Validation Methodology l Consistent Application of Validation Methodology Across All Sites

14. 14 Validation Methodology l Consistent Application of Validation Methodology Across All Sites

15. 15 Validation Methodology l Consistent Application of Validation Methodology Across All Sites

16. 16 Validation Methodology l Consistent Application of Validation Methodology Across All Sites of functionality, operability and performance pre-defined level of functionality, operability and performance

17. 17 Measures of Merit l MoMs Chosen to Test Hypotheses l Controller Workload l Airspace Capacity l Quality of Service l Usability l Safety l Measurements Taken: l e.g. Workload: ISA, TLX, R/T, ….. l Objective; Subjective; Observed

18. 18 Analysis l Descriptive Statistics

19. 19 Analysis l Descriptive Statistics l Inferential Statistics l significant at 5% confidence level

20. 20 Analysis l Descriptive Statistics l Inferential Statistics l significant at 5% confidence level l Backed up by Controller Subjective Comments lI can still handle the aircraft while doing the crossword!

21. 21 Analysis l Descriptive Statistics l Inferential Statistics l significant at 5% confidence level l Backed up by Controller Subjective Comments lI can still handle the aircraft while doing the crossword! desired level of confidence

22. 22 Effectiveness of PHARE Validation l Core Validation Group with Representation from all Participating Sites l Common Methodology Crucial l Methodology Improved with Experience l Validation Conducted by Individual Sites l Lessons Re-learnt &/or Lost l Effectiveness Diminished l Validation Should be Applied Throughout Project - Not Just at the End

23. 23 Summary l Validation Methodology Major Contributor to Credibility of PHARE l Forms the Backbone for much of the Validation Activities within European ATM R&D

24. 24 Summary l Validation Methodology Major Contributor to Credibility of PHARE l Forms the Backbone for much of the Validation Activities within European ATM R&D