1. 1 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader

2. 2 Analysis Is Not Easy!

3. 3 Unreasonable Demands

4. 4 Not Understood

5. 5 Does Anyone Listen?

6. 6 European Wide Validation NATS DLR DLR CENA CENANLR EEC EEC

7. 7 European Wide Problems!

8. 8 European Wide Solution

9. 9 What is Validation? The process through which a desired level of confidence in the ability of a deliverable to operate in a real-life environment may be demonstrated against a pre-defined level of functionality, operability and performance Validation builds confidence that the system is fit for purpose

10. 10 Methodology Matures with Time PD1 PC/TC PD1+ PD2 PD2 + PD3 PD1++

11. 11 l Experimental Design PD1 Validation Planning

12. 12 PHARE Experimental Design l Comparison To Baseline l For Users to Understand Results - Baseline must be Close to Current System

13. 13 PHARE Experimental Design l Comparison To Baseline l For Users to Understand Results - Baseline must be Close to Current System l Progressive Introduction of Experimental Variables in New Organisations

14. 14 PHARE Experimental Design l Comparison To Baseline l For Users to Understand Results - Baseline must be Close to Current System l Progressive Introduction of Experimental Variables in New Organisations

15. 15 PHARE Experimental Design l Comparison To Baseline l For Users to Understand Results - Baseline must be Close to Current System l Progressive Introduction of Experimental Variables in New Organisations

16. 16 l Experimental Design l Identifying Data Collection l Objective Data l What; How; Format l Subjective Data l Questionnaires l Debriefs l Concentration on Controller Workload l Instantaneous Self Assessment (ISA) l NASA TLX (paper) PD1 Validation Planning

17. 17 l Data Collected In One File l Tedious data analysis l No Recorded Observations l so-and-so said such-and- such……I think l Limited Material For Debrief Discussions l Video Data - but limited Controller Cognitive De-Brief PD1 Validation Execution

18. 18 l No Real Thoughts on Analysis Techniques l Descriptive Statistics l Appropriate for Nominal Data? l Inferential Statistics Employed as Analysis Progressed l Allowed Confidence Intervals l Limited Considerations to Measures of Merit l Capacity = fn(workload)? l Quality of Service PD1 Validation Execution

19. 19 PC/TC, PD1+, PD2, PD2+ Validation Preparation l Clearly Defined Aims l Objectives l Hypotheses l Measures of Merit l Controller Workload l Quality of Service l Capacity l Usability l Data Required l Accessible Storage l Formal Observation l Psychologists

20. 20 Formal Documentation l Engineering Plan l Layout, tools, equipment l Analysis Plan l Aims, objectives, H0, H1 l Design l Scenario,Traffic samples l No. exercises, Duration l Data recording l Organisations l Ensures matched pairs l Measurements l Analysis technique, Stat tests

21. 21 PC/TC, PD1+, PD2, PD2+ Validation Execution l Use of both Descriptive and Inferential Statistics l Defined by Analysis Plan l Supported with Subjective And Formally Observed Data l Immediate Input to Debrief l Electronic TLX l Trajectory plots l Relatively Small and Well Informed Core Group of validators

22. 22 PD3 Validation Planning l Validation Group expanded to non-analysts l Needed to Disseminate Knowledge - Documented l Experimental Plan Requirements l Analysis Plan Requirements l Aims, Objectives, H0, H1 l ……. l Analytical Methods l Acted as Consultancy

23. 23 PD3 Validation Execution l Planning Worked Well - Execution of Analysis Limited to CENA l Documentation can Only be a Guide to Practitioners of Best Practice l Cant Learn Analysis / Validation from a Book

24. 24 PD1++ Validation Planning l Planning - routine l Fast-Time Simulation (TAAM) Introduced l Development of airspace, traffic samples l Early examination of potential problem areas l Training l Introduced Temporary Operating Instructions

25. 25 PD1++ Validation Execution l Caught out by Complacency l Capacity measures used to date only applicable to non-changing airspace l Cannot compare capacity between different airspace designs as used in PD1++ l Initiated research study

26. 26 Effectiveness of PHARE Validation l Core Validation Group with Representation from all Participating Sites l...those involved in the validation should have a thorough understanding of the operational concept under investigation and of any restrictions/specifics of the simulation environment... l Common Methodology Crucial l …traceability of all input and output data is important for correct analysis and can not be over-emphasised... l...the metrics applied in a research programme like PHARE must be relevant to the performance of real life ATM...

27. 27 Effectiveness of PHARE Validation l Methodology Improved with Experience …the Validation Methodology avoided disruptive data collection. The use of intrusive measurements like eye tracking and heart-rate variability measurements could be valuable additions... …the Validation Methodology avoided disruptive data collection. The use of intrusive measurements like eye tracking and heart-rate variability measurements could be valuable additions... …the application of the appropriate technology to perform the investigations is very important. The early choice of PHARE to perform real time simulations has put a too strong focus on this type... …the application of the appropriate technology to perform the investigations is very important. The early choice of PHARE to perform real time simulations has put a too strong focus on this type...

28. 28 Effectiveness of PHARE Validation l Validation Should be Applied Throughout Project - Not Just at the End l...there has been too little input from the Validation project in the beginning of the PD/3 project. In hindsight it can be said that a stronger focus on the validation objectives of the project should have led to a better project definition...

29. 29 Validation is a Living Subject l PHARE Validation Methodology developed over time and through experience l Methods are documented

30. 30 Validation is a Living Subject

31. 31 Validation & Measurement Methods for the PHARE Demonstrations R A Whitaker Validation Project Leader next