FAA/Eurocontrol TIM 9 on Performance Metrics – INTEGRA Rod Gingell 16 May 2002

INTEGRA What is INTEGRA? What are INTEGRA metrics? Why are they necessary? How do they support concept evaluation and target setting?

INTEGRA Project The INTEGRA projects objective is to provide a quantified assessment of proposed automated ATM tools and associated procedures within an ATM simulation. This assessment is to be made in terms of: –Safety –Capacity –Efficiency, and –Environmental Impact The assessment is to be made at a system level rather than at the sub-system level

What are INTEGRA metrics? INTEGRA metrics have been designed to provide quantitative measures to establish the benefit or otherwise of advanced ATM systems containing a degree of controller assistance tools They have been designed to operate at the ATM system level measuring capacity, safety, efficiency and environmental impact

Why are they necessary? Previous simulations that have included advanced tools have been beneficial in demonstrating tools and concepts but have not provided quantitative measures of performance Typically current metrics for capacity are based on controller workload assessment –These are largely subjective measures, questionnaires etc –Only linked to capacity because the controller task is seen as the bottleneck.

Presentation Topics How INTEGRA metrics can be used to support the definition of future concepts and concept evaluation The importance of the Operational Concept and the relationship between metrics and the Operational Concept The need to measure the ATM system when it is operating in accordance with its design This presentation will, using the Capacity Metric, demonstrate the following:

INTEGRA Capacity Metric In current systems Controllers workload is considered to be the bottleneck in terms of capacity Capacity has therefore been linked to controller workload and a number of metrics have been developed to measure this workload In advanced ATM systems it is not evident that controller workload will be the limiting factor The INTEGRA capacity metric is designed to determine the capacity achievable using such systems

Capacity Metric Premises (1) There are a number of actors identifiable that perform tasks in an ATM system e.g. –Tactical Controller –Planning Controller –Conflict probe –etc. Information has to be processed to execute an ATM task e.g. –detect a conflict –resolve a conflict –monitor traffic –etc. Information processed is the same whichever actor is involved in processing it

Capacity Metric Premises (2) Each actor will have a different threshold as to the amount of information that can be handled Capacity of the system is reached when any one of the actors is overloaded

Principles of Capacity Metric Identification of actors from operational concept/procedures Identification of activities Allocation of activities to actors Algorithm operates with simulation output data to identify an amount of Information Processing Load (IPL) for each of these activities Allocation of IPL to the identified actor at the time that it occurs Sum the IPL for each actor and compare with a threshold to determine if overloaded at any time This sequence of events is illustrated in the following slides

Capacity Metric Sequence Operational Concept Identification of Actors Allocation of Activities to Actors Simulation - events recorded Metrics calculated Levels compared with threshold Identification of Activities

Configuring Actors Trajectory Predictor Flight Path Monitor Conflict Probe PilotPlanning Controller Tactical Controller etc Actors Co-ordination Other flight plan changes Monitoring Resolution Implementation Resolution Planning Interaction Detection Flight Acknowledgement Tasks

Capacity Metric Application Measurement of Symptoms Reconstruction of Events When recorded Data / Events do not match INTEGRA Metric input events Ideal Flow Alternative Flow Known Metrics Develop Concept Recorded Data / Events Simulation Design Simulation Calibration Comparison Extraction of Events Metrics Calculation Re-develop Concept and/or Re-design Simulation

Interaction Detection Zone Look-ahead Time Limit Conflict Detection Zone Resolution Implementation Interaction Detection / Resolution Planning Conflict with another aircraft Other Aircraft Sector Boundary Trajectory Edit Direction BAW1234 Example Trajectory Update

Actor : Controller Allocation for One Actor Interaction Detection & Resolution Planning Resolution Implementation Time IPL for One Trajectory Edit & One Actor

Actor : ControllerActor : Conflict Probe Allocation for Two Actors Interaction Detection & Resolution Planning Resolution Implementation Time IPL for One Trajectory Edit & Two Actors

Prerequisites Need to be able to calibrate metric output to determine threshold For computer processes such as a conflict probe or trajectory predictor the thresholds are readily obtainable e.g. memory usage, processor usage etc. For human actors it is more complicated. Traditional approaches can be used to obtain workload estimates to compare with the IPL values, these include: –questionnaires –ISA measurements –heart rate monitoring etc But these estimates may not necessarily correspond to the actual work required to control the system

Validation Requirements Three phases to Validation –Platform must operate correctly –Controller acceptance of concepts and system –Measurement of system capacity etc when operating according to requirements Each of these have to be addressed to obtain a true validation

INTEGRA Metrics in EACAC 2000 SIMULATION

Capacity - Results En-route Sector TJ Group 2 November 2000 Without ASASWith ASAS

Capacity - Results ETMA Sector AO Group 3 June 2000 Without ASASWith ASAS

Capacity - Results En-route Sector TJ Group 1 November 2000 Without ASASWith ASAS

Demonstrator Demonstrator has the following facilities: ATC HMI on which traffic can be controlled Processing of output data with Capacity and Safety algorithm Display of output data