Are homogeneous performance metrics for the whole ATM system life cycle available? Dr. Professor Francisco Saez & Professor Fernando Gomez Department of Aeronautical Infrastructure, Air and Space Systems and Airports. Polythecnic University of Madrid John N. Polky, Ph. D. Boeing Research & Technology Center, S.L.

ATM community objective: Objective of the work Illustrate the existent gap between high level performance indicators used normally for setting concept performance targets (ATM system capacity increase,etc... ) and low level metrics and indicators. Identify existing and recognised links. Identify those areas where additional R&D effort is still required for quantifying and modelling the various links and contribution between metrics. ATM community objective: To have available a measurement framework (an homogenous and coherent set of metrics) for the whole concept life cycle.

Context the new concepts definition process CCDTI suitability number of resolution advisories, etc.... Which are the performances that really characterise the ATM System? TOO CONSTRAINED BY THE IMPLEMENTATION! post implementation evaluation ? targets setting TOO HIGH LEVEL! ATM 2000+ SAFETY, CAPACITY, ECONOMICS.... PRC DELAY, ACCESS FLEXIBILTY. validation TOO LOW LEVEL! Controller workload number of conflicts, etc....

Context the performance measurement framework? The contribution of low level indicators to the high level indicators should be quantified and unique mathematics relationships should be established. That work is fundamental for completing the overall validation framework Which is the relationship between low level measurements and high levels performances? Which are the performances that really characterise the ATM System? R&D is required for Identify and develop the mayor gaps between High and Low level indicators

ATM system performance measurement High level performance areas vs. Local performance elements (I) ATM SYSTEM GLOBAL indicator ? Local Indicator Outcomes Results for customers Outputs Products and services Processes Activities to achieve outputs Inputs Resources ATM sub system GLOBAL INDICATORS LOCAL INDICATORS Modern approach to the ATM system performance measurement Focus on value of service provided to the user Extension beyond traditional measures of capacity and delay

Complex and not characterise relationships between: High level performance areas vs. Local performance elements (II) Complex and not characterise relationships between: High level/Global performance objectives (areas, performance...) Specific or Local performance indicators (factors, metrics,...) + - Specific or Local performance indicators are related to system components ( either airborne, ground systems, pilots, controllers, procedures,etc... High level/Global performance objectives are related to the Overall ATM systems performances Contributes or not to achieve the target performance objectives

High level performance areas vs. Local performance elements (III) Examples: High level/Global performance objectives (areas, performance...) Specific or Local performance indicators (factors, metrics,...) Factor that contributes to delay?: Controller workload: PRC/PRU: Performance area KPA Delay Local indicator : Percentage of time spent of the frequency Number of aircraft handled. Global Indicator KPI Total minutes of delay/Total number of flights Capacity index (increase in movements for a given ATFM delay)

- + + - -5% +30% Performance indicators framework elements Pilot’s workload Equipment Controller’s workload ... Controller’s situational awareness Separations standards Conflicts Delays Stress Final user perspective Safety ... Capacity Efficiency Cost effectiveness -5% - + +30% Pilot perspective + - ATS perspective Other stakeholders perspective Factors contributing to ATM System performances Local indicators Perspectives ATM System performances Global indicators

PHASE 1 PHASE 2 ON GOING WORK PHASE 3 PHASE 4 Work to be done Documentation and databases analysis Approach definition Methodology definition Structure and framework definition Gaps and R&D needs identification Supporting database creation Metrics population of the database Framework structure validity evaluation Framework rework and upgrade ON GOING WORK PHASE 2 PHASE 3 Metrics population of the database Feedback evaluation Framework rework and upgrade Framework and supporting database diffusion R+D actions Links and relationships modelling Continuous metrics assessment Continuos Framework rework and upgrade PHASE 4

Performance indicators framework scheme Stakeholders/ participants perspective or points of view High level / Global ATM Performance areas or system performances CODA TORCH PRC ATM 2000+ OCD CAF/T Local Factors ------ workload Metrics/ measurements ------ Total minutes on the frecuency ATS Airlines Airpots Final users ---- Human factors Atomatizacion Performances areas categorisation ------ Delay Metrics/ measurements ------ Total minutes of delay/Total number of flights Low level /Local indicators or subsystem performances Separation standards ATFM CNS Identify the missing contributions Develop a structured framework that allows interrelationships.

High/global ATM system performance level Developing a structured framework that allows interrelationships. (I) High/global ATM system performance level UPPER Stakeholder view relevant to: An unique and structured framework of performances that allows or accommodate the various stakeholders views. Each ones of the performances area will be characterised by their intrinsic characteristics and by the relationships with the others . Global indicators are proposed for each one of the performance that are their intrinsic characteristics Metric/units/measurements are compiled for each intrinsic area or global indicator Studied sources:OCD, ATM2000+, PRC, CAFT, INTEGRA, TORCH, NAS COLATERALL DELAY: Intrinsic characteristics LOWER Global indicators Metrics/measurements/units

ASSESSMENT OR MODELLING Developing a structured framework that allows interrelationships. (III) ASSESSMENT OR MODELLING Local Indicators

ASSESSMENT OR MODELLING Developing a structured framework that allows interrelationships. (IV) Performance aggregation process Once this low level characterisation is done performance aggregation models should be considered ASSESSMENT OR MODELLING

decomposition of factors s Final products Stakeholder views Gaps and R&D needs identification Measurement framework Performance area Life cycle state Relational database Subsystem under analysis Agregation models Structured breakdown decomposition of factors s contribution to each performance Performance roadmap

Evolution of the approaches towards performance metrics Contributes to Characterise Operational improvements Contributes to typify the received service PRC KPA OCD ATM 2000+ CAF/T Safety Capacity Efficiency Environment Safety Capacity Efficiency Environment Security Economic Uniformity Quality Delay Predictability Flexibility Efficiency Access Cost of service Safety Delay Predictability Access Flexibility Availability Equity Flight efficiency Cost-effectiveness Environment Productivity Capacity Traffic Safety Do not reflect perspectives of the stakeholders. Conceptual Level User oriented view No explicit measurements proposed

PRC (KPA) / CAFT ATM 2000+ OCD Safety Capacity Efficiency Environment Delay Predictability Access Flexibility Availability Equity Flight efficiency Cost-effectiveness Environment

Common conceptual definitions and aims but different metrics CAF/T versus PRC approach PRC KPA CAF/T It accounts for the time in excess of the optimum time that it takes a user to complete an operation It accounts for the abitly of the user to predict variation and build and maintain optimum flight schedules Delay Predictability Flexibility Efficiency Access Cost of service Delay Predictability Flexibility Flight efficiency Access Cost-effectiveness Availability Equity Environment Safety Common conceptual definitions and aims but different metrics It accounts for the ability of the system to accommodate changing user needs in real time and without penalty It accounts for the abitly of the ATM system to allow the user to adopt the preferred flight profile (route and level) It accounts for the ability to enter the ATC system and obtain services on demand The value for the money that the user received from the ATS Measures the availability of critical ATM resources and services Measures of an output over the cost to produce it Productivity Capacity Traffic Safety Number of operations using a resource in a specified period of time Measures equity of treatment Measures the conformance of air transport to environment regulations Number of total operation (flight passengers) Measures exposure of certain incidents Measures the conformance of air transport to specified safety targets

FUSION and STRUCTURATION Framework approach (I) European Performance Review System: CRCO, STATFORD, CFMU, CODA, EATCHIP CIP monitoring OCD CAF/T MAEVA R+D projects results FAA performance measurement database ATM2000+ PRC Airspace Management handbook FUSION and STRUCTURATION CAPACITY SAFETY ENVIRONMENT EFICIENCY DELAY FLIGHT EFFICIENCY PREDICTABILITY COST OF THE SERVICE ACCESS Thirteen top ATM issues FLEXIBILITY PRODUCTIVITY AVAILABLITY EQUITY

Example: Delay Performance Breakdown decomposition Identifying factor that contribuyes to the ATM system performances ! Example: Delay Performance Breakdown decomposition Main assessed approaches: Incremental approach PRC approach Initial Delay global indicators organisation Initial delay attributes characterisation Initial delay breakdown decomposition CAF/T approach + Initial identification of factor and subsystem contributing to the total ATM system Delay CFMU approach + Proposal of aggregation models. CODA approach + Integrated delay measurement framework delay intrinsic attributes delay indicators characterisation breakdown decomposition aggregation model R&D projects approach + + Operation assessment approach

Delay key attribute extension Initial DELAY intrinsic attributes Abstracted from PRC Delay key attributes Total delay/Total Nº of flights MAGNITUD SEVERITY ABSORPTION Breakdown decomposition by contributions (Phases of the flight operation, before and during the flight) Nº of delayed flights/Total Nº of flights DELAY Total delay /Nº of delayed flights DELAY DELAY DELAY DELAY DELAY DELAY Delay key attribute extension DELAY DELAY TD/Nº TF MAGNITUD SEVERITY ABSORPTION Nº DF/Nº TF Causal Factor = Delay/cause DELAY TD/NºDF

(PRC) PRC+CODA Extension PRC+CODA+ Operational assessment views Extended DELAY breakdown decomposition: Contributions to the total delay (PRC) PRC+CODA Extension PRC+CODA+ Operational assessment views

Initial Delay Aggregation Model DT/Nº TV MAGNITUD SEVERITY ABSORPTION Nº VD/Nº TV Breakdown decomposition by contributions (Phases of the flight operation, before and during the flight) DELAY DT/Nº VD DELAY DELAY DELAY DELAY Total delay/Total Nº of flights Nº of delayed flights/Total Nº of flights * Without repetition nor duplication Total delay /Nº of delayed flights

Mixed time references systems and origins (I) Several time reference system are used for measuring delays: There is a need of defining those references clearly and stabilised how to pass from one to another. Co-ordinates translation Actual times Difference between actual and schedule time Scheduled times Difference between schedule and planned time Planned times Difference between planned and preferred time Preferred times Difference between optimum and preferred time Optimum times PRC Defines : Optimum times Preferred times Planned times Scheduled times Actual times Event at Tn Difference between actual and optimum Difference between actual and planned time Delay can be calculate as difference between the various time reference as convenience

? DELAY Capacity Index Areas requiring further R+D Capacity CAPACITY place for R&D actions in the delay measurement framework? DT/Nº TV MAGNITUD SEVERITY ABSORPTION Nº VD/Nº TV DT/Nº VD Causal Factor = Delay/cause DELAY ? Capacity Index Delay/traffic curves as a measure of capacity variation Areas requiring further R+D Capacity CAPACITY Specially attention should be pay on traffic patterns definition when relating delay and capacity. Lack of models for understanding the high level relationships between performance parameters. Extension of the breakdown decomposition Definition of proper mechanisms that relates the various time reference systems so mechanisms for translation are available. Mapping delay metrics to the proposed structure and database population

Conclussions. Project object To have available a measurement framework (an homogenous and coherent set of metrics) for the whole concept life cycle. To unify sources and stakeholders views. To be corrected and completed by ATM Community To be used by anyone that could be interested in.