Metrics and Performance Management Session Report Alison Hudgell 27 June 2003

Introductory remarks Topic of increasing importance high-level performance targets demonstrate benefits of new developments Difficult in complex, multi-faceted system A real need for metrics that reflect what we are trying to achieve can measure our progress, year on year can be understood by decision makers at high levels in all parts of the industry Topic increasing importance: 6 metrics papers at Santa Fe and this year we had one of the largest sessions, with 15 papers. Reason - increasing move towards performance targets and performance mgt from high levels. Technology drive no longer sufficient, but requirement to demonstrate benefits of new developments before they are brought into service. Need to be able to show that we are buying the industry something it needs. Difficult in any complex and multi-faceted system like civil aviation. Some of difficulties will be highlighted later So there’s a real need for metrics that - reflect what we are trying to achieve; - are capable of measuring our progress, year on year; - can be understood by decision makers at high levels in all parts of the industry. 27 June 2003

Topics addressed & major results Metrics for classification Sector Complexity Index from workload per flight, using TAAM [58] Simple metrics, quick application to identify areas suitable for “super sectors” [34] Spectral analysis of delays used to identify sets of similar airports [46] Categorisation identifies like sectors/areas/ airports that can be treated as a group for applying analysis techniques for approaches to operational improvement Time does not allow description of all 15 papers in the stream. Mention some to highlight - results presented - issues discussed Complexity Index - identifying high complexity sectors could suggest candidates for redesign Motivation <bullet 2> 27 June 2003

Topics addressed & major results Examples... Sector categorisation by binary tree using different complexity factors [44] Applied to improve estimates of declared capacity based on simple workload measure Sector “efficiency” from comparing declared capacity with complexity measure [10] Critical but inefficient sectors could increase capacity by emulating “best in class” > critical, inefficient sectors can be improved by emulating “best in class” … but others need more radical approach 27 June 2003

Topics addressed & major results Sector complexity [7] [86] Attempts to fuse factors from proposed complexity metrics, using statistical analysis, to give a “best” metric Reduced metric better match to controller assessment of complexity than traffic count alone 27 June 2003

Topics addressed & major results Statistical analysis of conflict clusters in simulated traffic [33] Direct routes provide smaller & less complex clusters than structured routes Uncertainty increased number and size of clusters Uncertainty had more impact in structured than direct routes case Context of this work is automatic conflict resolution problem Graph theory used to understand theoretical structure of conflict clusters Real traffic data used to populate simulation, but no ATM procedures applied - these must help to break up clusters or traffic would be unmanageable by humans today 27 June 2003

Topics addressed & major results Flight efficiency benefit pool estimation [23] Consistency between studies estimating benefit pool for en-route flight time in US Order of $50-100 or 1-2 mins per flight Flight efficiency benefits pool Context is assessment of potential benefit of new ATM tools, procedures used real data Also: Performance assessment of TFM decisions [45] against delay outcome, using Bayesian network Insufficient data available Aggregate, stochastic model proposed to allow quick test of TFM strategies [3] Demonstrated success in predicting flows in centres First look at terminal area capacity and robustness 27 June 2003

Topics addressed & major results Performance assessment of ATM system development projects [92] Generalised Activity Network approach captures reality of project lifecycles Gives insight into which stages likely to overrun Combines to estimate likely extent of total overrun Compatibility with budget allocation processes? GAN Captures reality better than Pert Chart Formalises what we understand Acceptability - does not fit with requirements of budget allocation so can it be accepted as a management tool? Could build acceptability initial use to analyse risk in programme plans, and plan contingencies? 27 June 2003

Topics addressed & major results Need for standard metrics framework [28] 1500 metrics used for analysis of ATM systems 600 after eliminating duplicates Impact on communication, comparability of studies Standardisation of metrics required Vertical structure is key: breakdown from goals -> objectives -> metrics and re-aggregation FAA/EUROCONTROL workshop ref. Wim Post (EUROCONTROL) Duplicates = same concept, different words 120 safety metrics 27 June 2003

Issues discussed What is the goal? E.g. flight efficiency - compare flown route with direct route or airline requested route or ...? Presented metrics/methodologies can only capture part of the picture E.g. flight efficiency - consider en-route delay, but how add effect of ground delay? Data availability Metrics must be pragmatic - if data not available, can’t use “ideal” metric This slide & next summarise issues that generated discussion in question sessions ground delay = GDP/GS, taxi delay etc Data availability - many system variables => much data required for statistical analysis system changes over years so can’t aggregate many years 27 June 2003

Issues discussed Appropriate scoping and approach Deterministic or stochastic approaches Accommodating uncertainty in forecasts Linear vs non-linear models Adaptive behaviour system complicates analysis Degree of aggregation: dynamics (peaks) vs time-average Appropriate scoping and approach for models from which metrics are derived Not known whether ATM system qualities are stochastic or deterministic >> can we, given good enough input predictions, forecast workload, traffic complexity well - or are these stochastic quantities (like queue length) , so that the best we can do is predict the distribution? Forecasting metrics can be more useful if future values can be forecast effectiveness ltd by uncertainty intent uncertainty particularly challenging to incorporate 27 June 2003

Progress & trends Reported work focuses on capacity, complexity and delay Metrics work for ATM is not mature Complexity work reaching a new phase now trying to compare, assess and reduce to a standard set of factors work needs to converge - how can we reach accepted measures of complexity? Reported work focuses on capacity, complexity and delay Was capacity and delay at Santa Fe One paper on flight efficiency None on safety metrics Not mature many more papers this seminar than last few examples of building on previous work new metrics proposed for each new study Complexity - working towards something we can use for current applications (TFM, sector design) 27 June 2003

Progress & trends Recent increase in data availability in US but still a problem of lack of some types of data Sensitivity analyses are needed but not always conducted Sensitivity analyses may be neglected for budget reasons Recent increase in data available - US ETMS system and ASPM databases - now allowing more analyses to be done - data lacking includes weather, FM and ATM actions, causes, … 27 June 2003

Future directions? A real need for comprehensible metrics to support decision making Clarification of goals Standard, validated framework agreement of terms breakdown from goals -> objectives -> metrics and re-aggregation Trade-offs (capacity/delay/safety/efficiency) Safety metrics? As said at start - there’s a real need for metrics that -can be understood by decision makers at high levels in all parts of the industry also - reflect what we are trying to achieve; - are capable of measuring our progress, year on year; … or decisions will be made anyway on bad foundations Still a long way to go Safety - Target Level of Safety alone not sufficient for statistical analysis Can identify best practice - but can we measure its output? How to measure changes year on year? 27 June 2003