1. Metrics and Performance Management Session Report Alison Hudgell
27 June 2003

2. Introductory remarks Topic of increasing importance
high-level performance targets
demonstrate benefits of new developments
Difficult in complex, multi-faceted system
A real need for metrics that
reflect what we are trying to achieve
can measure our progress, year on year
can be understood by decision makers at high levels in all parts of the industry
Topic increasing importance: 6 metrics papers at Santa Fe and this year we had one of the largest sessions, with 15 papers.
Reason - increasing move towards performance targets and performance mgt from high levels.
Technology drive no longer sufficient, but requirement to demonstrate benefits of new developments before they are brought into service. Need to be able to show that we are buying the industry something it needs.
Difficult in any complex and multi-faceted system like civil aviation.
Some of difficulties will be highlighted later
So there’s a real need for metrics that
- reflect what we are trying to achieve;
- are capable of measuring our progress, year on year;
- can be understood by decision makers at high levels in all parts of the industry.
27 June 2003

3. Topics addressed & major results
Metrics for classification
Sector Complexity Index from workload per flight, using TAAM [58]
Simple metrics, quick application to identify areas suitable for “super sectors” [34]
Spectral analysis of delays used to identify sets of similar airports [46]
Categorisation identifies like sectors/areas/ airports that can be treated as a group
for applying analysis techniques
for approaches to operational improvement
Time does not allow description of all 15 papers in the stream.
Mention some to highlight
- results presented
- issues discussed
Complexity Index - identifying high complexity sectors could suggest candidates for redesign
Motivation <bullet 2>
27 June 2003

4. Topics addressed & major results
Examples...
Sector categorisation by binary tree using different complexity factors [44]
Applied to improve estimates of declared capacity based on simple workload measure
Sector “efficiency” from comparing declared capacity with complexity measure [10]
Critical but inefficient sectors could increase capacity by emulating “best in class”
> critical, inefficient sectors can be improved by emulating “best in class”
… but others need more radical approach
27 June 2003

5. Topics addressed & major results
Sector complexity [7] [86]
Attempts to fuse factors from proposed complexity metrics, using statistical analysis, to give a “best” metric
Reduced metric better match to controller assessment of complexity than traffic count alone
27 June 2003

6. Topics addressed & major results
Statistical analysis of conflict clusters in simulated traffic [33]
Direct routes provide smaller & less complex clusters than structured routes
Uncertainty increased number and size of clusters
Uncertainty had more impact in structured than direct routes case
Context of this work is automatic conflict resolution problem
Graph theory used to understand theoretical structure of conflict clusters
Real traffic data used to populate simulation, but
no ATM procedures applied - these must help to break up clusters or traffic would be unmanageable by humans today
27 June 2003

7. Topics addressed & major results
Flight efficiency benefit pool estimation [23]
Consistency between studies estimating benefit pool for en-route flight time in US
Order of $ or 1-2 mins per flight
Flight efficiency benefits pool
Context is assessment of potential benefit of new ATM tools, procedures
used real data
Also:
Performance assessment of TFM decisions [45] against delay outcome, using Bayesian network
Insufficient data available
Aggregate, stochastic model proposed to allow quick test of TFM strategies [3]
Demonstrated success in predicting flows in centres
First look at terminal area capacity and robustness
27 June 2003

8. Topics addressed & major results
Performance assessment of ATM system development projects [92]
Generalised Activity Network approach captures reality of project lifecycles
Gives insight into which stages likely to overrun
Combines to estimate likely extent of total overrun
Compatibility with budget allocation processes?
GAN Captures reality better than Pert Chart
Formalises what we understand
Acceptability - does not fit with requirements of budget allocation so can it be accepted as a management tool?
Could build acceptability initial use to analyse risk in programme plans, and plan contingencies?
27 June 2003

9. Topics addressed & major results
Need for standard metrics framework [28]
1500 metrics used for analysis of ATM systems
600 after eliminating duplicates
Impact on communication, comparability of studies
Standardisation of metrics required
Vertical structure is key: breakdown from goals -> objectives -> metrics and re-aggregation
FAA/EUROCONTROL workshop
ref. Wim Post (EUROCONTROL)
Duplicates = same concept, different words
120 safety metrics
27 June 2003

10. Issues discussed What is the goal?
E.g. flight efficiency - compare flown route with direct route or airline requested route or ...?
Presented metrics/methodologies can only capture part of the picture
E.g. flight efficiency - consider en-route delay, but how add effect of ground delay?
Data availability
Metrics must be pragmatic - if data not available, can’t use “ideal” metric
This slide & next summarise issues that generated discussion in question sessions
ground delay = GDP/GS, taxi delay etc
Data availability -
many system variables => much data required for statistical analysis
system changes over years so can’t aggregate many years
27 June 2003

11. Issues discussed Appropriate scoping and approach
Deterministic or stochastic approaches
Accommodating uncertainty in forecasts
Linear vs non-linear models
Adaptive behaviour system complicates analysis
Degree of aggregation: dynamics (peaks) vs time-average
Appropriate scoping and approach for models from which metrics are derived
Not known whether ATM system qualities are stochastic or deterministic
>> can we, given good enough input predictions, forecast workload, traffic complexity well - or are these stochastic quantities (like queue length) , so that the best we can do is predict the distribution?
Forecasting
metrics can be more useful if future values can be forecast
effectiveness ltd by uncertainty
intent uncertainty particularly challenging to incorporate
27 June 2003

12. Progress & trends
Reported work focuses on capacity, complexity and delay
Metrics work for ATM is not mature
Complexity work reaching a new phase
now trying to compare, assess and reduce to a standard set of factors
work needs to converge - how can we reach accepted measures of complexity?
Reported work focuses on capacity, complexity and delay
Was capacity and delay at Santa Fe
One paper on flight efficiency
None on safety metrics
Not mature
many more papers this seminar than last
few examples of building on previous work
new metrics proposed for each new study
Complexity -
working towards something we can use for current applications (TFM, sector design)
27 June 2003

13. Progress & trends Recent increase in data availability in US
but still a problem of lack of some types of data
Sensitivity analyses are needed but not always conducted
Sensitivity analyses may be neglected for budget reasons
Recent increase in data available
- US ETMS system and ASPM databases - now allowing more analyses to be done
- data lacking includes weather, FM and ATM actions, causes, …
27 June 2003

14. Future directions?
A real need for comprehensible metrics to support decision making
Clarification of goals
Standard, validated framework
agreement of terms
breakdown from goals -> objectives -> metrics and re-aggregation
Trade-offs (capacity/delay/safety/efficiency)
Safety metrics?
As said at start - there’s a real need for metrics that
-can be understood by decision makers at high levels in all parts of the industry
also
- reflect what we are trying to achieve;
- are capable of measuring our progress, year on year;
… or decisions will be made anyway on bad foundations
Still a long way to go
Safety - Target Level of Safety alone not sufficient for statistical analysis
Can identify best practice - but can we measure its output?
How to measure changes year on year?
27 June 2003