CARE ASAS Validation Framework System Performance Metrics 10th October 2002 M F (Mike) Sharples

2 Content Aims Approach Analysis System Performance Metrics

3 Aims Using recognised metrics is fundamental to measuring system performance The ASAS Validation Framework requires consistent metrics to provide comparable results The System Performance Metrics work demonstrates a method for identifying existing metrics for new scenarios System Performance Metrics

4 Approach Considerable existing work in this area –PRS –C/AFT –TORCH –INTEGRA Collating these required a consistent hierarchy & taxonomy System Performance Metrics

5 Hierarchy OBJECTIVES PERFORMANCE METRICS AREAS System Performance Metrics

6 Hierarchy OBJECTIVES –Tie in with ATM Strategy –High level & therefore no direct measure PERFORMANCE AREAS –Tie in with PRC (as this gives greatest commonality) –Lower level & therefore easier to measure METRICS –The measurements that can be made System Performance Metrics

7 Taxonomy System Performance Metrics METRICS INDICATORS CARE-ASASPRC ValidationC/AFT FrameworkTORCH METRIC DEFINITION MEASURE Event, ratio or unit that is quantifiable

8 Linkage (many to many) ECONOMICS ENVIRONMENT SECURITY / DEFENCE System Performance Metrics Delay (not capacity) Cost effectiveness Flight Efficiency Environment regulation Military Co-operation Military Access Air transport security Metrics

9 Further breakdown PERFORMANCE AREAS broken down into ASPECTS where appropriate Example: –ACCESS (PERFORMANCE AREA) Airports Sectors(ASPECTS) Routes Assists use with scenarios that look at specific airspace System Performance Metrics

10 Perspectives Different views (perspectives) can be applied to the selection of metrics: – Airline perspective as in C/AFT – ATM perspective as in PRC – Validation technique Permits further breakdown and filtering other than purely hierarchical System Performance Metrics

11 Example of perspective Performance Area: Flight efficiency – Airline perspective Actual fuel burn.v. planned fuel burn – ATM perspective Efficiency of route structure System Performance Metrics

12 Characteristics Further criteria for selecting metric suitability –ObjectivityObjective/subjective –IntrusiveHigh / Low –CostHigh / Low –ReliabilityHigh / Low –ValidityHigh / Low –UtilityHigh / Low –ExpertiseHigh / Low –ResourceHigh / Low System Performance Metrics

13 Analysis To illustrate feasibility of approach a demonstrator database was created 230 System Performance Metrics stored on database Derived from recognised sources Preliminary metric classification Perspectives available –ATS provider / Operator / ASAS / Analysis Type (or any combination of these) System Performance Metrics

14 Metrics storage System Performance Metrics

15 Cross link queries System Performance Metrics

16 Flexible output System Performance Metrics

17 ASAS case studies Time based sequencing in approach Airborne self-separation in en-route airspace System Performance Metrics

18 Metrics selection criteria Time based sequencing in approach –Selected Objectives: Safety; Capacity; Economics –Selected Performance Areas: Safety; Delay; Cost Effectiveness; Flexibility; Flight Efficiency –Methodology: Each of... 1 Analytic or fast-time simulation 2 Real-time simulation –Airspace: TMA / Airport –Perspective: ASAS & each of... 1 Operator 2 Service provider System Performance Metrics

19 Metrics selection criteria Airborne self-separation in en-route airspace –Selected Objectives: Safety; Capacity; Economics –Selected Performance Areas: Safety; Delay; Cost Effectiveness; Predictability; Flexibility; Flight Efficiency; Equity –Methodology: Each of... 1 Analytic or fast-time simulation 2 Real-time simulation –Airspace: En-route –Perspective: ASAS & each of... 1 Operator 2 Service provider System Performance Metrics

20 Metrics selection Microsoft Access prototype developed to demonstrate the filtering and selection process Automated selection process provides guidance –Identifies metrics used in previous work –List is not definitive or restrictive Once automatic selection process is complete, a manual overview can select the most appropriate metrics System Performance Metrics

21 Conclusions System performance metrics can be linked to the strategic objectives of ATM (and ASAS) The work has successfully consolidated metrics from a number of sources Effective filtering requires effective classification - this will necessarily be an ongoing and iterative process Selection process provides guidance - it is not definitive or restrictive System Performance Metrics