1. EUROCONTROL/FAA ATM R&D June, 20031 Standardizing Performance Metrics Almira Williams and Stéphane Mondoloni, Ph.D. for Tamara Breunig CSSI, Inc. Washington D.C. Diana Liang and Steve Bradford FAA Washington D.C. CSSI, Inc.

2. EUROCONTROL/FAA ATM R&D June, 20032 Outline Why metrics and standardization Framework research Inconsistencies in researched frameworks –Strategic goals –Objectives –Performance areas –Metrics Summarized framework Important issues Current activities

3. CSSI, Inc. EUROCONTROL/FAA ATM R&D June, 20033 Efforts Towards Standardization Why are metrics significant? –Means of measuring against strategic goals –Indication of system’s performance and trends –Indication of performance change with changing conditions Why standardize metrics? –Establish harmonized terminology –Avoid ambiguous metrics –Avoid misrepresentative metrics –Ease of comparing concepts (exactly “apples with apples”)

4. CSSI, Inc. EUROCONTROL/FAA ATM R&D June, 20034 Approach Basic Layered Framework Investigated Programs & Reports Classification of Objectives & Metrics Expanded Framework

5. CSSI, Inc. EUROCONTROL/FAA ATM R&D June, 20035 Framework Research Federal Aviation Administration (FAA) –Operational Evolution Plan (OEP) –Airspace Management Handbook (AMH) Eurocontrol –Cooperative Actions of Research and Development in Eurocontrol (CARE ) System Performance Metrics (WP 2) Human Performance Metrics (WP 3) –Performance Review Committee (PRC) –Required Total System Performance (RTSP) –TORCH (Technical, Economical and Operational Assessment of an ATM Concept Achievable from the Year 2005) National Aeronautics and Space Administration (NASA) –Advanced Air Transportation Technologies (AATT) –Virtual Airspace Modeling and Simulation (VAMS)

6. CSSI, Inc. EUROCONTROL/FAA ATM R&D June, 20036 Basic Framework Strategic Goals Objectives Metrics Generic Characteristics ~ Metric Categories Objective vs. Performance Area Scope of the Project ~ Outcomes Performance Indicators

7. CSSI, Inc. EUROCONTROL/FAA ATM R&D June, 20037 Performance Areas Summary of Objectives Access ATM Community Participation Availability Capacity Civil-Military Cooperation Cost Effectiveness Delay Economics Efficiency Environment Equity Expectations Flexibility Flight Efficiency Global Interoperability Human Involvement Predictability Productivity Military Access Quality Safety Security and Defense Security of Air Transport System Operational Performance Transition Uniformity

8. CSSI, Inc. EUROCONTROL/FAA ATM R&D June, 20038 Quadrant Performance Area Objective Metrics Strategic Goals Macro Meso Micro Subjective M. Objective M. Domain Specific Tool Specific Internal External Summarized Framework

9. CSSI, Inc. EUROCONTROL/FAA ATM R&D June, 20039 Important Issues Terminology, precise definitions Comparability of results Relationship between local and global metrics from the same category (aggregation) –departure delay + airborne delay = arrival delay –Allows rollup of performance from measured quantities to higher-level objectives Relationship between metrics from different categories –capacity vs. delay or throughput vs. delay –safety vs. efficiency Framework for concept and research design

10. CSSI, Inc. EUROCONTROL/FAA ATM R&D June, 200310 Addressing Issues Unified framework for ATM performance measurement Shared database of metrics conformant with framework –Precise definitions –Unique terms –International agreement Provides: –Ability to compare/understand results & differences –Ability to roll-up results –Improved communication and collaboration

11. CSSI, Inc. EUROCONTROL/FAA ATM R&D June, 200311 Current Activities Assemble all metrics defined under different frameworks Identify duplicate metrics Identify metrics having same names but different meanings Adopt precise definition and domain for each metric Objectives/performance areas –create final list of objectives (mutually exclusive?) –research the best representative metrics –research the relationships between objectives Create draft framework