EUROCONTROL/FAA ATM R&D June, Standardizing Performance Metrics Almira Williams and Stéphane Mondoloni, Ph.D. for Tamara Breunig CSSI, Inc. Washington D.C. Diana Liang and Steve Bradford FAA Washington D.C. CSSI, Inc.

EUROCONTROL/FAA ATM R&D June, Outline Why metrics and standardization Framework research Inconsistencies in researched frameworks –Strategic goals –Objectives –Performance areas –Metrics Summarized framework Important issues Current activities

CSSI, Inc. EUROCONTROL/FAA ATM R&D June, Efforts Towards Standardization Why are metrics significant? –Means of measuring against strategic goals –Indication of system’s performance and trends –Indication of performance change with changing conditions Why standardize metrics? –Establish harmonized terminology –Avoid ambiguous metrics –Avoid misrepresentative metrics –Ease of comparing concepts (exactly “apples with apples”)

CSSI, Inc. EUROCONTROL/FAA ATM R&D June, Approach Basic Layered Framework Investigated Programs & Reports Classification of Objectives & Metrics Expanded Framework

CSSI, Inc. EUROCONTROL/FAA ATM R&D June, Framework Research Federal Aviation Administration (FAA) –Operational Evolution Plan (OEP) –Airspace Management Handbook (AMH) Eurocontrol –Cooperative Actions of Research and Development in Eurocontrol (CARE ) System Performance Metrics (WP 2) Human Performance Metrics (WP 3) –Performance Review Committee (PRC) –Required Total System Performance (RTSP) –TORCH (Technical, Economical and Operational Assessment of an ATM Concept Achievable from the Year 2005) National Aeronautics and Space Administration (NASA) –Advanced Air Transportation Technologies (AATT) –Virtual Airspace Modeling and Simulation (VAMS)

CSSI, Inc. EUROCONTROL/FAA ATM R&D June, Basic Framework Strategic Goals Objectives Metrics Generic Characteristics ~ Metric Categories Objective vs. Performance Area Scope of the Project ~ Outcomes Performance Indicators

CSSI, Inc. EUROCONTROL/FAA ATM R&D June, Performance Areas Summary of Objectives Access ATM Community Participation Availability Capacity Civil-Military Cooperation Cost Effectiveness Delay Economics Efficiency Environment Equity Expectations Flexibility Flight Efficiency Global Interoperability Human Involvement Predictability Productivity Military Access Quality Safety Security and Defense Security of Air Transport System Operational Performance Transition Uniformity

CSSI, Inc. EUROCONTROL/FAA ATM R&D June, Quadrant Performance Area Objective Metrics Strategic Goals Macro Meso Micro Subjective M. Objective M. Domain Specific Tool Specific Internal External Summarized Framework

CSSI, Inc. EUROCONTROL/FAA ATM R&D June, Important Issues Terminology, precise definitions Comparability of results Relationship between local and global metrics from the same category (aggregation) –departure delay + airborne delay = arrival delay –Allows rollup of performance from measured quantities to higher-level objectives Relationship between metrics from different categories –capacity vs. delay or throughput vs. delay –safety vs. efficiency Framework for concept and research design

CSSI, Inc. EUROCONTROL/FAA ATM R&D June, Addressing Issues Unified framework for ATM performance measurement Shared database of metrics conformant with framework –Precise definitions –Unique terms –International agreement Provides: –Ability to compare/understand results & differences –Ability to roll-up results –Improved communication and collaboration

CSSI, Inc. EUROCONTROL/FAA ATM R&D June, Current Activities Assemble all metrics defined under different frameworks Identify duplicate metrics Identify metrics having same names but different meanings Adopt precise definition and domain for each metric Objectives/performance areas –create final list of objectives (mutually exclusive?) –research the best representative metrics –research the relationships between objectives Create draft framework