1. F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. Predictability and Uncertainty in Air Traffic Flow Management Len Wojcik, Josh Pepper, Kristine Mills June 2003 The contents of this document reflect the views of the author and The MITRE Corporation and do not necessarily reflect the views of the Federal Aviation Administration (FAA) or the Department of Transportation (DOT). Neither the FAA nor the DOT makes any warranty or guarantee, expressed or implied, concerning the content or accuracy of these views.

2. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 2 Work Presented Here is Related to Papers Presented at Previous Air Traffic Management (ATM) Research and Development Seminars ATM2000 –Presented agent-based model of traffic flow management (TFM) operations –Showed over-congestion effects (“tragedy of the commons”), even with perfect information available to all airlines, and the economic role of the ATM authority ATM2001 –Applied agent-based model to TFM decision making with imperfect information –Showed a tradeoff between average performance (across many events) and predictability in TFM operations; maximum average system performance occurs where the performance in a single event is highly unpredictable ATM2003 (presented here) –Analyzed actual TFM decision making across more than 400 events –We were not able to distinguish different strategic TFM decisions based on system performance metrics, but there are patterns at other levels of the TFM process –The TFM system is highly adaptive at a tactical level, and the ability to learn from past experience is limited to specific elements of the TFM process –Some unpredictability in TFM performance metrics results from adaptation at a tactical level, which may help improve overall performance on average

3. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 3 The United States (U.S.) TFM Collaborative Decision Making Process Includes Information Exchange, and Both Strategic and Tactical Decision Making by Airlines and the Federal Aviation Administration (FAA) Airline 1: may cancel, delay, combine, exchange, reroute flights according to system rules Airline 2: may cancel, delay, combine, exchange, reroute flights according to system rules Airline N: may cancel, delay, combine, exchange, reroute flights according to system rules... FAA may declare: Ground Delay Programs (GDP), Ground Stops (GS), Playbook Reroutes* Information exchange Flights Airport/airspace affected by weather Weather predictions Actual weather *Note: A GDP or GS affects flights scheduled to arrive at a particular airport GDP is strategic (usually at least 2 hours ahead) GS is usually tactical (usually effective immediately) Playbook reroutes are strategic

4. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 4 Uncertainty Must be Taken Into Account in Order to Learn from Experiences in TFM A conceptual process for learning from previous TFM events (in the U. S. collaborative decision making system): * *Strategic decisions are those made at least 2 hours in advance of the expected weather (e.g., GDP, playbook reroutes) **Tactical decisions are those made less than 2 hours in advance of the expected weather ** Feedback, taking into account the information available at time of decision

5. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 5 We Decided to Analyze Past TFM Events on Bayesian Networks Why? –Bayesian networks encode probability relationships between variables on a causal network, which is exactly what we need to do to account for uncertainty –Fairly extensive data has become available across many TFM events –The decision-making process is well defined for particular types of TFM events –Where data is missing, Bayesian networks can be populated with subjective inputs or inputs based on modeling and simulation results –Bayesian networks can be updated with new data as it becomes available So, we thought that Bayesian networks would be suitable for analyzing past TFM events

6. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 6 We Looked at Two Types of TFM Events of Key Economic and Operational Importance in the U. S. Pennsylvania (PA) en route events –Affects flights to and from airports in northeast U. S. –Arrivals have priority over westbound departures; can create “departure backlogs” at northeast airports –“Low probability” weather forecasts are common –Strategic FAA decisions include GDP for Severe Weather Avoidance Program (SWAP) and playbook reroutes –We examined 338 days of events during years 2000 and 2001 (prior to September 11, 2001 (9/11)) Chicago O’Hare (ORD) airport events –Both convective and non-convective (especially ceilings and winds) weather affect operations –Strategic FAA decisions include GS, GDP –We examined 91 days during year 2000

7. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 7 PA En Route Events Affect Flights to and from Airports in Northeast (NE) U.S. PA/New York en-route airspace Incoming flights to NE airports (priority) Outgoing flights from NE airports Playbook reroutes (north or south) CDR (coded departure routes) reroutes (north or south) NE airports Airports west of PA/New York en route airspace Cost-related metrics: Air delays, ground delays, diversions, cancellations, airport backlog, propagated effects FAA strategic decisions: GDPs into NE airports Playbook reroutes Wait, or do nothing FAA tactical decisions: GS’s into NE airports Departure delays from NE CDR reroutes from NE Airline decisions: Cancellations Delays Combos/substitutions Diversions Weather

8. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 8 We Started with Simple Bayesian Networks (in Netica) for PA En Route TFM Events We found no meaningful pattern relating performance to TFM decisions

9. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 9 We Then Tried Higher-Fidelity Bayesian Networks Representing the TFM Process for PA En Route Events Netica output

10. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 10 But We Found No Meaningful Causal Pattern Relating Performance to TFM Actions in PA En Route Events TFM actions Affected airport Out-to-off delay performance Number of time periods with PA en route wx impact

11. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 11 We Did Find Trends Across ORD Events (E.g., Ground Delays Increase with Weather Severity) Note that the spread is large across events

12. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 12 There is an Upward Trend with Weather Severity Across Many Cost-related Parameters in ORD TFM Events Incoming ground delays Number of flights heldNumber of diversions Minutes of departure backlog Percentage of flights cancelled Individual events typically can be understood better with detailed analysis; decision makers adapt differently in different events

13. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 13 We Implemented a Bayesian Network for ORD TFM Events, but Found No Meaningful Pattern Relating Performance to TFM Decisions Netica output

14. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 14 However, Our Work Suggests We Can Learn from Analysis of More Specific Elements of the TFM Decision Process Statistical analysis shows how the likelihood of weather with aviation impact depends strongly on both forecast and weather at the time of forecast Analysis shows how well the GDP SWAP executed under different initial conditions *High GDP/GS overlap suggests GDP did not work well **High score at GDP start indicates greatest weather impact ***

15. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 15 So What’s Going On? Here is a Plausible Interpretation Many factors influence system behavior; these are highly variable across different events The system adapts to sources of unpredictability through tactical decisions within each TFM event, and performance metrics measured at the end of this process are highly variable across different events The system is not “chaotic” or “out of control” –Individual events can be understood better with detailed analysis –There are meaningful relationships between measures of specific elements of the TFM process –And, there are correlations with weather severity across five different system performance metrics Tactical adaptation by airline and FAA decision makers contributes to overall unpredictability observed in system metrics –This source of unpredictability may help improve overall system performance

16. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 16 Taxonomy of Large-Scale Unpredictability in National Airspace System (NAS) Operations

17. 6/14/2016 F066-B03-015 © 2003 The MITRE Corporation. All Rights Reserved. 17 Implications of Our Work Although today’s TFM system is highly adaptive, the ability to learn from past TFM experience is limited to specific elements of the TFM decision process –Analysis of specific elements of the TFM process, rather than comprehensive analysis of strategic TFM decision making, is the most viable approach to learn from past experience Adaptation at a tactical level by airlines and the FAA helps maintain overall system performance while contributing to overall system unpredictability –Analysis and improvement of specific elements of the TFM process can be helpful to improving system performance Future TFM concept research should include work on understanding sources of unpredictability –Understanding sources of unpredictability may be helpful to achieving major improvements in system performance