Case Study: FAA Air Traffic Data JMP Explorers Series Seminar February 6, 2013 Tom Donnelly, PhD Systems Engineer & Co-insurrectionist JMP Federal Government Team

Outline Discuss the Problem Key Steps Fit and compare models Summary Get the data Augment original data Shape the data for visualization and modeling Split data into Train, Validate/(Tune) and Test subsets Fit and compare models Partition vs. Neural vs. 2nd-Order Polynomial With and Without Federal Holiday factor in the model Summary GOAL… “Data > Information > Knowledge > Understanding > Action”

Problem and Situation FAA already using a classification and regression tree tool to model air traffic to better understand staffing needs While comparing results to this tool during evaluation, JMP’s answers were sometimes different. “Why?” Initial FAA analyses done separately for each Air Route Traffic Control Center (ARTCC Facility) and for one year only – resulting in 23 files of 365 rows of data Initial FAA focus was on using Weekday and Month as factors with which to model Total Traffic By using seven years of data we can also learn about the effect and trends of Fiscal Year Using 7 years of data for 21 ARTCCs (53,697 data rows) makes it easier to study the effect of Federal Holidays

Download Air Traffic Data From FAA Site

Download ARTCC Data From FAA Site

Jacksonville (ZJX) vs. Albuquerque (ZAB) ? ZAB

Include Modeling of 10 Federal Holidays Symbol 1 K W M 4 L C V T X

Augment Original Data Use Tables Menu to: Join Holiday information Join Latitude/Longitude Information Stack Air Traffic Type Use Formula to extract and/or expand information from original m/d/y formatted Date column Day of Week, Month, Year Fiscal Year – use conditional & comparison formulas Use Value Ordering to make better sense of ordinal data Weekday, Month, Fiscal Year and Federal Holidays

Make the Visualization Dynamic Use Data Filters and Distributions to select on the fly what to view Use Column Switcher to compare model predictions in Graph Builder

Randomly Assign Data into 3 Groups – Train, Validate (Tune), and Test (60%/20%/20%)

Actual vs. Predicted Total Traffic for ALL Data in Test Subset

Actual vs. Predicted Total Traffic for NO Fed Holidays in Test Subset

Actual vs. Predicted Total Traffic for ALL Fed Holidays in Test Subset

Actual vs. Predicted Total Traffic for Thanksgiving Holiday in Test Subset

Actual vs. Predicted Total Traffic for Christmas Holiday in Test Subset

Actual vs. Predicted Total for October 2012 w & w/o “Sandy” Data for 29th, 30th & 31st

Actual vs. Predicted Total Traffic - Partition - for 10 Federal Holidays by Holdback Subset

Actual vs. Predicted Total Traffic – Neural - for 10 Federal Holidays by Holdback Subset

Actual vs. Predicted Total Traffic - 2nd Order - for 10 Federal Holidays by Holdback Subset

Fit Models Using Honest Assessment Method i. e Fit Models Using Honest Assessment Method i.e. Train, Validate(Tune) and Test Subsets Use JMP’s Random Indicator to assign subsets In this case used a 60/20/20 split Original work used just one year of data and fit each facility separately. This meant “outliers” especially Federal Holidays were only represented in one subset – making it impossible to take their behavior into account Using seven years of data makes these Federal Holidays randomly fall into all three groups – making it possible to account for their behavior Using seven years of data also makes it possible to model effect of Fiscal Year on Total Traffic.

Potential Enhancements to Analysis Model non-Federal holidays Model the holidays people really observe Model SQRT(Total Traffic) Normalize variance – a usual regression assumption Prevent nonsensical predictions – negative traffic! Model Type of Traffic separately or combined Model impact of major storms

Prediction Profiler and Actual Values at 5 ARTCCs for Thanksgiving Day Test Data (Holdback = 2) for Fiscal Year 2010 ARTCC Total Traffic ZAB 2126 ZAN 683 ZAU 3699 ZBW 2879 ZDC 4265

Prediction Profiler Beside 3-D Response Surface

Bubble Plot

Q & A