1. David Schmitt, AICP With very special thanks to Hongbo Chi
Garbage In…Garbage Out? Do Inaccurate Inputs Cause the Inaccuracy & Bias in Transit Demand Forecasts?
David Schmitt, AICP
With very special thanks to Hongbo Chi
May 16, 2017

2. Optimistically biased forecasts fall below this line
Overview
Optimistically biased forecasts fall below this line
See 'A Transit Forecasting Accuracy Database – Beginning to Enjoy the “Outside View”' from 2015 Applications Conference for background
USA transit demand forecasts have been historically inaccurate and biased
We need to understand why these forecasts are inaccurate and biased
Inputs and exogenous forecasts given to forecasters have been optimistically biased

3. Motivations Heard at transit forecaster gatherings...
"Our demand forecasts were wrong because the land use assumptions didn’t come true"
"Our forecasts are always too high because of those inputs they give us to use"
Implications:
Accurate inputs  accurate demand forecasts
Accurate inputs  unbiased demand forecasts
Model specification, validation or other issues are not problematic
 Assess whether inaccurate & biased inputs resolve the inaccuracy & bias of demand forecasts
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
David Schmitt
May 16, 2017

4. Methodology
Begin with Transit Forecasting Accuracy Database: 66 projects, with forecasting inputs categorized by accuracy
Process:
Quantify the level of inaccuracy for each input/assumption
Compute change in forecasted demand by apply elasticity to corrected assumption
Compute adjusted demand forecast
Compute the adjusted forecast accuracy ratio:
Ratio = actual / forecasted ridership
Ratio < 1.00, optimistically biased
Ratio > 1.00, conservatively biased
Follows similar process used in “Pickrell Report”
Mean = 0.65 (n=66)
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
David Schmitt
May 16, 2017

5. Quantifying Input Inaccuracy
Each of the 10 project inputs and exogenous forecasts are placed into 1 of 5 categories:
Also, any differences between the forecast and actual ridership year are adjusted using national historical ridership growth
Project service levels
Employment estimates
Project travel time
Population estimates
Project fare
Supporting transit network
Economic conditions
Competing transit network
Auto fuel price
Roadway congestion
Category
Approximate Range
Inaccuracy Magnitude for Analysis
Well above assumed levels
25+%
+30%
10 to 25%
+18%
At assumed levels
-10 to +10% 0%
Below assumed levels
-25 to -10%
-18%
Well below assumed levels
< -25%
-30%
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
David Schmitt
May 16, 2017

6. Shaded cells reflect selected values
Elasticities
Elasticities
Show table with value and source
Used proxy values for some inputs
Note correction for differences between forecast & observed years
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
Shaded cells reflect selected values
David Schmitt
May 16, 2017

7. Results Average accuracy ratio: 0.65  0.74 (+14%)
Inaccurate inputs contribute to ( ) / 0.35 = 26% of demand forecast inaccuracy
73% of projects (48 projects) did not significantly change accuracy level
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
David Schmitt
May 16, 2017

8. Allowed to vary by ±15ppts
Test 1B
Weakness of methodology of Test 1:
Exact values of elasticities & values of input inaccuracy are unknown
Perform additional test: randomly vary elasticities and input variability for 10,000 iterations
Percentage of inaccuracy (by category)
Elasticity
Allowed to vary by ±15ppts
Allowed to vary by ±0.3
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
David Schmitt
May 16, 2017

9. Mean: 33% Median: 31% Range: 9%, 78% AveDev: 8%
These chart shows the same results as on the previous slide, but the x-axis is changed. It now reflects the percentage of project demand inaccuracy related to input inaccuracy. For example, the mean here is 33%, which means that correcting the inputs reduces the demand inaccuracy by one-third.
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
David Schmitt
May 16, 2017

10. Test #2
Hypothesis: inputs may explain ‘expansion’ project inaccuracy more than for ‘starter’ projects
Uncertain reactions to new modes lowers model’s ability to provide accurate demand forecasts
 Input inaccuracies should more fully describe demand inaccuracy for ‘expansion’ projects
Projects divided into starter project (n=31) and expansion project (n=35) groups
Re-ran 10,000 simulations for each group allowing elasticities and input inaccuracy values to vary
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
David Schmitt
May 16, 2017

11. Mean: 42% Median: 41% Range: 16%, 88% AveDev: 9%
Results indicate input inaccuracy describes ~22% of demand inaccuracy for starter projects, ~42% for expansion projects
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
David Schmitt
May 16, 2017

12. Observations
Test
Test 1
Test 1B
Test 2
‘Expansion’ Projects
‘Starter’ Projects
Original Accuracy Ratio
0.65
Adjusted Accuracy Ratio
0.74
% demand inaccuracy ‘explained’ by inaccurate inputs
26%
25% - 41%
33% - 51%
14% - 30%
No test confirmed the anecdotal explanations for demand forecast inaccuracy or bias
Input inaccuracies do not appear to explain demand forecast bias
Input inaccuracies ‘explain’ less than 50% of forecast demand inaccuracy
Evidence suggests that other causes of demand inaccuracy and bias exist
Knowledge of travel patterns on modes already in operation within the region seems to heighten the impact of inputs on demand forecast accuracy
Reliable inputs lead to “garbage out”
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
David Schmitt
May 16, 2017

13. Thank you! David Schmitt, AICP daves1997@gmail.com
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
David Schmitt
May 16, 2017

14. References David Schmitt May 16, 2017
Schmitt, David. “Beginning to Enjoy the 'Outside View': A First Glance at Transit Forecasting Uncertainty and Accuracy Using the Transit Forecasting Accuracy Database”
Taleb, Nassim Nicholas. Fooled By Randomness: The Hidden Role of Chance in Life and in the Markets: Second Edition. Random House, iBooks.
Taleb, Nassim Nicholas. Antifragile: Things That Gain from Disorder. Random House, iBooks.
Taleb, Nassim Nicholas. The Black Swan: Second Edition. Random House Trade Paperbacks, iBooks.
Transportation Research Board. TCRP Report 95: Traveler Response to Transportation System Changes
U.S. Department of Transportation: Federal Transit Administration. Before-and-After Studies of New Starts Projects [annual reports to Congress]
U.S. Department of Transportation: Federal Transit Administration. Predicted and Actual Impacts of New Starts Projects: Capital Cost, Operating Cost and Ridership Data. September 2003.
U.S. Department of Transportation: Federal Transit Administration. The Predicted and Actual Impacts of New Starts Projects : Capital Cost and Ridership. April 2008.
U.S. Department of Transportation: Transportation Systems Center. Urban Rail Transit Projects: Forecast Versus Actual Ridership and Costs. October 1989.
U.S. Department of Transportation: Federal Transit Administration. Travel Forecasting for New Starts: A Workshop Sponsored by the Federal Transit Administration. Phoenix and Tampa, 2009.
U.S. Department of Transportation: Travel Model Improvement Program Webinar: Shining a Light Inside the Black Box (Webinar I). February 14, 2008.
Victoria Transport Policy Institute. “Transit Price Elasticities and Cross-Elasticities”
Transit Forecasting Accuracy Database: Garbage In…Garbage Out?
David Schmitt
May 16, 2017