Transportation leadership you can trust. presented to Transportation Planning Applications Committee (ADB50) presented by Sarah Sun Federal Highway Administration David Kurth Cambridge Systematics, Inc. January 13, 2009 Travel Model Improvement Program Model Validation Improvement Efforts

1 Travel Model Improvement Program (TMIP) Goals Conduct outreach and education to advance the state of the practice Promote research on modeling Improve model quality

2 Travel Model Improvement Program Outreach Training Disseminating ideas Engaging stakeholders Facilitating research implementation

3 TRB Special Report 288

4 TRB Special Report 288 – Model Practice Shortcomings Too frequently found Inadequate data Optimism bias Quality control Validation errors

5 TRB Special Report 288 – Summary Recommendations Develop and implement new modeling approaches Ensure better practice Federal, state, regional collaboration

6 Travel Model Validation Practices Peer Exchange – May 9, 2008 In response to the recommendation of TRB Special Report 288 take steps to ensure better practice Sponsored by FHWA through TMIP Project manager – Sarah Sun Project staff – CS and TTI Exchange participating experts 9 from public agencies 2 from academia 4 from consulting 4 from Federal agencies (FHWA & FTA)

7 Peer Exchange Objective Use the experience of each panel member to provide practical advice and “action items” useful for moving the validation of travel models closer to “What it should be.” “…What should it be?”

8 Summary of Current MPO Practices Model Component Validation Independent data sparse Estimation and validation using household survey data CTPP used for home-based work validation Overall Model Validation Traffic counts Transit boardings Focus on statistics: R 2, % RMSE, % Error, VMT Error… (Documented) Sensitivity Tests Lacking Few temporal validations

9 Summary of Current MPO Practices Guidelines and “Standards” Used: Calibration and Adjustment of System Planning Models (1990) TMIP Model Validation and Reasonableness Checking Manual (1997) NCHRP Report 365 (1998) NCHRP Report 255 (1992) Various state guidelines

10 Peer Exchange Summary Assessment of Current Practices Unwarranted focus on attaining standards Valid = attain measure of accuracy Double-edged sword Perpetuates myths regarding predictive power of models Model validation and techniques are not context specific Consider “business process” Consider intended use of model & forecasts Planning for and documentation of validation is lacking ⇒ “You get the $$$”

11 Peer Exchange Summary Improved Validation Process Steps to verify the ability of the model system to make reasonable predictions over a range of development patterns, transportation operations, and external factors. Frank Koppelman

12 Peer Exchange Summary Improved Validation Process Model validation must be context specific Type of analysis considered Scope of analysis considered Agency business process being supported Model validation must consider the entire model Consider the validity of the validation data Document validation expectations and results

13 Peer Exchange Summary “7 Questions” How important is it to match base-year observations? Important, but not sufficient to establish validity How close is “close enough?” It all depends… What steps are needed to ensure accurate validation data? A conscious, planned effort Initiate when model estimations / updates are planned Budget necessary resources

14 Peer Exchange Summary “7 Questions” How should model sensitivity be tested? Forecasting & backcasting Test range of alternatives to determine validity Other tests to assess ability to forecast future travel? Peer review panels Analysis of components of change (à la FTA New Starts) Should risk analysis of future forecasts be performed? Depends on decisions being made…

15 Peer Exchange Summary “7 Questions” What is the role of validation documentation in raising model credibility? Necessary for understanding models, resulting in… −confidence in models −knowledge of the limitations of models

16 Model Validation Practices Peer Exchange White Paper See… ources/clearinghouse/brow se/list/6/1285http://tmip.fhwa.dot.gov/res ources/clearinghouse/brow se/list/6/1285

17 Travel Model Validation and Reasonability Checking Manual Draft In Preparation Incorporates White Paper suggestions All elements of current manual Expanded focus beyond trip-based Focus on temporal validation and sensitivity testing Model validation plan specification Identification of data requirements & budgeting Validation documentation recommendations

18 Travel Model Validation and Reasonability Checking Manual Chapters Introduction Model Validation Plan Specification Validating Model Inputs Socioeconomic Models Amount of Travel / Activity

19 Travel Model Validation and Reasonability Checking Manual Chapters Trip Distribution / Destination Choice / Location Choice Mode Choice / Auto Occupancy Time-of-Day Choice / Time-of-Day Split / Directional Split Factors Assignment Procedures Temporal Validation and Sensitivity Testing Validation Documentation

20 Update of the Travel Model Validation and Reasonability Checking Manual Published manual Web document Searchable & keyword hyperlinks Possible wiki format Publicize updated manual Conferences TMIP list Virtual Mentoring & Technical Support Center (VMTSC) Webinars

21 Information The Travel Model Improvement Program (TMIP) Suggestions, comments or good practices to share −Contact Sarah Sun at or (202)