1. 1 He Says vs. She Says Model Validation and Calibration Kevin Chang HNTB Corporation kchang@hntb.com

2. 2 Model Validation and Calibration Keys to a Successful Simulation Model Model Validation Concept Stories on Model Validation Lessons Learned Contents

3. 3 Model Validation and Calibration CALIBRATION – an iterative procedure to fine tune model parameters and settings so that the model can achieve what the modeler wants it to perform. VALIDATION – an analytical process to verify if the model’s behavior and output statistics can truly represent actual traffic system operations. PURPOSE – to have a valid simulation model that is able to generate representative numerical results that replicate traffic operations in the modeled network for analyses.

4. 4 Facts Model Calibration –Results may be limited by the tool used –Modeler’s knowledge of the simulation tool –Usually the most time consuming process Model and Model Validation –Law and Kelton (1991) : “a simulation model of a complex system can only be an approximation to the actual system.” –Pegden et al. (1995) : “no model can ever be absolutely correct”. “A model is created for a specific purpose, and its adequacy or validity can only be evaluated in terms of that purpose.” –Fu, M : “Model validation is more an art work than science.”

5. 5 KEYS TO A SUCCESSFUL MODEL Use the Right Tool Modeler’s knowledge on the –System: traffic environment, operations, controls, management, etc. –Tools used –Issues to be addressed Data availability –Usually the most critical element : availability and accuracy Model Validation –Right people to review validation results –Selection of validation objects and focus on objectives –Art of work –Sometimes good luck

6. 6 Model Validation Concept Traffic Flow Traffic Operations Actual System Highway/Freeway Network ?=?= Modeled System Output Statistics Model Logic - Vehicle movements and interactions - Traffic assignment, routing decision - Weaving, merging, lane change - Queuing and delay - Traffic controls and managements - etc. Input Data

7. 7 STORIES ON MODEL VALIDATION Inaccurate traffic demand for model inputs –Demand vs. flows (throughputs) Incompatible performance measures –Delays: definition and collection –LOS criteria –Average, maximum, 90 th percentile, etc. Inconsistent data –Data collected at different times, locations, methods, etc. Validating MOE against MEMORY –Usually best or worst scenario will be memorized –Always talk to the right person with field experience MOE selection –Quantifiable and collectable with a clear definition: queue length, delay vs. speed, link density

8. 8 LESSONS LEARNED Know your tools Clear about project issues, system environment, scope of work Always budget for data collection and analysis Select “right” MOEs for model validation and presentation Talk to the right person Upgrade hardware and update software – be sensitive to the time required to run your models and to get output stats Art of work and good luck