Calibration and Validation Simulation calibration and validation • Microscopic traffic calibration and validation

Issues in Simulation Issues in Model Development – Verification – Validation • Issues in Simulation use – Calibration

Simulation Development Process • Problem structuring and formulation • System and simulation specification • Data collection and input modeling • Model formulation and construction • Verification and validation • Experimentation and analysis • Implementation

System and Simulation Specification Typical specification includes: – Simulation objectives – System description and modeling approach – Animation – Model input and output

Simulation Study Steps

Verification • Verification is a procedure to ensure that the model is built according to specifications and to eliminate errors in the structure, algorithm, and computer implementation of the model. • In other words: – Coding the model right, or – Debugging the model

Validation A process by which we ensure that the model is suitable for the purpose for which it is built – in other words: building the right model – all we have to do is check that the model behaves as the real-world does under the same conditions.

Verification and Validation • System  Model  “Code” • Validation: Is Model = System? • Verification: Is “Code” = Model?” (debugging)

Verification Build and verify small sub-models • Test with various combinations of input parameter values • Use debugging tools: trace, step-by-step execution, watch points, breakpoints, etc • Observe the animation (if available)

Validation Based on the collection of reliable data • Using existing data to see if model outputs match system performances – Use correct statistical analysis of the data: t-test, paired t-test, etc. • Testing the model’s ability to predict the future behavior of the real system – under known historical circumstances – under new circumstances

Validation Techniques – Inspections – Turing test – Cause-effect graphing – Interface analysis – Object-flow testing – Sensitivity analysis – Special input testing – Statistical techniques – Visualization/animation

Is it a valid Model? • What it means, if a simulation model is valid? • Validation process depends on: – The complexity of the system – Whether the system exists • There is no such thing as absolute model validity – The more time you spend on model development, the more valid it should become • Model that is valid for one purpose may not be for another • Validation is not something you do after the simulation model has been developed

Issues in Using Simulation Calibration Validation Multiple runs Results presentation

Calibration Procedure • Builds validity by verifying that simulation results match the field observation • The calibration can be achieved by adjusting – Input data – Default values, and – Model embedded data Until the simulation results replicates the field conditions

Multiple Runs • Multiple runs using different random number seeds to model stochastic behavior • Fixed random number stream typically used for model development and verification (for ease of debugging and testing) • Multiple runs most critical in near or over saturation conditions • Use multiple run statistics for final results (instead of those from one run)

Microscopic Traffic Simulation Calibration Several levels • Calibrate the network – Code links and intersections/junctures to accurately represent and reflect existing conditions • Calibrate vehicle/driver performances – Adjust vehicle/driver characteristics to make the model more accurately reflect existing conditions • Calibrate model parameters – Car-following – Lane change – Gap acceptance – Etc.

Model Calibration Use default parameters first • Check code errors first! • Adjust the model embedded data – The default values embedded typically have been validated against real world results. They are the results of model development and in many times they are accurate and sufficient – Calibration is most important when the project has unique characteristics • Area driver population • Special operation • Congested network • Etc.

Surface Street Calibration example Field measurements on surface street – Vehicle throughput – Travel times – Vehicle delay – Queue lengths

Surface Street Calibration example • Calibration parameters on surface street/intersection – Desired free flow speed – Queue discharge headway – Start-up lost time

Surface Street Calibration example • Factors affecting free flow speed on a link – Intersection spacing – Speed limit – # lanes – Lane width – Parking – Mid-block pedestrians – Land use access – Availability of turning bays – Etc.

Surface Street Calibration example Factors affecting start-up lost time – Perception/reaction time – Driver attentiveness – Driver caution for red-light running, pedestrian violations, etc. – Driver familiarity with the area

Calibration Example: CORSIM • Comprehensive parameters (network/driver/driver) – Free flow speed – Start-up lost time/queue discharge headway – OD and routes (u-turns, circling, etc…) – Vehicle composition – Driver composition

Calibration Example: CORSIM Geometric conditions – Warning signs to: • HOV • Off-ramp • Incident Blockage • Lane Drop/Add – Intersection geometry (turning bay length etc.) – Entry link length

Calibration Example: CORSIM • Vehicle/driver related parameters – Car-following – Lane change – Driver familiarity with routes – % “cooprative” drivers – Probability joining spillback

Calibration Example: CORSIM Traffic control related parameters – Jumpers/sneakers – Acceptable gaps • permitted left • right turns (RTOR) • crossing at stop sign

Calibration Example: CORSIM Calibration for congested network – Adjust percentages of vehicles that join spill-backs – Check the proper lengths of turning lanes – Change acceptable gaps for turning and crossing – Calibrate lane change parameters – Increase cooperative driver percentage – Increase percentage of vehicles that are familiar with the network – Adjust driver aggressiveness – Place warning sign further upstream – Etc.

Calibration Example: CORSIM

Calibration Example: CORSIM

Calibration Example: CORSIM

Calibration Example: CORSIM

Model Validation Goal: Model agrees with system • Systematic validation • Multiple level: – From aggregated average to microscopic • Multiple tools – Using animation very critical • High quality data very critical