Uncertainty in transport models IDA workshop 7th May 2014

Presentation outline Rationale Uncertainty in transport models How uncertainty is quantified Case study General comments Uncertainty in transport models 07-05-2014

Rationale Transport models consist of equations combining exogenous variables (input) and coefficient (parameters) that express how the model output depends on the exogenous variables (De Jong et al. 2007) Transport models play a prominent role in many decision making processes; the output they generate, transport demand, is a key input for a wide range of analyses including: Cost Benefit Analysis (CBA) / Multi Criteria Analysis (MCA), e.g.: urban development planning strategy, sustainable mobility polices evaluation, etc. financial analysis for new (tolled) infrastructures Thus, wrong transport demand (traffic) forecast might have considerable socio/economic consequences (e.g. misplacement of public resources) Uncertainty in transport models 07-05-2014

Rationale Transport models have been criticized for lack of accuracy; an extensive literature has shown that often there is a considerable inaccuracy between forecasted and observed traffic volumes Source: Flyvbjerg et al. 2006 Uncertainty in transport models 07-05-2014

Rationale The main sources of inaccuracy can be broadly divided between technical and political/decision process Inaccuracy Technical Uncertainty Political Project selection positive bias toward projects with high forecasted traffic Big projects, more difficult to assess, preferred to small ones Disagreement among stakeholders might cause delays making the model results not up to date Standard model used for a problem it is not developed for, then overlooking core causal relationships Wrong assumptions on the transport system, e.g. (i) expectations on how competitors would react to new fare and service (strategy ) (ii) other projects being postponed (network) Uncertainty in transport models 07-05-2014

Rationale Transport models have inherent uncertainty : reproduce complex systems generating transport supply (i.e. services, infrastructure, regulation) and transport demand (i.e. traffic) receive inputs from other complex system (models): economics, sociology, psychology, technology, etc. The consequence is that transport models output is unpredictable, so it is not possible to use a deterministic approach In fact uncertainty can be defined as “any departure from the unachievable ideal of complete deterministic knowledge of the relevant system” (Walker et al. 2003) “(...) modelled output is better expressed as a central estimate and an overall range of uncertainty margins articulated in terms of values and likelihood of occurrence” (Boyce 1999), rather than as a point value Uncertainty in transport models 07-05-2014

Uncertainty in transport models Model definition Uncertainty related to the definition of the boundaries of the system being modelled (context) e.g. the geographical size of the area of interest, future changes in the transport demand environment Model specification Uncertainty about the assumed model structure and the technical implementation of the model (e.g. the methodology chosen) Model calibration Uncertainty about input (data collected) and parameters (calibrated variables) Model validation Uncertainty propagated to the transport model output (sequential frameworks) Adapted from Leleur 2000 Uncertainty in transport models 07-05-2014

How uncertainty is quantified Different levels of uncertainty require/allow different uncertainty calculation techniques: Determinism Statistical uncertainty Scenario uncertainty Recognised ignorance Ignorance (Indeterminacy) 1) Scenario hypoteses + sensitivity tests Adapted from Walker et al. 2003 2) Confidence intervals + sensitivity tests 3) Stochastic sampling methods + sensitivity tests (Monte Carlo Simulation, Bootstrap) Uncertainty in transport models 07-05-2014

How uncertainty is quantified Deterministic approach Scenario analysis Confidence intervals Stochastic simulation Uncertainty in transport models 07-05-2014

Case study: Næstved model Uncertainty analysis (model input and parameters) through MCS combined with sensitivity analysis Analysis focuses on: uncertainty propagation pattern effects on output uncertainty of uncertainty location (input, parameters and total) effects on output uncertainty of network congestion three levels of generated traffic (1,1.5 and 3 times base traffic) Stochastic sampling (multivariate LHS,100 draws): variables mean values: from the model variability: Coefficient of Variation (CV) 0.1 and 0.3 (CV=St Dev /Mean) probability distribution: input (Triangular) parameters (Log-normal) Sensitivity tests (100 runs) Uncertainty in transport models 07-05-2014

Case study: Næstved model Uncertainty in transport models 07-05-2014

Case study: Næstved model Uncertainty in transport models 07-05-2014

Case study: Næstved model Uncertainty in transport models 07-05-2014

Case study: NTM forecasts Uncertainty analysis (model socio-economic input growth rate forecasts 2015/2025) through MCS combined with sensitivity analysis : population, employment, GDP (real) and fuel prices (both petrol and diesel) Analysis focuses on: uncertainty propagation pattern over time aggregated and for different input Stochastic sampling (multivariate LHS,100 draws): variables mean values: official forecasts variability: standard deviations calculated by 5 years intervals from time series probability distribution: Normal distribution This approach is meant to reflect an increased level of uncertainty throughout the forecasted period Uncertainty in transport models 07-05-2014

Case study: NTM forecasts Uncertainty in transport models 07-05-2014

Case study: NTM forecasts Uncertainty in transport models 07-05-2014

Conclusions Transport models output, demand of transport, is a key input for policy decision making processes; problems arise from the fact that transport models refer and describe complex systems, thus showing unpredictability in their output The final output of the transport models is no more a point value but a range of possible outcome (with a given probability to occur) Assessing uncertainty inherent to transport models is important to produce informative output (demand of traffic) However, analyzing uncertainty of a whole transport model requires a relevant work effort; the question then is how expensive are decisions based on too uncertain forecasts and how time-consuming is implementation of uncertainties analyses? Uncertainty in transport models 07-05-2014

Extra Transport models uncertainty has a temporal dimension and occur in both static and dynamic (forecast) modelling However, with respect to the influence on the model output, the prevalence of some uncertainty location relatively to the others is affected by the temporal and geographical dimension of the system modelled The influence of model and parameters uncertainty is higher in static or short run forecasts whilst context and input uncertainty influence increases in long run forecasts. The same happens for geographical dimension of the area modelled, respectively small or big size Uncertainty in transport models 07-05-2014

Extra Uncertainty in transport models 07-05-2014

Calibration (parameters) Extra Definition Programming Calibration (input) Calibration (parameters) Validation Generation Subdivision into categories? Number and size of zones? Which socio economic variables? Model type? (Category analysis, linear regression, Logit, Probit) How does data and zone structure match? When is data collected? How well does data fit? How many parameters? Estimated, calibrated parameters? Are data representative? Model output (propagated) uncertainty Distribution Which variables describe travel resistance? Time intervals? Peak hours etc. Pivot table? Size of zones? Model type? (gravity, growth factor, Logit, Probit) Travel resistance function? Linear travel resistance function? Which matrix adjustment? How old is data? Is the generalized cost matrix the result of feedback from traffic assignment? Estimated or calibrated? Which calibration method? Mode choice Which modes? Is travel resistance included? Which variables are used Is it 2nd or 3rd model step? Model type? (Logit, Probit, mixed Logit) Is travel resistance a result of traffic assignment? Which characteristics are included? Assignment Which variables are included in the utility function? Different trip purposes? Different link types and speed-flow functions? Number of iterations with feed back? Linear utility function? Correlation? Model type? (Logit, Probit, mixed Logit, nested Logit) Deterministic or stochastic? Capacity constraint? Solution algorithm? Level of detail? Which characteristics are associated with the links? Uncertainty in transport models 07-05-2014

Case study: Næstved model - Extra Uncertainty in transport models 07-05-2014

Case study: Næstved model - Extra Uncertainty in transport models 07-05-2014