Confidence Interval Estimation in System Dynamics Models Gokhan Dogan* MIT Sloan School of Management System Dynamics Group *Special thanks to John Sterman for his support

Automated Calibration (e.g. Vensim, Powersim) Motivation Calibration Manual Calibration Automated Calibration (e.g. Vensim, Powersim)

Motivation Once model parameters are estimated with automated calibration, next step: Estimate confidence intervals! Questions: -Are there available tools at software packages? -Do these methods have any limitations? -Are there alternative methods?

Why are confidence intervals important We reject the claim that the parameter value is equal to 0 (with 95% probability) We can’t reject the claim that the parameter value is equal to 0 (with 95% probability) 95% Confidence Interval 95% Confidence Interval Parameter Estimate θ Parameter Estimate θ

How can we estimate confidence intervals? Used in the System Dynamics Software (Vensim) /Literature The method we suggest for System Dynamics models Likelihood Ratio Method Bootstrapping Both methods yield approximate confidence intervals!

Likelihood Ratio Method The likelihood ratio method is used in system dynamics software packages (Vensim) and literature (Oliva and Sterman, 2001). It relies on asymptotic theory (large sample assumption).

However Likelihood Ratio Method (as it is used at software packages) assumes: At system dynamics models: -Large Sample -It is not always possible to have large sample -No feedback (autocorrelation) -There are many feedback loops -Normally distributed error terms -Error terms are not always normally distributed

Bootstrapping Introduced by Efron (1979) and based on resampling. Extensive survey in Li and Maddala (1996). It seems more appropriate for system dynamics models because - It doesn’t require large sample - It is applicable when there is feedback (autocorrelation) - It doesn’t assume normally distributed error terms

Drawbacks of bootstrapping The software packages do not implement it. It is time consuming.

Bootstrapping Fit the model and estimate parameters Compute the Error Terms

Bootstrapping uses resampling Nonparametric: Reshuffle Them and Generate many many new error term sets using the autocorrelation information Parametric: Fit a distribution and Generate many many new error term sets using the autocorrelation and distribution information

Resampling the Error Terms If we know that: - The error terms are autocorrelated - Their variance is not constant (heteroskedasticity) - They are not normally distributed => We can use this information while resampling the error terms Flexibility of bootstrapping stems from this stage

FABRICATED ERROR TERMS FABRICATED “HISTORICAL” DATA . . . . . . + =

FABRICATED “HISTORICAL” DATA . . . Fit the model and estimate parameters Parameter Estimate Parameter Estimate Fit the model and estimate parameters Parameter Estimate 500 Parameter Estimates Fit the model and estimate parameters

Distribution of a model parameter

Experiments We had experimental time series data from 240 subjects. Subjects were beer game players. For each subject we had 48 data points, so we estimated parameters and confidence intervals using 48 data points.

Model (Same as Sterman 1989) Ot = Max[0, θLRt + (1–θ)ELt + α(S' – St –βSLt) + error termt] Parameters to be estimated are θ, α, β, S‘

Likelihood Ratio Method Individual Results 95% Confidence Intervals for θ Likelihood Ratio Method 95% CI 1 0.77 θ=0.95 Bootstrapping 95% CI 1 0.01 θ=0.95

Individual Results 95% Confidence Intervals for β Likelihood Ratio Method 95% CI 0.2 Significantly Different From 0!!! β =0.01 Bootstrapping 95% CI 0.2 β =0.01

Overall Results Average 95% Confidence Interval Length   Theta Alpha Beta S-Prime Likelihood Ratio Method 0.19 0.11 13.20 Bootstrapping 0.67 0.30 0.52 973.59 Median of 95% Confidence Interval Length   Theta Alpha Beta S-Prime Likelihood Ratio Method 0.10 0.08 0.06 2.32 Bootstrapping 0.84 0.24 0.48 10.10

Overall Results Percentage of Subjects for whom the bootstrapping confidence interval is wider than the likelihood ratio method confidence interval Theta Alpha Beta S-Prime Bootstrapping CI wider than Likelihood Ratio Method CI 97.76% 98.81% 100% 98.56%

Likelihood Ratio Method vs Bootstrapping Is easy to compute Very fast BUT depends on assumptions that are usually violated by system dynamics models Yields very tight confidence intervals Bootstrapping: Is NOT easy to compute Takes longer time DOES NOT depend on assumptions that are usually violated by system dynamics models Yields larger confidence intervals. Usually more conservative.