1. Confidence Interval Estimation in System Dynamics Models
Gokhan Dogan*
MIT Sloan School of Management
System Dynamics Group
*Special thanks to John Sterman for his support

2. Automated Calibration (e.g. Vensim, Powersim)
Motivation
Calibration
Manual Calibration
Automated Calibration (e.g. Vensim, Powersim)

3. Motivation
Once model parameters are estimated with automated calibration,
next step: Estimate confidence intervals!
Questions:
-Are there available tools at software packages?
-Do these methods have any limitations?
-Are there alternative methods?

4. Why are confidence intervals important
We reject the claim that the parameter value is equal to 0 (with 95% probability)
We can’t reject the claim that the parameter value is equal to 0 (with 95% probability)
95% Confidence Interval
95% Confidence Interval
Parameter Estimate θ
Parameter Estimate θ

5. How can we estimate confidence intervals?
Used in the System Dynamics Software (Vensim) /Literature
The method we suggest for System Dynamics models
Likelihood Ratio Method
Bootstrapping
Both methods yield approximate confidence intervals!

6. Likelihood Ratio Method
The likelihood ratio method is used in system dynamics software packages (Vensim) and literature (Oliva and Sterman, 2001).
It relies on asymptotic theory (large sample assumption).

7. However
Likelihood Ratio Method (as it is used at software packages) assumes:
At system dynamics models:
-Large Sample
-It is not always possible to have large sample
-No feedback (autocorrelation)
-There are many feedback loops
-Normally distributed error terms
-Error terms are not always normally distributed

8. Bootstrapping
Introduced by Efron (1979) and based on resampling. Extensive survey in Li and Maddala (1996).
It seems more appropriate for system dynamics models because
- It doesn’t require large sample
- It is applicable when there is feedback (autocorrelation)
- It doesn’t assume normally distributed error terms

9. Drawbacks of bootstrapping
The software packages do not implement it.
It is time consuming.

10. Bootstrapping Fit the model and estimate parameters
Compute the Error Terms

11. Bootstrapping uses resampling
Nonparametric: Reshuffle Them and
Generate many many new error term sets
using the autocorrelation information
Parametric: Fit a distribution and
Generate many many new error term sets
using the autocorrelation and distribution information

12. Resampling the Error Terms
If we know that:
- The error terms are autocorrelated
- Their variance is not constant (heteroskedasticity)
- They are not normally distributed
=> We can use this information while resampling the error terms
Flexibility of bootstrapping stems from this stage

13. FABRICATED ERROR TERMS
FABRICATED “HISTORICAL” DATA
. . .
. . . + =

14. FABRICATED “HISTORICAL” DATA
. . .
Fit the model and estimate parameters
Parameter Estimate
Parameter Estimate
Fit the model and estimate parameters
Parameter Estimate
500
Parameter Estimates
Fit the model and estimate parameters

15. Distribution of a model parameter

16. Experiments We had experimental time series data from 240 subjects.
Subjects were beer game players.
For each subject we had 48 data points, so we estimated parameters and confidence intervals using 48 data points.

17. Model (Same as Sterman 1989)
Ot = Max[0, θLRt + (1–θ)ELt + α(S' – St –βSLt) + error termt]
Parameters to be estimated are θ, α, β, S‘

18. Likelihood Ratio Method
Individual Results
95% Confidence Intervals for θ
Likelihood Ratio Method
95% CI 1
0.77
θ=0.95
Bootstrapping
95% CI 1
0.01
θ=0.95

19. Individual Results 95% Confidence Intervals for β
Likelihood Ratio Method
95% CI
0.2
Significantly Different From 0!!!
β =0.01
Bootstrapping
95% CI
0.2
β =0.01

20. Overall Results Average 95% Confidence Interval Length
Theta
Alpha
Beta
S-Prime
Likelihood Ratio Method
0.19
0.11
13.20
Bootstrapping
0.67
0.30
0.52
973.59
Median of 95% Confidence Interval Length
Theta
Alpha
Beta
S-Prime
Likelihood Ratio Method
0.10
0.08
0.06
2.32
Bootstrapping
0.84
0.24
0.48
10.10

21. Overall Results
Percentage of Subjects for whom the bootstrapping confidence interval is wider than the likelihood ratio method confidence interval
Theta
Alpha
Beta
S-Prime
Bootstrapping CI wider than Likelihood Ratio Method CI
97.76%
98.81%
100%
98.56%

22. Likelihood Ratio Method vs Bootstrapping
Is easy to compute
Very fast
BUT depends on assumptions that are usually violated by system dynamics models
Yields very tight confidence intervals
Bootstrapping:
Is NOT easy to compute
Takes longer time
DOES NOT depend on assumptions that are usually violated by system dynamics models
Yields larger confidence intervals. Usually more conservative.