1. Materials for Lecture 18 Chapters 3 and 6
Chapter 16 Section 4.0 and 5.0
Lecture 18 Validation Tests.xlsx
Next 4 slides are added because right about now most students are confused about PDF parameters and what functions to use

2. Parameter Estimation
Parameters for a distribution define the shape and position on the number scale
Uniform( Min, Max)
Norm( Mean, Std Dev)
Mean (Ỹ or Ῡ) and risk as Empirical( Si, P(Si))
Shape can be skewed right or left, can be tall or squatty (kurtosis)
Parameters reflect amount of variability in the stochastic variable
Must validate random variables against their parameters
We use the parameters to simulate the distributions

3. Same Mean Different Std Dev

4. Review Steps for Parameter Estimation
Step 1: Check for presence of a trend, cycle or structural pattern
If trend or structural model, work with the residuals (ẽt)
If no trend use actual data (X’s)
Step 2: Estimate parameters for several assumed distributions using the X’s or the residuals (ẽt)
Step 3: Simulate the different distributions
Step 4: Pick the best match based on
Mean, Variability -- use validation tests
Minimum and Maximum
Shape of the CDF vs. historical series
Penalty function CDFDEV() to quantify differences

5. Univariate Parameter Estimation
When do you use UPES?
When there is no trend in the data
When you want to use the historical mean as your forecasted y-hat
Test an unknown random variable for its shape
Or use residuals

6. Univariate Parameter Estimation
Empirical distribution fits your data best because it lets the data define the shape
Prefer to use the EMP with deviations as a percent or fraction from Y-hat
If there is a trend, then account for it with deviations from trend
Otherwise use deviations from mean
EMP allows us to model low probability events (uncertainty)
Test with =CDFDEV(original data, sim data)

7. Multivariate Parameter Estimation
Multivariate distribution SHOULD be used most of the time.
Our economic and production data have correlation or systematic relationships
If you ignore these forces results will be biased
Test the simulated variables to insure you captured the correlation
Test Correlation test

8. Next Step is Model Validation
You built a model, now does it work correctly?
Do the simulated values for the random variables reproduce their assumed parameters?
Does the model accurately forecast the system?
Do the results conform to theoretical expectations?
Do the results conform to expectations of experts?
Touring Test of simulation model results
Show the results to experts, using alternative assumptions about the input values

9. Four P’s for Validation
Planning – in the initial model preparation mode, developer should plan how to validate the model
Personal – it’s the developer’s responsibility to verify every equation, coefficient, and random variable; check if results are theoretically correct?
Peers – utilize experts in the field to review model results using Touring Test; use sensitivity testing of model
Prospective Clients – do the results conform to their expectations? Are the results useful to the client?

10. Model Verification Check all equations for arithmetic accuracy
Use Function F2 to make sure each cell uses the appropriate cells for their inputs
Use Excel’s “Trace Dependence” functions
Check linkage of variables coming into each equation
Check model in “Expected Value” and “Stochastic” mode
Do stochastic values make subsequent variables stoch?
Insure that the variables in each equation are theoretically correct (Signs are Correct)
Make sure the model contains all of the necessary equations to calculate the KOVs

11. Model Validation
Use statistical tests for each random variable to insure that it:
reproduces the historical distribution
reproduces the historical correlation matrix among random variables
Statistical Tests
Student t test – test the means
F test – test the variance
Chi Square test – test the standard deviation
Correlation test – test the correlation matrix

12. Statistical Tests for Validation
Test the means of the random variables against their historical values
Statistically equal at 95% level based on a t-test
Test the variance against historical values
Statistically equal at 95% level based on an F-test
Check the historical vs. simulated coefficient of variation
Needs to be constant over time, do this using eye ball test
Check the minimum and maximum
For a Normal distribution are they reasonable? Should be:
Min ≈ Mean + Std Dev * (-3)
Max ≈ Mean + Std Dev * (3)
For an Empirical distribution compare simulated min and max to values the model “should” simulate or
Xmin should get = Y-hat * (1+Minimum Fractional Deviate)
Xmax should get = Y-hat * (1+Maximum Fractional Deviate)
Check the correlation matrix for the simulated variables vs. the historical correlation matrix using t-tests

13. Validation Tests in Simetar
Statistical Validation Tests in Simetar
Hypothesis tests icon
Compare Two Series Historical Data vs. Simulated Values
1st Data Series is history
2nd Data Series is simulated
Test means and
variances for two series,
i.e., are they statistically equal
Test works for a pair of
variables and for
comparing two multivariate
distributions (matrices)

14. Statistical Tests for Validation
Compare Two Series Historical Data vs. Simulated Values
1st Data Series is history
2nd Data Series is simulated

15. Validation Tests in Simetar
Compare mean and standard deviation of simulated data to the user’s specified values
“Data Series” is the simulated values
Type in the mean or cell reference
Specify the Std Dev as a
value or a cell reference
Test is used when
Only mean and std dev
are known, i.e., there is
no history for the variable
Mean is a projected
Or assumed value which is different from the history

16. Validation Tests in Simetar
Compare mean and standard deviation of simulated data to the user’s specified values
Test is used when only mean and std dev
are known, i.e., there is no history for the variable
Or the mean is a projected value different from history
Note the Given Values are Mean = 10 and Std Dev = 3

18. Validation Tests in Simetar
Test simulated values for Multivariate Distributions (MVE and MVN) to test if the historical correlation matrix is reproduced in the simulation
Data Series is the simulated values for all random variables in the MV distribution, a matrix of variables in SimData
The original correlation matrix used to simulate the MVE or MVN distribution
OK, if the majority of correlation
coefficients are statistically the
same as the historical
correlation matrix

19. Charts for Validation
Test simulated values for Multivariate Distributions (MVE and MVN) to test if the historical correlation matrix is reproduced in the simulation results

20. Test Correlation for MV Distributions
Test simulated values for MVE and MVN distribution to insure the historical correlation matrix is reproduced in simulation
Data Series is the simulated values for all random variables in the MV distribution
The original correlation matrix used to simulate the MVE or MVN distribution

21. Validation Tests in Simetar
Student t Test is used to calculate statistical significance of simulated correlation coefficient to the historical correlation coefficient
You want the test coefficient to be less than the Critical Value
If the calculated t statistic is larger than the Critical value it is bold

22. Note on Testing Correlation
When you have no trend in the data and simulate a MV distribution (NORM or EMP)
Use the correlation matrix for the original data in Check Correlation test
When you de-trend the data for a MVEMP distribution
Use the correlation Matrix for the de-trended data in the Check Correlation test
When you use a structural model with residuals
Use a corrlation matrix for the residuals

23. Note on Testing Means
When you use the historical mean as your mean in the random variable
The t test in “Compare Two Series” will work without a problem, if you did the simulation correctly
When you use a forecasted mean that is different from the historical mean
The t test in “Compare Two Series” will NOT work, because your assumed mean differs from history
Use the “Test Parameters” test

24. Using Charts for Visual Validation
Use a CDF to compare historical series to simulated series, tests the min and max
Use a PDF to compare historical series to simulated series, tests the shape
Use a Box Plot to compare historical series to simulated series, checks the variability
Use a Probability graph to compare historical series to simulated series, P(x) vs. F(x)
Use a Fan graph to show the range of the risk and level of the mean over time, visual test of CV constant over time