1. Pitfalls

2. No problem definition "We have many problems in our line of business. Just start building a simulation model of our business while we figure out which questions we want to be answered." When the questions arrive (if at all answerable by simulation), the model cannot be used. Too much details: complex simulation model many parameters to assess many errors to correct lengthy simulation runs Too few details: not enough information at best only partial answers

3. One-sided problem definition "What investments do I have to make to keep my customers satisfied?" Clarify: possible investments, quality of service, assumptions about demand. Both client and consultant should understand and agree. make formal and explain! Problem definition = simulation contract. understand

4. Hidden assumptions Many assumptions are made for a simulation study. Keep track of assumptions made. "In DCT case, total crane service time of an X-type truck succeeding an Y-type truck is normally distributed with average a(X,Y) and standard deviation s(X,Y)." If possible, validate (compare with recorded trace). In all cases: mention in report. Hidden assumptions: used in model but not reported. Hidden assumptions may lead to costly mistakes!

5. Force fit the model Tandem queue arrival: expo(1), p1, p2: triangle(0.5,0.9,1.3) 10 subruns of 1000; initial run of 10 measured queue sizes p1: 1.71, p2: 0.42 First model: correct average but variance too small arrival: expo(1), p1, p2: triangle(0.7,0.8,0.9) measured queue sizes p1: 1.56, p2: 0.11 Fiddle with average to make model fit p1: triangle(0.71,0.81,0.91), p2: triangle(0.735,0.835,0.935) measured queue sizes p1: 1.78, p2: 0.42 Analyze 11% increased demand. 3.5/0.85 versus 4.6/3.9!

6. “Giving Statistical Packages to Engineers is like Giving Guns to Children,” Dick Mensing "Statistics don't lie, but liars use statistics" Darell Huff Improper use of statistics

7. Methodology pitfall Simulation = Experimentation. Design of experiments: established rules. Hypothesis before experiment. New hypothesis = new experiment. Hypothetical case: beer appreciation. Brewers have designed "X", a new low-cost beer. Hypothesis: appreciation of "X" not lower than other beers. Consumer appreciation is tested in various Dutch cities, relatively to "Amstel" and "Grolsch".

8. Beer appreciation Table of average appreciations per city. city X Ams Grol A'dam 6.5 7.2 6.9 R'dam 7.1 6.8 7.3 den Haag 6.3 6.5 7.1 Utrecht 5.7 6.1 6.2 E'hoven 7.1 6.9 7.0 Enschede 5.9 6.6 7.5 Gr'ingen 6.2 6.3 6.5 Tilburg 6.9 7.0 6.7 Haarlem 6.8 7.2 6.8 Breda 7.2 6.9 7.2 Nijmegen 7.0 7.1 7.3 AVG 6.616.786.95 cityXAmsGrol E'hoven7.16.97.0 Tilburg6.97.06.7 Breda7.26.97.2 AVG7.076.936.97 Statistical analysis: with 98% confidence, people from N-Brabant prefer (on average) "X" above the other two. Hypothesis rejected.

9. Beer marketing strategy Market "X" as a local beer in N-Brabant. Call it "Brabobier". Make a lot of money. What is wrong? With 98% confidence, people from N-Brabant prefer "X" above the other two. New hypothesis based on sample observation. Confidence cannot be assessed. New, independent experiment is needed.

10. Observe random sample Any recorded sample contains remarkable data. You need a second sample to confirm a hypothesis based on observing the first sample. Dice throwing experiment (Arena): 25 throws; sequence 2321116466156134222666665 Hypothesis: by throwing a dice 25 times, five equals in row will occur. Confirmed with 98.5% confidence (on given sample). five equals in a row! x(19)=...=x(24)=6: probability 0.00013 twenty possibilities for first: 0.0026 five equals (instead of sixes): 0.0156

11. Simulation analogon Problem in business process (e.g. DCT). Assess tentative solution through simulation. Hypothesis: performance indicator x after implementing the solution will be lower on average than in current situation. Boss needs 90% confidence in order to start implementing it. IndicatorAverageHalf Width x 4.180.79 x 3.890.85 current tentsol 30 subruns of 1000 hrs Test hypothesis: evaluate C T result: 0.73

12. Longer simulation IndicatorAverageHalf Width x 4.270.41 x 3.920.47 current tentsol 30 subruns of 4000 hrs IndicatorAverageHalf Width x 4.320.38 x 3.910.41 current tentsol 30 subruns of 6000 hrs hypothesis confidence: 0.87 outcome: 0.93 conclusion: with 93% reliability solution gives lower average x ?

13. Persistent experimentation Repeat an experiment until it satisifies hypothesis. Statistically analyze each experiment individually. Hypothesis corroboration with incorrect confidence! IndicatorAverageHalf Width x 4.320.38 x 3.910.41 current tentsol After 3 experiments goal of 90% confidence attained. A new, independent simulation run is needed to confirm! Choose different initial value for random generator and repeat the last experiment.

14. Cheating recipes Confirm hypothesis by simulation with 95% reliability. Do many independent simultation experiments. Only retain favorable ones. Claim you did not execute the unfavorable ones. Speeding up the process: Perform e.g. 50 iterations. Look for a most favorable sequence of 30, e.g. 7-36. Define iterations 0-6 as initial run. Perform 30 iterations. Round off favorably (model parameters and simulation results)

15. Abuse of statistics Recent uncovered evidence shows that nurse Lucia de B has been convicted of murder on the basis of statistical arguments only and by abusing just about every rule in the book. * Whether or not an incident was classified as suspect depended on whether or not Lucia was on duty (simply a question of checking which nurse is on duty and then asking enough doctors till you get a "suspicious" verdict. * Data was collected in this way till there was enough to condemn her. * A professor of statistics in law, and trained mathematician, does not know the meaning of one of the most basic statistical concepts - the p-value. (Statistician xxx multiplied three independent p-values in order to obtain a combined p-value). It somehow reminds me of the old method to see if someone is a witch - if they drown they were innocent, if they are guilty you can burn them.

16. Quote "The only way to have real success in science...is to describe the evidence very carefully without regard to the way you feel it should be." Richard Feynman