1. Hypothesis Testing for Simulation 1 hypothesis testing with special focus on simulation

2. Hypothesis Testing for Simulation 2  Hypothesis Test answers yes/no question with some statistical certainty  H 0 = default hypothesis  is a statement  H a = alternate hypothesis  is the precise opposite

3. Hypothesis Testing for Simulation 3  X = test statistic (RANDOM!)  sufficient (uses all avail. data)  often Z, T, N are used as notation  F X = its probability distribution   = P[reject H 0 | H 0 true]

4. Hypothesis Testing for Simulation 4  c  = critical region for    = P[X in c  | H 0 ]   is our (controllable) risk

5. Hypothesis Testing for Simulation 5 TWISTED LOGIC  We WANT to reject H 0 and conclude H a, so...  We make  very small, so...  If we can reject, we have strong evidence that H a is true  This construct often leads to inconclusive results  “There is no significant statistical evidence that...”

6. Hypothesis Testing for Simulation 6 IMPORTANT  Inability to reject <> H 0 true

7. Hypothesis Testing for Simulation 7 POWER OF THE TEST   = P[X not in c  | H a ]  1- = P[correctly rejecting]

8. Hypothesis Testing for Simulation 8 VENACULAR   is type I error  Probability of incorrectly rejecting   is type II error  Probability of incorrectly missing the opportunity to reject

9. Hypothesis Testing for Simulation 9 UNOFFICIAL VENACULAR  type III error – answered the wrong question  type IV error – perfect answer delivered too late

10. Hypothesis Testing for Simulation 10 EXAMPLE!  Dial-up ISP has long experience & knows...

11. Hypothesis Testing for Simulation 11 We get DSL, observe 12 samples

12. Hypothesis Testing for Simulation 12 IS DSL FASTER?  H 0 :  DSL = 50  H a :  DSL < 50  test with P[type I] = 0.01

13. Hypothesis Testing for Simulation 13 PROBABILITY THEORY  Z ~ t n-1  Must know the probability distribution of the test statistic IOT construct critical region

14. Hypothesis Testing for Simulation 14  for n = 12,  = 0.01, c  = -2.718 99% of the probability above -2.718

15. Hypothesis Testing for Simulation 15 our test statistic -2.33

16. Hypothesis Testing for Simulation 16  0.021 called the p-value  Given H 0, we expect to see a test statistic as extreme as Z roughly 2% of the time. -2.718 (0.01) -1.796 (0.05) -2.33 (0.021)

17. CONFIDENCE INTERVALS Hypothesis Testing for Simulation 17 ll uu  For a given   P[l  <=  <= u  ] = 1-  Based on the sample So they are RANDOM!

18. Hypothesis Testing for Simulation 18 GOODNESS-OF-FIT TEST  Discrete, categorized data  Rolls of dice  Miss distances in 5-ft. increments  H 0 assumes a fully-specified probability model  H a : the glove does not fit!

19. Hypothesis Testing for Simulation 19 TEST STATISTIC “chi-squared distribution with gnu degrees of freedom”

20. Hypothesis Testing for Simulation 20  = observations - estimated param  Did you know... if Z i ~N(0, 1), then Z 1 2 + Z 2 2 +...+ Z n 2 ~  n 2

21. Hypothesis Testing for Simulation 21 CELLS  H 0 always results in a set of category cells with expected frequencies  EXAMPLE  Coin is tossed 100 times  H 0 : Coin Fair

22. Hypothesis Testing for Simulation 22 CELLS AND EXPECTED FREQUENCIES EXPECT H50 T

23. Hypothesis Testing for Simulation 23 EXAMPLE  Cannon places rounds around a target  H 0 : miss distance ~ expon(0.1m)  Record data in 5m intervals  (0-5), (5-10),...(25+)

24. Hypothesis Testing for Simulation 24 EXPONENTIALS E(X)=1/

25. Hypothesis Testing for Simulation 25 RESULTS RIGHTOBS1-exp(-0.1x)PROBEXPECT(OBS-EXPECT)^2 0.00 5.00300.39 39.352.22 10.00170.630.2423.871.97 15.00210.780.1414.472.94 20.00110.860.098.780.56 25.00110.920.055.336.05 30+101.000.088.210.39 100.00 14.14

26. Hypothesis Testing for Simulation 26

27. Hypothesis Testing for Simulation 27 TEST RESULTS  Degrees of Freedom  6 cells  0 parameters estimated  = 6  For the  6 2 distribution, the p- value for 14.14 is about p=0.025  REJECT at any  > 0.025

28. Hypothesis Testing for Simulation 28 DIFFERENT H 0  H 0 : the miss distances are exponentially distributed  H a : the exponential shape is incorrect  We estimate the parameter, we lose one degree of freedom

29. Hypothesis Testing for Simulation 29 RESULTS 2 LEFTRIGHTOBS1-exp(-0.0738x)PROB EXPE CT(OBS-EXPECT)^2 0.00 5.00300.31 30.860.02 5.0010.00170.520.2121.340.88 10.0015.00210.670.1514.752.65 15.0020.00110.770.1010.200.06 20.0025.00110.840.077.052.21 25.0030+101.000.1615.802.13 7.95

30. Hypothesis Testing for Simulation 30

31. Hypothesis Testing for Simulation 31  = 5  p-value for 7.83 is larger than 0.05  CANNOT REJECT  CONCLUSION?

32. SIMULATION vs. STATISTICS  Statistics  Sample is fixed and given  Conclusion is unknown  Significance is powerful  Simulation  Sample is arbitrarily large  Conclusion is known  We need another thought about what is meaningful Hypothesis Testing for Simulation 32

33. SAMPLE SIZE EFFECT Hypothesis Testing for Simulation 33  = 100  = 10

34. HOW LARGE IS A DIFFERENCE BEFORE IT IS MEANINGFUL? Hypothesis Testing for Simulation 34

35. Hypothesis Testing for Simulation 35 SUMMARY  You probably knew the mechanics of HT  You might have a new perspective