hypothesis testing with special focus on simulation Mike Bailey MSIM 852 Analysis II Hypothesis Testing for Simulation

Hypothesis Testing for Simulation Hypothesis Test answers yes/no question with some statistical certainty H0 = default hypothesis is a statement Ha = alternate hypothesis is the precise opposite Hypothesis Testing for Simulation

Hypothesis Testing for Simulation X = test statistic (RANDOM!) sufficient (uses all avail. data) often Z, T, N are used as notation FX = its probability distribution a = P[reject H0 | H0 true] Hypothesis Testing for Simulation

Hypothesis Testing for Simulation ca = critical region for a a = P[X in ca | H0] a is our (controllable) risk Hypothesis Testing for Simulation

Hypothesis Testing for Simulation TWISTED LOGIC We WANT to reject H0 and conclude Ha, so... We make a very small, so... If we can reject, we have strong evidence that Ha is true This construct often leads to inconclusive results “There is no significant statistical evidence that...” Hypothesis Testing for Simulation

Hypothesis Testing for Simulation IMPORTANT Inability to reject <> H0 true Hypothesis Testing for Simulation

Hypothesis Testing for Simulation POWER OF THE TEST b = P[X not in ca | Ha] 1-b = P[correctly rejecting] Hypothesis Testing for Simulation

Hypothesis Testing for Simulation VENACULAR a is type I error Probability of incorrectly rejecting b is type II error Probability of incorrectly missing the opportunity to reject Hypothesis Testing for Simulation

Hypothesis Testing for Simulation UNOFFICIAL VENACULAR type III error – answered the wrong question type IV error – perfect answer delivered too late Hypothesis Testing for Simulation

Hypothesis Testing for Simulation EXAMPLE! Dial-up ISP has long experience & knows... Hypothesis Testing for Simulation

We get DSL, observe 12 samples Hypothesis Testing for Simulation

Hypothesis Testing for Simulation IS DSL FASTER? H0: mDSL = 50 Ha: mDSL < 50 test with P[type I] = 0.01 Hypothesis Testing for Simulation

Hypothesis Testing for Simulation PROBABILITY THEORY Z ~ tn-1 Must know the probability distribution of the test statistic IOT construct critical region Hypothesis Testing for Simulation

Hypothesis Testing for Simulation 99% of the probability above -2.718 for n = 12, a = 0.01, ca = -2.718 Hypothesis Testing for Simulation

Hypothesis Testing for Simulation our test statistic -2.33 Hypothesis Testing for Simulation

Hypothesis Testing for Simulation -1.796 (0.05) -2.33 (0.021) -2.718 (0.01) 0.021 called the p-value Given H0, we expect to see a test statistic as extreme as Z roughly 2% of the time. Hypothesis Testing for Simulation

Hypothesis Testing for Simulation GOODNESS-OF-FIT TEST Discrete, categorized data Rolls of dice Miss distances in 5-ft. increments H0 assumes a fully-specified probability model Ha: the glove does not fit! Hypothesis Testing for Simulation

Hypothesis Testing for Simulation TEST STATISTIC “chi-squared distribution with gnu degrees of freedom” Hypothesis Testing for Simulation

Hypothesis Testing for Simulation n = observations - estimated param Did you know... if Zi~N(0, 1), then Z12+ Z22+...+ Zn2 ~ cn2 Hypothesis Testing for Simulation

Hypothesis Testing for Simulation CELLS H0 always results in a set of category cells with expected frequencies EXAMPLE Coin is tossed 100 times H0: Coin Fair Hypothesis Testing for Simulation

CELLS AND EXPECTED FREQUENCIES   EXPECT H 50 T Hypothesis Testing for Simulation

Hypothesis Testing for Simulation EXAMPLE Cannon places rounds around a target H0: miss distance ~ expon(0.1m) Record data in 5m intervals (0-5), (5-10), ...(25+) Hypothesis Testing for Simulation

Hypothesis Testing for Simulation EXPONENTIALS E(X)=1/l Hypothesis Testing for Simulation

Hypothesis Testing for Simulation RESULTS RIGHT OBS 1-exp(-0.1x) PROB EXPECT (OBS-EXPECT)^2   0.00 5.00 30 0.39 39.35 2.22 10.00 17 0.63 0.24 23.87 1.97 15.00 21 0.78 0.14 14.47 2.94 20.00 11 0.86 0.09 8.78 0.56 25.00 0.92 0.05 5.33 6.05 30+ 10 1.00 0.08 8.21 100.00 14.14 Hypothesis Testing for Simulation

Hypothesis Testing for Simulation

Hypothesis Testing for Simulation TEST RESULTS Degrees of Freedom 6 cells 0 parameters estimated n = 6 For the c62 distribution, the p- value for 14.14 is about p=0.025 REJECT at any a > 0.025 Hypothesis Testing for Simulation

Hypothesis Testing for Simulation DIFFERENT H0 H0: the miss distances are exponentially distributed Ha: the exponential shape is incorrect We estimate the parameter, we lose one degree of freedom Hypothesis Testing for Simulation

Hypothesis Testing for Simulation RESULTS 2 LEFT RIGHT OBS 1-exp(-0.0738x) PROB EXPECT (OBS-EXPECT)^2   0.00 5.00 30 0.31 30.86 0.02 10.00 17 0.52 0.21 21.34 0.88 15.00 21 0.67 0.15 14.75 2.65 20.00 11 0.77 0.10 10.20 0.06 25.00 0.84 0.07 7.05 2.21 30+ 10 1.00 0.16 15.80 2.13 7.95 Hypothesis Testing for Simulation

Hypothesis Testing for Simulation

Hypothesis Testing for Simulation p-value for 7.83 is larger than 0.05 CANNOT REJECT CONCLUSION? Hypothesis Testing for Simulation

Hypothesis Testing for Simulation SUMMARY You probably knew the mechanics of HT You might have a new perspective Hypothesis Testing for Simulation