Hypothesis Testing for Simulation 1 hypothesis testing with special focus on simulation

Hypothesis Testing for Simulation 2  Hypothesis Test answers yes/no question with some statistical certainty  H 0 = default hypothesis  is a statement  H a = alternate hypothesis  is the precise opposite

Hypothesis Testing for Simulation 3  X = test statistic (RANDOM!)  sufficient (uses all avail. data)  often Z, T, N are used as notation  F X = its probability distribution   = P[reject H 0 | H 0 true]

Hypothesis Testing for Simulation 4  c  = critical region for    = P[X in c  | H 0 ]   is our (controllable) risk

Hypothesis Testing for Simulation 5 TWISTED LOGIC  We WANT to reject H 0 and conclude H a, so...  We make  very small, so...  If we can reject, we have strong evidence that H a is true  This construct often leads to inconclusive results  “There is no significant statistical evidence that...”

Hypothesis Testing for Simulation 6 IMPORTANT  Inability to reject <> H 0 true

Hypothesis Testing for Simulation 7 POWER OF THE TEST   = P[X not in c  | H a ]  1- = P[correctly rejecting]

Hypothesis Testing for Simulation 8 VENACULAR   is type I error  Probability of incorrectly rejecting   is type II error  Probability of incorrectly missing the opportunity to reject

Hypothesis Testing for Simulation 9 UNOFFICIAL VENACULAR  type III error – answered the wrong question  type IV error – perfect answer delivered too late

Hypothesis Testing for Simulation 10 EXAMPLE!  Dial-up ISP has long experience & knows...

Hypothesis Testing for Simulation 11 We get DSL, observe 12 samples

Hypothesis Testing for Simulation 12 IS DSL FASTER?  H 0 :  DSL = 50  H a :  DSL < 50  test with P[type I] = 0.01

Hypothesis Testing for Simulation 13 PROBABILITY THEORY  Z ~ t n-1  Must know the probability distribution of the test statistic IOT construct critical region

Hypothesis Testing for Simulation 14  for n = 12,  = 0.01, c  = % of the probability above

Hypothesis Testing for Simulation 15 our test statistic -2.33

Hypothesis Testing for Simulation 16  called the p-value  Given H 0, we expect to see a test statistic as extreme as Z roughly 2% of the time (0.01) (0.05) (0.021)

CONFIDENCE INTERVALS Hypothesis Testing for Simulation 17 ll uu  For a given   P[l  <=  <= u  ] = 1-  Based on the sample So they are RANDOM!

Hypothesis Testing for Simulation 18 GOODNESS-OF-FIT TEST  Discrete, categorized data  Rolls of dice  Miss distances in 5-ft. increments  H 0 assumes a fully-specified probability model  H a : the glove does not fit!

Hypothesis Testing for Simulation 19 TEST STATISTIC “chi-squared distribution with gnu degrees of freedom”

Hypothesis Testing for Simulation 20  = observations - estimated param  Did you know... if Z i ~N(0, 1), then Z Z Z n 2 ~  n 2

Hypothesis Testing for Simulation 21 CELLS  H 0 always results in a set of category cells with expected frequencies  EXAMPLE  Coin is tossed 100 times  H 0 : Coin Fair

Hypothesis Testing for Simulation 22 CELLS AND EXPECTED FREQUENCIES EXPECT H50 T

Hypothesis Testing for Simulation 23 EXAMPLE  Cannon places rounds around a target  H 0 : miss distance ~ expon(0.1m)  Record data in 5m intervals  (0-5), (5-10),...(25+)

Hypothesis Testing for Simulation 24 EXPONENTIALS E(X)=1/

Hypothesis Testing for Simulation 25 RESULTS RIGHTOBS1-exp(-0.1x)PROBEXPECT(OBS-EXPECT)^

Hypothesis Testing for Simulation 26

Hypothesis Testing for Simulation 27 TEST RESULTS  Degrees of Freedom  6 cells  0 parameters estimated  = 6  For the  6 2 distribution, the p- value for is about p=0.025  REJECT at any  > 0.025

Hypothesis Testing for Simulation 28 DIFFERENT H 0  H 0 : the miss distances are exponentially distributed  H a : the exponential shape is incorrect  We estimate the parameter, we lose one degree of freedom

Hypothesis Testing for Simulation 29 RESULTS 2 LEFTRIGHTOBS1-exp( x)PROB EXPE CT(OBS-EXPECT)^

Hypothesis Testing for Simulation 30

Hypothesis Testing for Simulation 31  = 5  p-value for 7.83 is larger than 0.05  CANNOT REJECT  CONCLUSION?

SIMULATION vs. STATISTICS  Statistics  Sample is fixed and given  Conclusion is unknown  Significance is powerful  Simulation  Sample is arbitrarily large  Conclusion is known  We need another thought about what is meaningful Hypothesis Testing for Simulation 32

SAMPLE SIZE EFFECT Hypothesis Testing for Simulation 33  = 100  = 10

HOW LARGE IS A DIFFERENCE BEFORE IT IS MEANINGFUL? Hypothesis Testing for Simulation 34

Hypothesis Testing for Simulation 35 SUMMARY  You probably knew the mechanics of HT  You might have a new perspective