1. hypothesis testing with special focus on simulation
Mike Bailey
MSIM 852
Analysis II
Hypothesis Testing for Simulation

2. Hypothesis Testing for Simulation
Hypothesis Test answers yes/no question with some statistical certainty
H0 = default hypothesis
is a statement
Ha = alternate hypothesis
is the precise opposite
Hypothesis Testing for Simulation

3. Hypothesis Testing for Simulation
X = test statistic (RANDOM!)
sufficient (uses all avail. data)
often Z, T, N are used as notation
FX = its probability distribution
a = P[reject H0 | H0 true]
Hypothesis Testing for Simulation

4. Hypothesis Testing for Simulation
ca = critical region for a
a = P[X in ca | H0]
a is our (controllable) risk
Hypothesis Testing for Simulation

5. Hypothesis Testing for Simulation
TWISTED LOGIC
We WANT to reject H0 and conclude Ha, so...
We make a very small, so...
If we can reject, we have strong evidence that Ha is true
This construct often leads to inconclusive results
“There is no significant statistical evidence that...”
Hypothesis Testing for Simulation

6. Hypothesis Testing for Simulation
IMPORTANT
Inability to reject <> H0 true
Hypothesis Testing for Simulation

7. Hypothesis Testing for Simulation
POWER OF THE TEST
b = P[X not in ca | Ha]
1-b = P[correctly rejecting]
Hypothesis Testing for Simulation

8. Hypothesis Testing for Simulation
VENACULAR
a is type I error
Probability of incorrectly rejecting
b is type II error
Probability of incorrectly missing the opportunity to reject
Hypothesis Testing for Simulation

9. Hypothesis Testing for Simulation
UNOFFICIAL VENACULAR
type III error – answered the wrong question
type IV error – perfect answer delivered too late
Hypothesis Testing for Simulation

10. Hypothesis Testing for Simulation
EXAMPLE!
Dial-up ISP has long experience & knows...
Hypothesis Testing for Simulation

11. We get DSL, observe 12 samples
Hypothesis Testing for Simulation

12. Hypothesis Testing for Simulation
IS DSL FASTER?
H0: mDSL = 50
Ha: mDSL < 50
test with P[type I] = 0.01
Hypothesis Testing for Simulation

13. Hypothesis Testing for Simulation
PROBABILITY THEORY
Z ~ tn-1
Must know the probability distribution of the test statistic IOT construct critical region
Hypothesis Testing for Simulation

14. Hypothesis Testing for Simulation
99% of the probability
above
for n = 12, a = 0.01, ca =
Hypothesis Testing for Simulation

15. Hypothesis Testing for Simulation
our test statistic
-2.33
Hypothesis Testing for Simulation

16. Hypothesis Testing for Simulation
-1.796
(0.05)
-2.33
(0.021)
-2.718
(0.01)
0.021 called the p-value
Given H0, we expect to see a test statistic as extreme as Z roughly 2% of the time.
Hypothesis Testing for Simulation

17. Hypothesis Testing for Simulation
GOODNESS-OF-FIT TEST
Discrete, categorized data
Rolls of dice
Miss distances in 5-ft. increments
H0 assumes a fully-specified probability model
Ha: the glove does not fit!
Hypothesis Testing for Simulation

18. Hypothesis Testing for Simulation
TEST STATISTIC
“chi-squared distribution with gnu degrees of freedom”
Hypothesis Testing for Simulation

19. Hypothesis Testing for Simulation
n = observations - estimated param
Did you know... if Zi~N(0, 1), then
Z12+ Z Zn2 ~ cn2
Hypothesis Testing for Simulation

20. Hypothesis Testing for Simulation
CELLS
H0 always results in a set of category cells with expected frequencies
EXAMPLE
Coin is tossed 100 times
H0: Coin Fair
Hypothesis Testing for Simulation

21. CELLS AND EXPECTED FREQUENCIES
EXPECT H 50 T
Hypothesis Testing for Simulation

22. Hypothesis Testing for Simulation
EXAMPLE
Cannon places rounds around a target
H0: miss distance ~ expon(0.1m)
Record data in 5m intervals
(0-5), (5-10), ...(25+)
Hypothesis Testing for Simulation

23. Hypothesis Testing for Simulation
EXPONENTIALS
E(X)=1/l
Hypothesis Testing for Simulation

24. Hypothesis Testing for Simulation
RESULTS
RIGHT
OBS
1-exp(-0.1x)
PROB
EXPECT
(OBS-EXPECT)^2
0.00
5.00 30
0.39
39.35
2.22
10.00 17
0.63
0.24
23.87
1.97
15.00 21
0.78
0.14
14.47
2.94
20.00 11
0.86
0.09
8.78
0.56
25.00
0.92
0.05
5.33
6.05
30+ 10
1.00
0.08
8.21
100.00
14.14
Hypothesis Testing for Simulation

25. Hypothesis Testing for Simulation

26. Hypothesis Testing for Simulation
TEST RESULTS
Degrees of Freedom
6 cells
0 parameters estimated
n = 6
For the c62 distribution, the p- value for is about p=0.025
REJECT at any a > 0.025
Hypothesis Testing for Simulation

27. Hypothesis Testing for Simulation
DIFFERENT H0
H0: the miss distances are exponentially distributed
Ha: the exponential shape is incorrect
We estimate the parameter, we lose one degree of freedom
Hypothesis Testing for Simulation

28. Hypothesis Testing for Simulation
RESULTS 2
LEFT
RIGHT
OBS
1-exp( x)
PROB
EXPECT
(OBS-EXPECT)^2
0.00
5.00 30
0.31
30.86
0.02
10.00 17
0.52
0.21
21.34
0.88
15.00 21
0.67
0.15
14.75
2.65
20.00 11
0.77
0.10
10.20
0.06
25.00
0.84
0.07
7.05
2.21
30+ 10
1.00
0.16
15.80
2.13
7.95
Hypothesis Testing for Simulation

29. Hypothesis Testing for Simulation

30. Hypothesis Testing for Simulation
p-value for 7.83 is larger than 0.05
CANNOT REJECT
CONCLUSION?
Hypothesis Testing for Simulation

31. Hypothesis Testing for Simulation
SUMMARY
You probably knew the mechanics of HT
You might have a new perspective
Hypothesis Testing for Simulation