1. Statistical Comparison of Two or More Systems The most relevant of all the Basic Theory Lectures. No Holidays.

2. THE MISSION Your analysis task involves manipulating conditions of the system of interest from a prescribed set of options. Design of Experiments: Determine if the different options are really different. Is the best one really statistically better? Ranking and Selection: What’s the probability that the best sample indicates the best system setting?

3. VOCABULARY Factor An element of the system that will be manipulated Setting or Level A value that a Factor may assume

4. EXAMPLE : Simulation model of Football (EA Sports) Factors Quarterback Running Back Strong Safety Settings or Levels for Quarterback Dante’ Bret Johnny U.

5. TYPES OF DESIGNS One Factor, Two Settings Paired samples Behrens-Fischer Question: Which is Best? More than one Factor Factorial Designs Partially Exhaustive Designs Question: Are the settings significant difference- makers?

6. PAIRED SAMPLES Example: Quarterback Controversy! Simulate St. Louis Rams vs. Tampa Bay Bucs, recording the Quarterback Rating Level 1: Curt Warner Level 2: Mark Bulger Run the simulation 28 times for each player, resulting in data set W1, W2,..., W28 B1, B2,..., B28 Is E[B] > E[W]?

7. BRUTE FORCE Confidence interval on the quantity E[W]-E[B] If it doesn’t include 0.0, we have conclusive evidence that there is a difference Equivalent to the Hypothesis Test H0: E[B]=E[W]

8. CALCULATIONS ON VARIANCES: SOME BASICS Let X and Y be random variables

9. CALCULATIONS ON VARIANCES: SOME BASICS Let X and Y be random variables COV=0 if X and Y are independent.

10. SAMPLE MEAN

11. CONFIDENCE INTERVAL  /2 probability of Type I error on each end of the confidence interval basic interval for X-bar is

12. BASIC CONFIDENCE INTERVAL

13. SPREADSHEET HIGHLIGHTS 1 (U-0.5)*SQRT(12) zero mean unit stddev  + (U-0.5)*SQRT(12)*  mean  stddev  uniform over an interval centered at  and  *SQRT(12)/2 wide

14. COMMON RANDOM NUMBERS Correlation is not always BAD! Suppose we could INDUCE CORRELATION between the W’s and the B’s without adding any bias? Reduces the theoretical variance of W-bar – B-bar FREE POWER (the probability of correctly rejecting H0: equal means)

15. STREAMING Segregate the random number generation task into streams connected to phenomena seed1seed2 Inter-arrival times Service times Z i =aZ i-1 mod m 1. Change features of the service. 2. Use exact same arrival stream for comparing each service setting.

16. SPREADSHEET HIGHLIGHTS 2 Use same results of RAND() for building Bulger samples Warner samples Note CI shrinkage Try with identical sigma Discuss “Estimation”

17. Behrens-Fischer Problem Comparison of Means No pairs, equal sample sizes, or equal variances Remember that we are after the variance of W- bar – B-bar Common use: New samples vs. History

18. SPREADSHEET HIGHLIGHTS

19. MULTI-SETTING CASE Can involve many Factors or just one Treatment i has mean  i Analysis of Variance (ANOVA) Data from treatment 1, 2,..., n H0:  1 =...  n-1 =  n Are the treatments distinguishable?

20. DESIGN OF EXPERIMENT Determine Factors and Settings Collect Data According to Design Design = Which Factors, Which Settings for each Treatment Perform ANOVA State Conclusion

21. FULL FACTORIAL Build sample of All Combinations Factors Quarterback (2) Running Back (3) Strong Safety (3) 2x3x3=18 Treatments

22. HOW ANOVA WORKS Xi,j is ith sample from jth treatment point Assumed iid Normal (never!) Decomposition of variability Observation (Obs) Treatment vs. Grand Mean (Tr) Within Treatment (Res)

23. HYPOTHESIS H0 The treatment variability is random variability The size of the treatment variability is in-scale with the residual variability ANOVA uses sums of squares g treatments n t samples from treatment t

24. ANOVA TABLE degrees freedom

25. REMEMBER chi-SQUARED? From our Goodness-of-Fit Test X~N(0,1) for n independent X’s sum of n X 2 is chi-SQUARED with n degrees of freedom if estimates (X-bar, sigma) were used to make X’s N(0,1), lose one d.f. per estimate

26. F-distribution X is chi-sq with n d.f. Y is chi-sq with m d.f. (X/n)/(Y/m) has F distribution

27. ANOVA HYPOTHESIS TEST The normalizing  cancels!

28. ANOVA HYPOTHESIS TEST Compare the test statistic to a table Reject if its big and conclude that...  the Treatments are Different!

29. SPREADSHEET HIGHLIGHTS