1 Experimental Statistics - week 5 Chapter 9: Multiple Comparisons Chapter 15: Randomized Complete Block Design (15.3)

2 PC SAS on Campus Library BIC Student Center SAS Learning Edition $125

3 1-Factor ANOVA Model y ij =  i  ij mean for i th treatment unexplained part

4 1-Factor ANOVA Model y ij =  i  ij or observed data mean for i th treatment unexplained part

5 1-Factor ANOVA Model y ij =  i  ij y ij =  i  ij or observed data mean for i th treatment unexplained part

6 1-Factor ANOVA Model y ij =  i  ij y ij =  i  ij or observed data

7 1-Factor ANOVA Model y ij =  i  ij y ij =  i  ij or mean for i th treatment

8 1-Factor ANOVA Model y ij =  i  ij y ij =  i  ij or unexplained part

9 were rewritten as:

10 In words: TSS (total SS) = total sample variability among y ij values SSB (SS “between”) = variability explained by differences in group means SSW (SS “within”) = unexplained variability (within groups)

11 Analysis of Variance Table Note: unequal sample sizes allowed

12 CAR DATA Example For this analysis, 5 gasoline types (A - E) were to be tested. Twenty cars were selected for testing and were assigned randomly to the groups (i.e. the gasoline types). Thus, in the analysis, each gasoline type was tested on 4 cars. A performance-based octane reading was obtained for each car, and the question is whether the gasolines differ with respect to this octane reading. A B C D E

13 Problem 1. Descriptive Statistics for CAR Data The MEANS Procedure Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ

14 Problem 3. Descriptive Statistics by Gasoline gas=A The MEANS Procedure Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=B Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=C Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=D Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=E The MEANS Procedure Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ

15 Gasoline Example - Completely Randomized Design -- All 5 Gasolines The GLM Procedure Dependent Variable: octane Sum of Source DF Squares Mean Square F Value Pr > F Model Error Corrected Total R-Square Coeff Var Root MSE octane Mean Source DF Type I SS Mean Square F Value Pr > F gas

16 Problem 6. 1-factor ANOVA for first 3 GAS Types The GLM Procedure Dependent Variable: octane Sum of Source DF Squares Mean Square F Value Pr > F Model Error Corrected Total R-Square Coeff Var Root MSE octane Mean Source DF Type I SS Mean Square F Value Pr > F gas

17 Problem 3. Descriptive Statistics by Gasoline gas=A The MEANS Procedure Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=B Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=C Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=D Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=E The MEANS Procedure Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ

18

19 Question 1: Which gasolines are different? Question 2: Why didn’t we just do t-tests to compare all combinations of gasolines? i.e. compare A vs B A vs C... D vs E

20 Simulation: i.e. using computer to generate data under certain known conditions and observing the outcomes

21 Setting : Normal population with:  and  Simulation Experiment: Generate 2 samples of size n = 10 from this population and run t-test to compare sample means. Question: What do we expect to happen? i.e test:

t-test procedure:  Reject H 0 if | t | > Simulation Results: t =.235 so we do not reject H

Now - suppose we obtain 10 samples and test Simulation results: Note: Comparing means 4 vs 5 we get t = 2.33 What does this mean?

24 Suppose we run all possible t-tests at significance level   to compare 10 sample means of size n = 10 from this population - it can be shown that there is a 63% chance that at least one pair of means will be declared significantly different from each other F-test in ANOVA controls overall significance level.

25 Probability of finding at least 2 of k means significantly different using multiple t-=tests at the  level when all means are actually equal. k Prob

Protected LSD: Preceded by an F-test for overall significance. Unprotected: Not preceded by an F-test (like individual t-tests). Only use the LSD if F is significant. Fisher’s Least Significant Difference (LSD) X

27 Gasoline Example - Completely Randomized Design -- All 5 Gasolines The GLM Procedure Dependent Variable: octane Sum of Source DF Squares Mean Square F Value Pr > F Model Error Corrected Total R-Square Coeff Var Root MSE octane Mean Source DF Type I SS Mean Square F Value Pr > F gas

28 Problem 3. Descriptive Statistics by Gasoline gas=A The MEANS Procedure Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=B Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=C Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=D Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ gas=E The MEANS Procedure Analysis Variable : octane Mean Std Dev Minimum Maximum ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ

29

30

31 PROC GLM; (or ANOVA) CLASS gas; MODEL octane=gas; TITLE 'Gasoline Example - Completely Randomized Design'; MEANS gas/lsd; RUN;

32 Gasoline Example - Completely Randomized Design The GLM Procedure t Tests (LSD) for octane NOTE: This test controls the Type I comparisonwise error rate, not the experimentwise error rate. Alpha 0.05 Error Degrees of Freedom 15 Error Mean Square Critical Value of t Least Significant Difference Means with the same letter are not significantly different. t Grouping Mean N gas A E B D B C B C C B C B B C C A

33 Gasoline Example - Completely Randomized Design The GLM Procedure t Tests (LSD) for octane NOTE: This test controls the Type I comparisonwise error rate, not the experimentwise error rate. Alpha 0.05 Error Degrees of Freedom 15 Error Mean Square Critical Value of t Least Significant Difference Means with the same letter are not significantly different. t Grouping Mean N gas A E B D B C B C C B C B B C C A

Bonferroni Multiple Comparisons (BSD) Number of Pairwise Comparisons

35

36

37 PROC GLM; (or ANOVA) CLASS gas; MODEL octane=gas; TITLE 'Gasoline Example - Completely Randomized Design'; MEANS gas/bon; RUN;

38 Gasoline Example - Completely Randomized Design The GLM Procedure Bonferroni (Dunn) t Tests for octane NOTE: This test controls the Type I experimentwise error rate, but it generally has a higher Type II error rate than REGWQ. Alpha 0.05 Error Degrees of Freedom 15 Error Mean Square Critical Value of t Minimum Significant Difference Means with the same letter are not significantly different. Bon Grouping Mean N gas A E A B A D B B C B B B B B A

39 Gasoline Example - Completely Randomized Design The GLM Procedure Bonferroni (Dunn) t Tests for octane NOTE: This test controls the Type I experimentwise error rate, but it generally has a higher Type II error rate than REGWQ. Alpha 0.05 Error Degrees of Freedom 15 Error Mean Square Critical Value of t Minimum Significant Difference Means with the same letter are not significantly different. Bon Grouping Mean N gas A E A B A D B B C B B B B B A

40 Extracted from From Ex. 8.2, page Methods for Reducing Hostility 12 students displaying similar hostility were randomly assigned to 3 treatment methods. Scores (HLT) at end of study recorded. Method Method Method Test:

41 ANOVA Table Output - hostility data - calculations done in class Source SS df MS F p-value Between <.001 samples Within samples Totals

42

43 Extracted from From Ex. 8.2, page Methods for Reducing Hostility 12 students displaying similar hostility were randomly assigned to 3 treatment methods. Scores (HLT) at end of study recorded. Method Method Method Test:

44

45 Hostility Data - Completely Randomized Design The GLM Procedure t Tests (LSD) for score NOTE: This test controls the Type I comparisonwise error rate, not the experimentwise error rate. Alpha 0.05 Error Degrees of Freedom 9 Error Mean Square Critical Value of t Least Significant Difference Means with the same letter are not significantly different. t Grouping Mean N method A M1 B M2 B B M3

46 Hostility Data - Completely Randomized Design The GLM Procedure Bonferroni (Dunn) t Tests for score NOTE: This test controls the Type I experimentwise error rate, but it generally has a higher Type II error rate than REGWQ. Alpha 0.05 Error Degrees of Freedom 9 Error Mean Square Critical Value of t Minimum Significant Difference Means with the same letter are not significantly different. Bon Grouping Mean N method A M1 B M2 B B M3

47 Begin Thursday, February 10 Lecture

48 Some Multiple Comparison Techniques in SAS FISHER’S LSD (LSD) BONFERRONI (BON) STUDENT-NEWMAN-KEULS (SNK) DUNCAN DUNNETT RYAN-EINOT-GABRIEL-WELCH (REGWQ) SCHEFFE TUKEY

Balloon Data Col observation number Col. 3 - color (1=pink, 2=yellow, 3=orange, 4=blue) Col inflation time in seconds

50 ANOVA --- Balloon Data General Linear Models Procedure Dependent Variable: TIME Sum of Mean Source DF Squares Square F Value Pr > F Model Error Corrected Total R-Square C.V. Root MSE TIME Mean Mean Source DF Type I SS Square F Value Pr > F Color

51 Experimental Design: Concepts and Terminology Designed Experiment - an investigation in which a specified framework is used to compare groups or treatments Factors - up to this point we’ve only looked at experiments with a single factor - any feature of the experiment that can be varied from trial to trial

52 Experimental Units - subjects, material, etc. to which treatment factors are randomly assigned - there is inherent variability among these units irrespective of the treatment imposed Replication - we usually assign each treatment to several experimental units - these are called replicates - conditions constructed from the factors (levels of the factor considered, etc.) Treatments

53 Examples: Car Data Hostility Data Balloon Data treatments experimental units replicates

54 Scatterplot Using GPLOT

55 Plot of time*id. Legend: A = 1 obs, B = 2 obs, etc. time ‚ 30 ˆ ‚ ‚ A 28 ˆ ‚ 26 ˆ ‚ A ‚ ‚ A ‚ A A 24 ˆ A ‚ ‚ A ‚ A A A 22 ˆ ‚ ‚ A A A 20 ˆ A A ‚ A ‚ A A ‚ A 18 ˆ A ‚ A 16 ˆ A A A A ‚ 14 ˆ A Šƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒƒƒƒƒƒƒƒˆƒƒ id Scatterplot Using PLOT

56 RECALL: 1-Factor ANOVA Model - random errors follow a Normal ( N) distribution, are independently distributed ( ID ), and have zero mean and constant variance -- i.e. variability does not change from group to group

57 Model Assumptions: Checking Validity of Assumptions 1. F-test similar to 2-sample case - Hartley’s test (p.366 text) - not recommended 2. Graphical - side-by-side box plots - equal variances - normality Equal Variances

58 Graphical Assessment of Equal Variance Assumption

59 y ij =  i  ij Assessing Normality of Errors  ij = y ij  (  i ) so  ij is estimated by = y ij  i

60 proc glm; class color; model time=color; title 'ANOVA --- Balloon Data'; output out=new r=resid; means color/lsd; run; proc univariate normal plot; var resid; title 'Normal Probability Plot for Residuals - Balloon Data'; run;

61 Normal Probability Plot *+ | * *+++ | *+++ | +*+ | *** | **** 0.5+ ***+ | ++** | ++*** | ***** | +*+ | *+*+* *

62 Homework Problem using Balloon Data: - Run ANOVA using SAS -- Do not use the 4-step procedure. Instead, describe your findings based on the P-value. - Run multiple comparisons (both Fisher’s LSD and Bonferroni) -- by hand -- using SAS for Balloon Data - Give graphical assessment of the normality and equal variance assumptions and discuss your results

63 Model for Gasoline Data y ij =  i  ij y ij =  i  ij or unexplained part mean for i th gasoline observed octane -- car-to-car differences -- temperature -- etc.

64 Gasoline Data Question: What if car differences are obscuring gasoline differences? Similar to diet t-test example: Recall: person-to-person differences obscured effect of diet

65 Possible Alternative Design: Test all 5 gasolines on the same car - in essence we test the gasoline effect directly and remove effect of car-to-car variation Question: How would you randomize an experiment with 4 cars?

66 Blocking an Experiment - dividing the observations into groups (called blocks) where the observations in each block are collected under relatively similar conditions - comparisons can may times be made more precisely this way

67 Terminology is based on Agricultural Experiments Consider the problem of testing fertilizers on a crop - t fertilizers - n observations on each

68 Completely Randomized Design A A B B C C B A C C B A A B C t = 3 fertilizers n = 5 replications Randomly selected 15 plots

69 Randomized Complete Block Strategy B | A | C A | C | B C | A | B A | B | C C | B | A t = 3 fertilizers - randomly select 5 “blocks” - randomly assign the 3 treatments to each block Note: The 3 “plots” within each block are similar - similar soil type, sun, water, etc

70 Randomized Complete Block Design Randomly assign each treatment once to every block Car Example Car 1: randomly assign each gas to this car Car 2:.... etc. Agricultural Example Randomly assign each fertilizer to one of the 3 plots within each block

71 y ij =  i  j  ij Model For Randomized Complete Block Design effect of i th treatment effect of j th block unexplained error (car)(gasoline)

72