1. The Multiple Comparisons Problem in IES Impact Evaluations: Guidelines and Applications Peter Z. Schochet and John Deke June 2009, IES Research Conference

2. What Is the Problem? Multiple hypothesis tests are often conducted in impact studies –Outcomes –Subgroups –Treatment groups Standard testing methods could yield: – Spurious significant impacts – Incorrect policy conclusions Multiple hypothesis tests are often conducted in impact studies –Outcomes –Subgroups –Treatment groups Standard testing methods could yield: – Spurious significant impacts – Incorrect policy conclusions 2

3. Overview of Presentation Background Testing guidelines adopted by IES Examples of their use by the RELs New guidance on statistical methods for “between-domain” analyses Background Testing guidelines adopted by IES Examples of their use by the RELs New guidance on statistical methods for “between-domain” analyses 3

4. Background

5. Assume a Classical Hypothesis Testing Framework Test H 0j : Impact j = 0 Reject H 0j if p-value of t-test <  =.05 Chance of finding a spurious impact is 5 percent for each test alone Test H 0j : Impact j = 0 Reject H 0j if p-value of t-test <  =.05 Chance of finding a spurious impact is 5 percent for each test alone 5

6. But If Tests Are Considered Together and No True Impacts… Probability  1 t-test Number of Tests a Is Statistically Significant 1.05 5.23 10.40 20.64 50.92 a Assumes independent tests 6

7. Impact Findings Can Be Misrepresented Publishing bias A focus on “stars” Publishing bias A focus on “stars” 7

8. Adjustment Procedures Lower  Levels for Individual Tests Methods control the “combined” error rate Many available methods: –Bonferroni: Compare p-values to (.05 / # of tests) –Fisher’s LSD, Holm (1979), Sidak (1967), Scheffe (1959), Hochberg (1988), Rom (1990), Tukey (1953) –Resampling methods (Westfall and Young 1993) –Benjamini-Hochberg (1995) Methods control the “combined” error rate Many available methods: –Bonferroni: Compare p-values to (.05 / # of tests) –Fisher’s LSD, Holm (1979), Sidak (1967), Scheffe (1959), Hochberg (1988), Rom (1990), Tukey (1953) –Resampling methods (Westfall and Young 1993) –Benjamini-Hochberg (1995) 8

9. These Methods Reduce Statistical Power: The Chances of Finding Real Effects Simulated Statistical Power a Number of Tests Unadjusted Bonferroni 5.80.59 10.80.50 20.80.41 50.80.31 a Assumes 1,000 treatments and 1,000 controls, 20 percent of all null hypotheses are true, and independent tests 9

10. Basic Testing Guidelines Balance Type I and II Errors

11. Problem Should Be Addressed by First Structuring the Data Structure will depend on the research questions, previous evidence, and theory Adjustments should not be conducted blindly across all contrasts Structure will depend on the research questions, previous evidence, and theory Adjustments should not be conducted blindly across all contrasts 11

12. The Plan Must Be Specified Up Front To avoid “fishing” for findings Study protocols should specify: –Data structure –Confirmatory analyses –Exploratory analyses –Testing strategy To avoid “fishing” for findings Study protocols should specify: –Data structure –Confirmatory analyses –Exploratory analyses –Testing strategy 12

13. Delineate Separate Outcome Domains Based on a conceptual framework Represent key clusters of constructs Domain “items” are likely to measure the same underlying trait (have high correlations) –Test scores –Teacher practices –Student behavior Based on a conceptual framework Represent key clusters of constructs Domain “items” are likely to measure the same underlying trait (have high correlations) –Test scores –Teacher practices –Student behavior 13

14. Testing Strategy: Both Confirmatory and Exploratory Components Confirmatory component –Addresses central study hypotheses –Used to make overall decisions about program –Must adjust for multiple comparisons Exploratory component –Identify impacts or relationships for future study –Findings should be regarded as preliminary Confirmatory component –Addresses central study hypotheses –Used to make overall decisions about program –Must adjust for multiple comparisons Exploratory component –Identify impacts or relationships for future study –Findings should be regarded as preliminary 14

15. Focus of Confirmatory Analysis Is on Experimental Impacts Focus is on key child outcomes, such as test scores Targeted subgroups: eg. ELL students Some experimental impacts could be exploratory –Subgroups –Secondary child and teacher outcomes Focus is on key child outcomes, such as test scores Targeted subgroups: eg. ELL students Some experimental impacts could be exploratory –Subgroups –Secondary child and teacher outcomes 15

16. Confirmatory Analysis Has Two Potential Parts 1. Domain-specific analysis 2. Between-domain analysis 1. Domain-specific analysis 2. Between-domain analysis 16

17. Domain-Specific Analysis: Test Impacts for Outcomes as a Group Create a composite domain outcome –Weighted average of standardized outcomes  Equal weights  Expert judgment  Predictive validity weights  Factor analysis weights  MANOVA not recommended Conduct a t-test on the composite Create a composite domain outcome –Weighted average of standardized outcomes  Equal weights  Expert judgment  Predictive validity weights  Factor analysis weights  MANOVA not recommended Conduct a t-test on the composite 17

18. Between-Domain Analysis: Test Impacts for Composites Across Domains Are impacts significant in all domains? –No adjustments are needed Are impacts significant in any domain? –Adjustments are needed –Discussed later Are impacts significant in all domains? –No adjustments are needed Are impacts significant in any domain? –Adjustments are needed –Discussed later 18

19. Application of Guidelines by the Regional Educational Labs

20. Basic Features of the REL Studies 25 Randomized Control Trials –Single treatment and control groups –Testing diverse interventions –Typically grades K-8 –Fall-spring data collection, some longer –Collecting data on teachers and students 25 Randomized Control Trials –Single treatment and control groups –Testing diverse interventions –Typically grades K-8 –Fall-spring data collection, some longer –Collecting data on teachers and students 20

21. Each RCT Provided a Detailed Analysis Plan to IES Confirmatory research questions Confirmatory domains and outcomes Within- and between-domain testing strategy Study samples Statistical power levels Confirmatory research questions Confirmatory domains and outcomes Within- and between-domain testing strategy Study samples Statistical power levels 21 Each Plan Included Information on:

22. Key Features of Confirmatory Domains Student academic achievement domains are specified in all RCTs Some domains pertain to: –Behavioral outcomes –A specific time period for longitudinal studies –Subgroups: ELL students Student academic achievement domains are specified in all RCTs Some domains pertain to: –Behavioral outcomes –A specific time period for longitudinal studies –Subgroups: ELL students 22

23. Most RCTs Have Specified Structured Research Questions Most have fewer than 3 domains –Some have only 1 –Most domains have a small number of outcomes Main between-domain question: “ Are there positive impacts in any domain?” Most have fewer than 3 domains –Some have only 1 –Most domains have a small number of outcomes Main between-domain question: “ Are there positive impacts in any domain?” 23

24. Adjustment Methods for Between-Domain Confirmatory Analyses

25. Focus on Methods to Control the Familywise Error Rate (FWER) FWER = Prob (find ≥1 significant impact given that no impacts truly exist) Preferred over the false discovery rate developed by Benjamini-Hochberg (BH) –BH is a preponderance-of-evidence method –BH does not control the FDR for all forms of dependencies across test statistics FWER = Prob (find ≥1 significant impact given that no impacts truly exist) Preferred over the false discovery rate developed by Benjamini-Hochberg (BH) –BH is a preponderance-of-evidence method –BH does not control the FDR for all forms of dependencies across test statistics 25

26. Consider Four FWER Adjustment Methods Sidak: Exact adjustment when tests are independent Bonferroni: Approximate adjustment when tests are independent Generalized Tukey: Adjusts for correlated tests that follow a multivariate t-distribution Resampling: Robust adjustment for correlated tests for general distributions Sidak: Exact adjustment when tests are independent Bonferroni: Approximate adjustment when tests are independent Generalized Tukey: Adjusts for correlated tests that follow a multivariate t-distribution Resampling: Robust adjustment for correlated tests for general distributions 26

27. Main Research Questions How do these four methods work? Are the more complex methods likely to provide more powerful tests for between- domain analyses? –There are no single-routine statistical packages for the complex methods under clustered designs How do these four methods work? Are the more complex methods likely to provide more powerful tests for between- domain analyses? –There are no single-routine statistical packages for the complex methods under clustered designs 27

28. Basic Setup for the Between- Domain Analysis Assume N domain composites Test whether any domain composite is statistically significant Aim to control the FWER at  =.05 All methods reduce the  level for individual tests:  * =.05/fact Assume N domain composites Test whether any domain composite is statistically significant Aim to control the FWER at  =.05 All methods reduce the  level for individual tests:  * =.05/fact 28

29. Sidak Uses the relation that the FWER = [1 – Pr(correctly rejecting all N null hypotheses)] For independent tests, FWER = 1 – (1-  * ) N Sidak picks  * so that FWER = 0.05 For example, if N = 3: –  * = 0.017 –fact = 0.05/ 0.017 = 2.949 Uses the relation that the FWER = [1 – Pr(correctly rejecting all N null hypotheses)] For independent tests, FWER = 1 – (1-  * ) N Sidak picks  * so that FWER = 0.05 For example, if N = 3: –  * = 0.017 –fact = 0.05/ 0.017 = 2.949 29

30. The Bonferroni Method Tends to Be More Conservative  * = (.05 / N); fact = N 30 NSidakBonferroni 111 21.9752 32.9493 43.9244 54.8995 The Value of fact for the Sidak and Bonferroni

31. Sidak and Bonferroni Are Likely To Be Conservative with Correlated Tests Correlated tests can occur if: –Domain composites are correlated –Treatment effects are heterogeneous Yields tests with lower power Correlated tests can occur if: –Domain composites are correlated –Treatment effects are heterogeneous Yields tests with lower power 31

32. Generalized Tukey and Resampling Methods Adjust for Correlated Tests Let p i be the p-value from test i Both methods use the relation: FWER = Pr(min(p 1, p 2, p 3,…, p N )≤.05 | H 0 is true) Both methods calculate FWER using the distribution of min(p 1, p 2, p 3,…, p N ) or max(t 1, t 2, t 3,…, t N ) Let p i be the p-value from test i Both methods use the relation: FWER = Pr(min(p 1, p 2, p 3,…, p N )≤.05 | H 0 is true) Both methods calculate FWER using the distribution of min(p 1, p 2, p 3,…, p N ) or max(t 1, t 2, t 3,…, t N ) 32

33. Generalized Tukey Assumes test statistics have multivariate t distributions with known correlations The MULTCOMP package in R can implement this adjustment (Hothorn, Bretz, Westfall 2008) –Multi-stage procedure that requires user inputs Assumes test statistics have multivariate t distributions with known correlations The MULTCOMP package in R can implement this adjustment (Hothorn, Bretz, Westfall 2008) –Multi-stage procedure that requires user inputs 33

34. Using the MULTCOMP Package Inputs are a vector of impact estimates and the corresponding variance-covariance matrix Challenge is to get cross-equation covariances of the impact estimates One option: use the suest command in STATA, then copy resulting covariance matrix to R –Uses GEE rather than HLM to adjust for clustering Inputs are a vector of impact estimates and the corresponding variance-covariance matrix Challenge is to get cross-equation covariances of the impact estimates One option: use the suest command in STATA, then copy resulting covariance matrix to R –Uses GEE rather than HLM to adjust for clustering 34

35. Resampling/Bootstrapping The distribution of the maximum t-statistic can be estimated through resampling (Westfall and Young 1993) –Allows for general forms of correlations and outcome distributions Resampling must be performed “under the null hypothesis” The distribution of the maximum t-statistic can be estimated through resampling (Westfall and Young 1993) –Allows for general forms of correlations and outcome distributions Resampling must be performed “under the null hypothesis” 35

36. Homoskedastic Bootstrap Algorithm 1. Calculate impacts and tstats using the original data 2. Define Y* as the residuals from these regressions 3. Repeat the following at least 10,000 times: –Randomly sample schools, with replacement, from Y* –Randomly assign sampled schools to treatment and control groups in the same proportion as in the original data –Calculate impacts and save the maximum absolute tstat 4. Adjusted p-values = proportion of maximum tstats that lie above the absolute value of the original tstats 1. Calculate impacts and tstats using the original data 2. Define Y* as the residuals from these regressions 3. Repeat the following at least 10,000 times: –Randomly sample schools, with replacement, from Y* –Randomly assign sampled schools to treatment and control groups in the same proportion as in the original data –Calculate impacts and save the maximum absolute tstat 4. Adjusted p-values = proportion of maximum tstats that lie above the absolute value of the original tstats 36

37. Example of Resampling Method 37 Original tstats are 0.793 and 3.247; Adjusted p-values are 0.89 and 0.00 tstat 1tstat 2Maximum abs(tstat) a 0.9092.6352.635 1 0.8921.2271.227 1 -2.7681.3422.768 1 0.570-0.2370.570 -0.574-1.4721.472 1 -1.245-0.5451.245 1 0.7980.0830.798 1 -0.1380.0270.138 1 -1.8100.4941.810 1 a 1 = Max tstat > 0.793; 2 = Max tstat > 3.247

38. Implementation of Resampling The MULTTEST procedure in SAS implements resampling, but only for non- clustered data Simple approach: Aggregate data to the school level, and use MULTTEST More complex approach: Write a program to implement the algorithm with clustering The MULTTEST procedure in SAS implements resampling, but only for non- clustered data Simple approach: Aggregate data to the school level, and use MULTTEST More complex approach: Write a program to implement the algorithm with clustering 38

39. Comparing Methods Assume 3 composite domain outcomes with correlations of 0.20, 0.50, and 0.80 Outcomes are normally distributed or heavily skewed normals (focus on skewed) Four types of comparisons: –FWER –Values of fact –Minimum Detectable Effect Size (MDES) –“Goal Line” scenario Assume 3 composite domain outcomes with correlations of 0.20, 0.50, and 0.80 Outcomes are normally distributed or heavily skewed normals (focus on skewed) Four types of comparisons: –FWER –Values of fact –Minimum Detectable Effect Size (MDES) –“Goal Line” scenario 39

40. FWER Values Are Similar by Method Except With Large Correlations 40 FWER Values, by Method and Test Correlations ρ=0.2ρ=0.5ρ=0.8 No Adjustment0.1460.1250.097 Bonferroni0.0480.0450.034 Sidak0.0500.0480.036 Generalized Tukey0.0490.0510.049 Bootstrap0.0540.0520.051

41. Values of fact Are Similar by Method Except With Large Correlations 41 Values of fact, by Method and Test Correlations ρ=0.2ρ=0.5ρ=0.8 Bonferroni3.00 Sidak2.85 Generalized Tukey2.842.582.02 Bootstrap2.832.572.01

42. All Methods Yield Similar MDES 42 MDE Values, by Method and Test Correlations a ρ=0.2ρ=0.5ρ=0.8 No Adjustment0.21 Bonferroni0.25 Sidak0.24 Generalized Tukey0.24 0.23 Bootstrap0.24 0.23 a Assumes 60 schools, 60 students per school, R 2 =0.50, ICC=0.15

43. “Goal Line” Scenario: The Method Could Matter for Marginally Significant Impacts 43 Adjusted p-values, by Method and Test Correlations a a Assumes 60 schools, 60 students per School, R 2 =0.50, ICC=0.15 ρ=0.2ρ=0.5ρ=0.8 No Adjustment0.019 Bonferroni0.057 Sidak0.054 Generalized Tukey0.0540.0490.038 Bootstrap0.0540.0490.038

44. Summary and Conclusions Multiple comparisons guidelines: –Specify confirmatory analyses in study protocols –Delineate outcome domains –Conduct hypothesis tests on domain composites RELs have implemented guidelines Multiple comparisons guidelines: –Specify confirmatory analyses in study protocols –Delineate outcome domains –Conduct hypothesis tests on domain composites RELs have implemented guidelines 44

45. Summary and Conclusions Adjustments are needed for between- domain analyses –For calculating MDEs in the design stage, using the Bonferroni is sufficient –For estimating impacts, the more complex methods may be preferred in “goal-line situations” when test correlations are large Adjustments are needed for between- domain analyses –For calculating MDEs in the design stage, using the Bonferroni is sufficient –For estimating impacts, the more complex methods may be preferred in “goal-line situations” when test correlations are large 45

46. References and Contact Information Guidelines in Multiple Testing in Impact Evaluations (Schochet 2008) –ies.ed.gov/ncee/pubs/20084018.asp Resampling-Based Multiple Testing (Westfall and Young 1993; John Wiley and Sons) pschochet@mathematica-mpr.com jdeke@mathematica-mpr.com Guidelines in Multiple Testing in Impact Evaluations (Schochet 2008) –ies.ed.gov/ncee/pubs/20084018.asp Resampling-Based Multiple Testing (Westfall and Young 1993; John Wiley and Sons) pschochet@mathematica-mpr.com jdeke@mathematica-mpr.com 46