1. Incorporating the sample correlation between two test statistics to adjust the critical points for the control of type-1 error
Dror Rom and Jaclyn McTague
Prosoft Clinical
Joint Statistical meetings, 2019

2. The setting: A test drug is compared to a reference drug on two endpoints. The null hypotheses are about the equality of the mean responses:
𝐻 01 : 𝜇 11 = 𝜇 12
𝐻 02 : 𝜇 21 = 𝜇 22
𝐻 0 = 𝐻 01 ∩ 𝐻 02 is the global null hypothesis
We are looking to prove that the test drug is better than the reference on at least one of the endpoints
The null hypotheses are tested against one-sided alternatives. Overall type-1 error needs to be strongly controlled at

3. 𝑛 1 and 𝑛 2 (𝑁= 𝑛 1 + 𝑛 2 ) patients are randomized to the two treatment groups.
Two variables are measured on each subject. Measurements are normally distributed. There is an unknown correlation 𝜌 between the measurements.
The test statistics for the respective hypotheses are:
𝑇 1 = 𝑋 11 − 𝑋 𝑆 𝑛 𝑛 2 , 𝑇 2 = 𝑋 21 − 𝑋 𝑆 𝑛 𝑛 2 ,
To control type-1 error, we need to find a critical value, 𝐶 𝜌 (or 𝛼 𝜌 such that 𝛼 𝜌 =𝑝𝑟𝑜𝑏( 𝑇 𝑖 > 𝐶 𝜌 ).
𝛼≥ 𝑃𝑟({ 𝑇 1 ≥ 𝐶 𝜌 }∪{ 𝑇 2 ≥ 𝐶 𝜌 })

4. Because 𝜌 is unknown, we typically use non-parametric, or distribution free procedures to calculate the critical values, for example, Bonferroni, or modified Bonferroni (Hochberg and Hommel) step-up procedures.
The modified Bonferroni procedures have been shown to control type-1 error under a variety of distributional settings, including one-sided testing of normal and t- statistics with positive correlations, and with some modifications, also with negative correlations.

5. If the sample sizes are large, 𝑇 1 and 𝑇 2 are approximately normal (0,1) under the respective null hypotheses The joint distribution of the test statistics is approximately a standard bivariate normal with correlation 𝜌. For large sample sizes, the sample correlation, r, can be used to get critical values from the standard bivariate normal with correlation r. Replace 𝐶 𝜌 with 𝐶 𝑟 and 𝛼 𝜌 with 𝛼 𝑟
We can calculate the critical values by integrating a bivariate normal with correlation r

6. Critical 𝛼 𝑟 as a function of the sample correlation r
We then get a set of alpha-critical values as a function of the sample correlation. Notice that when r is close to -1, the critical value approaches alpha/2 which is the Bonferroni critical value. When r approaches 1, the critical value approaches 1 since the two endpoints are essentially the same.
When the sample size are not large, the use of this set of critical values will cause type-1 error inflation. So we need to adjust this set based on the sample size.

7. To control type 1 error exactly, we want to calculate a critical point, 𝐶 𝑟 , such that: 𝛼 ≥ 𝑃𝑟({ 𝑇 1 ≥ 𝐶 𝑟 }∪{ 𝑇 2 ≥ 𝐶 𝑟 }) =1−𝑃𝑟({ 𝑇 1 < 𝐶 𝑟 }∩{ 𝑇 2 < 𝐶 𝑟 }) =1−𝑃𝑟 𝑋 11 − 𝑋 12 𝑆 1 1 𝑛 𝑛 2 < 𝐶 𝑟 ∩ 𝑋 21 − 𝑋 22 𝑆 2 1 𝑛 𝑛 2 < 𝐶 𝑟 =1−𝑃𝑟 𝑋 11 − 𝑋 12 𝜎 1 1 𝑛 𝑛 2 < 𝑆 1 𝜎 1 𝐶 𝑟 ∩ 𝑋 21 − 𝑋 22 𝜎 2 1 𝑛 𝑛 2 < 𝑆 2 𝜎 2 𝐶 𝑟 =1−𝑃𝑟 𝑍 1 < 𝑆 1 𝜎 1 𝐶 𝑟 ∩ 𝑍 2 < 𝑆 2 𝜎 2 𝐶 𝑟 .
To calculate the type-1 error for a given set of critical values, we rewrite the type 1 error. We multiply both sides by s/sigma and we get the probability in terms of standard normal statistics. This probability is a probability that the z statistics fall under these critical values which are themselves random variables, which are functions of s1, s2 and r. to calculate this probability we do the following

8. 𝜌 =𝑟= 𝑤 12 𝑤 1 𝑤 2 be the sample estimate of the correlation
Let:
𝑤 1 = 𝒋=𝟏 𝟐 𝒊=𝟏 𝒏 𝒋 𝑿 𝟏𝒋𝒊 − 𝑿 𝟏𝒋 𝟐 𝑠 1 = 𝑤 1 𝑁−2
𝑤 2 = 𝒋=𝟐 𝟐 𝒊=𝟏 𝒏 𝒋 𝑿 𝟐𝒋𝒊 − 𝑿 𝟐𝒋 𝟐 𝑠 2 = 𝑤 2 𝑁−2
𝑤 12 = 𝒋=𝟏 𝟐 𝒊=𝟏 𝒏 𝒋 𝑿 𝟏𝒋𝒊 − 𝑿 𝟏𝒋 𝑿 𝟐𝒋𝒊 − 𝑿 𝟐𝒋 𝑠 12 = 𝑤 12 𝑁−2
𝜌 =𝑟= 𝑤 𝑤 1 𝑤 2 be the sample estimate of the correlation
We put s1, s2 and r as functions of the sample sum of squares and sum of products, w1, w2 , w12. The joint distribution of w1, w2, w12 is a Wishart

9. →1−𝑃𝑟 𝑍 1 < 𝑆 1 𝜎 1 𝐶 𝑟 ∩ 𝑍 2 < 𝑆 2 𝜎 2 𝐶 𝑟
=1− 𝑤 1 =0 ∞ 𝑤 2 =0 ∞ 𝑤 12 =−∞ ∞ 𝑊 𝑤 12 , 𝑤 1 , 𝑤 2 𝛷 𝑟 𝑤 1 𝑁−2 𝜎 1 𝐶 𝑟 , 𝑤 2 𝑁−2 𝜎 2 𝐶 𝑟 𝑑 𝑤 12 𝑑 𝑤 1 𝑑 𝑤 2 ,
where 𝛷 𝑟 is the cumulative distribution function of a bivariate normal with correlation r, and 𝑊~ 𝑊 2 (𝑁−2,𝛴) is the Wishart density, with 𝛴= 𝜎 1 2 𝜌 𝜎 1 𝜎 2 𝜌 𝜎 1 𝜎 2 𝜎 Without loss of generality, 𝜎 1 2 = 𝜎 2 2 =1
So we can put the type 1 error as an expected value, where the expectation is taken over w1 w2 and w12. We note that the wishart distribution is a function of of the population variance/covariance matrix, rho, sigma1 and sigma 2. while we do not know sigma1 and sigma 2, not that this probability depend on sigma only through a ratio of s/sigma, which is chi-squred with n-2 degrees of freed irrespective of sigma. so we can assume sigma is 1. So the wishart density is just a function of rho.

10. → 𝛴= 1 𝜌 𝜌 1
→ 1− 𝑤 1 =0 ∞ 𝑤 2 =0 ∞ 𝑤 12 =−∞ ∞ 𝑊 𝑤 12 , 𝑤 1 , 𝑤 2 𝛷 𝑟 𝑤 1 𝑁−2 𝐶 𝑟 , 𝑤 2 𝑁−2 𝐶 𝑟 𝑑 𝑤 12 𝑑 𝑤 1 𝑑 𝑤 2 (1)
Note that (1) is a function of an unknown correlation 𝜌. It can however be maximized for all −1≤𝜌≤1. The 𝐶 𝑟 ′𝑠 for which the maximum of (1) for all −1≤𝜌 ≤1 is ≤𝛼 ensure control of type-1 error. Denote them by 𝐶 𝑟
Because we don’t know rho, we can calculate this probability for all rho to the maximum type-1 error for a given critical value

11. Critical 𝛼 𝑟 as a function of the sample correlation r

12. Sample Size 10 30 80 160 ∞ Sample Correlation <=0 0.0125 0.05
Sample Size
Sample Correlation 10 30 80
160 ∞
<=0
0.0125
0.05
0.0126
0.10
0.15
0.20
0.0127
0.25
0.0128
0.30
0.0129
0.35
0.0130
0.40
0.0131
0.45
0.0132
0.50
0.0134
0.55
0.0135
0.0136
0.60
0.0138
0.0139
0.65
0.0141
0.0142
0.70
0.0133
0.0144
0.0145
0.75
0.0149
0.0150
0.80
0.0137
0.0154
0.0155
0.85
0.0161
0.0163
0.90
0.0171
0.0173
0.95
0.0151
0.0176
0.0186
0.0188
0.0189

13. Bonferroni r- adjusted Hochberg/Hommel 𝑧 1− 𝛼 2 𝑧 1− 𝛼 𝑟 𝑧 1−𝛼 𝑧 1−𝛼

14. Sample Size Rho r-Adj HH BON 30 Effect Size: 1, 1
Effect Size: 1, 1
Effect Size: 0.5, 1.25
Effect Size: 0, 1.25
87.5
87.4
86.1
87.6
87.0
85.0
84.9
0.5
81.2
80.7
79.1
86.0
85.4
85.2
85.5
84.8
0.7
78.5
77.6
75.6
86.3
86.2
0.9
75.5
74.1
70.8 80
Effect Size: 0.6, 0.6
Effect Size: 0.3, 0.75
Effect Size: 0, 0.75
88.1
88.7
88.2
88.4
87.9
85.7
81.7
81.8
80.4
86.6
85.9
76.9
86.9
85.6
76.5
74.6
72.0
88.3 ∞
Effect Size: 0.35, 0.35
Effect Size: 0.2, 0.45
Effect Size: 0, 0.45
87.3
87.2
89.8
90.2
89.7
80.8
81.0
79.7
86.8
78.1
76.1
88.0
86.7
73.6
71.0
89.4

15. Conclusion
The explicit use of the sample correlation to test two endpoints can have a marked increase in power
Improvement of any procedure whose critical points are calculated conservatively, such as Hochberg and Hommel is possible by incorporating the sample correlation. Such improvements have been suggested before (Dunnett and Tamhane 1992) in the case of known correlations

16. r- adjusted Hochberg/Hommel 𝑧 1− 𝛼 2 𝑧 1− 𝛼 𝑟 𝑧 1−𝛼 𝑧 1−𝛼 𝑧 1− 𝛼 𝑟

17. References
Dunnett and Tamhane (1992). A Step-Up Multiple Test Procedure
Hochberg (1988) A sharper Bonferroni procedure for multiple tests of significance
Hochberg and Rom (1995) Extensions of multiple testing procedures based on Simes' test
Hommel (1988) A stagewise rejective multiple test procedure based on a modified Bonferroni test
Samuel-Cahn (1996). Is the Simes improved Bonferroni procedure conservative?
Sarkar (1998) Some probability inequalities for ordered MTP 2
random variables: a proof of the Simes conjecture