1 Controlling False Positive Rate Due to Multiple Analyses Controlling False Positive Rate Due to Multiple Analyses Unstratified vs. Stratified Logrank Test Peiling Yang, Gang Chen, George Y.H. Chi DBI/OB/OPaSS/CDER/FDA The view expressed in this talk are those of the authors and may not necessarily represent those of the Food and Drug Administration.

2 Motivation: Example of Drug X

3 Issues to Explore Implication of these tests/analyses. Eligibility of efficacy claim based on these tests/analyses. Practicability of multiple testing/analyses.

4 Outline Notations / Settings Introduction to logrank test –Unstratified, stratified Comparisons –Hypotheses, test statistic, test procedure, inference Practicability of hypotheses Testing Multiple testing/analyses Example of Drug X Summary

5 Settings / Notations 2 arms (control j=1; experimental: j=2). K strata: k=1,.., K Patients randomized within strata t 1 < t 2 < …< t D : distinct death times d ijk : # of deaths & Y ijk : # of patients at risk at death time t i, in j th arm & k th stratum.

6 Settings / Notations

7 Hazard ratio (ctrl./exper.): constant –Across strata: c –Within stratum: c k Non-informative censoring

8 Introduction: Unstratified Logrank

9 W u ~ N(0,1) under least favorable parameter configuration (c=1) in. Reject if W u > z . Type I error rate is controlled at level .

10 Introduction: Stratified Logrank

11 Introduction: Stratified Logrank W s ~ N(0,1) under least favorable parameter configuration (c k = 1 for all k) in. Reject if W s > z . Type I error rate is controlled at level .

12 Comparison of Hypotheses Different hypotheses formulations:

13 Comparison of Test Statistics Corr(W u, W s ) = 1 because of same r.v. d.1. W s = a W u + b, wherewhere W u ~ N(0, 1)  W s ~ N(b, a 2 )

14 Comparison of Test Procedure

15 Comparison of Test Procedure

16 Comparison of Inference Rejection of : –Infer overall positive treatment effect in entire population. Rejection of : –Can only infer positive treatment effect in "at least one stratum". –Further testing to identify those strata required to make claim & error rate for identifying wrong strata also needs to be controlled.

17 Practicability of Hypotheses Testing Unstratified hypotheses are tested when desired to infer overall positive treatment effect in entire population. Stratified hypotheses are tested when desired to infer positive treatment effect in certain strata. Multiple testing of both unstratified & stratified hypotheses ok when not sure whether treatment is effective in entire population or certain strata (but both nulls need to be prespecified in protocol).

18 Multiple Testing/Analyses Multiple testing unstratified (use W u ) & stratified (use W s ) hypotheses. Error to control: strong familywise error (SFE), including the following: –When c  1 & all c k  1: falsely infer c or some c k ’s>1. –When c  1 & some c k ’s>1: falsely infer c>1 or wrong c k ’s>1 Note: parameter space of “all c k  1 but c>1” impossible.

19 Multiple Testing/Analyses c  1 & all c k  1 c>1 & at least one c k >1 impossible space c  1 & at least one c k >1 Property of SFE: FE nested in another FE. FE Which c k >1? Nested FE

20 Example -- Drug X W s = aW u +b, where a = 1.039, b=0.409 Critical value using W s should be adjusted to az  +b. False positive error rate using W s w/o adjustment = 0.066; –Inflation = = Ans.: This finding is not statistically significant. for

21 Figure 1: False positive rate vs. desired level (w/o adjustment)

22 Summary Hypotheses (unstratified or stratified or both) –should reflect what is desired to claim. –need to be prespecified in protocol. If stratified null is rejected, further testing required to identify in which strata treatment effect is positive. Strong family error rate needs to be controlled regardless of single or multiple testing.