1. August, 2002MCP A General Multiplicity Adjustment Approach and Its Application to Evaluating Several Independently Conducted Studies Qian Li and Mohammad Huque CDER/FDA

2. August, 2002MCP Disclaimer The views expressed in this talk are those of the authors and do not necessarily represent those of the Food and Drug Administration

3. August, 2002MCP Outlines Motivation Extending the union-intersection method Issues in application –relationship of the decision errors and decision rules –choosing a decision error –power characteristics An example and closing remarks

4. August, 2002MCP Motivation More than one independently conducted Phase III study in NDA submissions to support efficacy evaluation Current practice –count the number of studies that are significant –combine studies Interpretation of regulatory requirement –two successful studies –no consistent interpretation when more than two studies are conducted

5. August, 2002MCP Union-Intersection method Roy (1953) first proposed a method of constructing a hypothesis test: H 0 :  K i=1 H 0i, for K hypothesis tests. P H0 (T i > , i=1,2,…,K)=  where  is a critical cut point.

6. August, 2002MCP Extending the UI Approach P H0 (p (1)  1  p (2)  2 ...  p (K)  K )  ’ –where  1   2 ...   K  1 are p-value cut points –p (1), p (2), …, p (K) are ordered p-values of p 1, p 2, …, p K –  ’ is an overall type I error

7. August, 2002MCP Definitions of overall hypotheses Possible choices of overall hypotheses: –at least one alternative is true H 0 1/K :  i=1 K H 0i vs. H A 1/K :  i=1 K H Ai –at least two alternatives are true H 0 2/K :  j=1 K (  i=1 to K,i  j H 0i ) vs. H A 2/K :  j=1 K (  i=1 to K,i  j H 0i ) … –all the alternatives are true H 0 K/K :  j=1 K H 0i vs. H A K/K :  j=1 K H Ai

8. August, 2002MCP Overall type I error and p-value cut points For H 0 1/k, the extended approach can be rewritten as follows when p-values are independent

9. August, 2002MCP Overall type I error and p-value cut points For H A m+1/k (m>1), –max overall type I error occur when m studies have power 1 to reject individual null –  I ’s satisfy the following and  1 =  2 = … =  m   m+1.

10. August, 2002MCP A special case of two hypotheses H 0 1/2 : H 01  H 02 H 01 :  1  0, H 02 :  2  0 The null space is the third quadrant max decision error occur at (  1 =0,  2 =0)  1 and  2 satisfy 2  1  2 -  1 2  ’ More than one set of  1 and  2 For  ’=0.05, if  1 = 0.025,  2 =1 if  1 = 0.05,  2 =0.525 22 11 (0,0)

11. August, 2002MCP Rejection regions 0.525  1 =0.050  2 =0.525  1 =0.025  2 =1.000 1 p1p1 p1p1 p2p2 p2p2 0.05 1 0.025

12. August, 2002MCP Possible decision rules for two studies when  ’= 0.025 2

13. August, 2002MCP Special case of two studies H 0 2/2 : H 01  H 02 The null space is all the area except first quadrant max decision error occurs at (  1 = ,  2 =0) & (  1 =0,  2 =  ) max overall type I error is controlled when  1 =  2  ’ 11 22 (0,0)

14. August, 2002MCP Issues in application Choice of overall hypothesis Choice of decision error Power characteristics

15. August, 2002MCP Relationship of decision errors among different overall hypotheses For a set of K independent p-values, there exists a common rejection region p 1 = p 2 =…=p k . The corresponding decision errors for the overall hypotheses are: H 0 1/k :  ’=  k H 0 2/k :  ’=  k-1 … H 0 k-1/k :  ’=  2 H 0 k/k :  ’= 

16. August, 2002MCP Relationship of decision rules It can be shown that the decision rules derived from a stringent hypothesis can also be derived from a less stringent hypothesis, given the relationship of the decision errors among different overall hypotheses.

17. August, 2002MCP Relationship of decision errors Exist a common rejection region in (p 1, p 2 ) That is to require two significant studies, p 1 = p 2  Decision errors : –H 0 1/2 : H 01  H 02  ’=  2 –H 0 2/2 : H 01  H 02  ’=  p1p1 p2p2   0

18. August, 2002MCP Strategies for choosing decision errors Considering 4 strategies –Use the same decision error for all H A k/k –Use the same decision error for all H A 1/k –Find the decision errors that keep a constant power for H A k/k –Control the power increase for H A k/k

19. August, 2002MCP Same decision error for H A k/k Require all the K studies significant at level  ’ Similar to UI decision rules Power is low Each study has 90% power at level 0.025 K: 12 3 4 5 6 Error:0.0250.0250.0250.0250.0250.025 Power: 90.080.872.965.659.053.1 Not fair to use the same decision error for H A k/k when K is large

20. August, 2002MCP Same decision error for H A 1/K Decision error and power for H A K/K Each study has 90% power at level 0.025 K: 12 3 4 5 6 Error:0.00060.0250.0850.1580.2290.292 Power: 50.081.096.998.799.499.6 When K increases, there is a large increase in decision error, therefore large increase in power

21. August, 2002MCP Keeping consistent power for H A K/K This strategy is in between the first two strategies Decision error and power for H A K/K Each study has 90% power at level 0.025 K: 12 3 4 5 6 Error:0.0090.0250.0400.0540.0660.077 Power: 81.081.081.081.081.081.0 In case not satisfied...

22. August, 2002MCP Increasing power for H A K/K Decide a reasonable power for H A K/K, then figure out the error rate Decision error and power for H A K/K Each study has 90% power at level 0.025 K: 12 3 4 5 6 Error:0.0070.0250.0460.0680.0930.121 Power: 78.681.083.085.087.089.0 Less conservative than the previous strategy

23. August, 2002MCP Power characteristics Power function y K =P HA (p (1)  1  p (2)  2 ...  p (K)  K ) Tedious to write when K>3 Can be evaluated numerically Search the optimal power numerically

24. August, 2002MCP Two studies - H 0 1/2 : H 01  H 02 Power curve for  1 =  2 =1.5,  ’=0.05

25. August, 2002MCP An example Three studies are conducted Use H A 1/3,  ’=0.046 3 p-value cut points are –0.025, 0.025, 0.067 Observed p-values were –0.0065, 0.0125, 0.06

26. August, 2002MCP Remarks having the flexibility to choose p-value cut point allows us to control decision error for multiple studies possible to balance variations among p-values