False Discovery Rates for Discrete Data Joseph F. Heyse Merck Research Laboratories Graybill Conference June 13, 2008

Graybill.ppt. Introduction u Almost all multiplicity considerations in clinical trial applications are designed to control the Family Wise Error Rate (FWER). u Benjamini and Hochberg (1995) argued that in certain settings, requiring control of the FWER is often too conservative. u They suggested controlling the “False Discovery Rate” (FDR) is a more powerful alternative. u Accounting for the discrete endpoints can further improve the power of FDR (and FWER) methods.

Graybill.ppt. Outline 1. Definition and properties of FDR 2. FDR for discrete data 3. Application: Genetic variants of HIV 4. Summary of simulation results 5. Application: Rodent carcinogenicity study 6. Concluding remarks

Graybill.ppt. Familywise Error Rate (FWER) u Let F = {H 1,H 2 … H K } denote a family of K hypotheses. u FWER = Pr(any true H i  F is rejected).  The procedures currently used for clinical studies are intended to control the FWER  . u Benjamini & Hochberg (1995) proposed controlling the “False Discovery Rate” (FDR) as a more powerful alternative to FWER.

Graybill.ppt. False Discovery Rate (FDR) (Benjamini & Hochberg, 1995) u u When R=0, FDR is defined to be 0.

Graybill.ppt. False Discovery Rate (FDR) (cont’d) (Benjamini & Hochberg) u Example (K=4)

Graybill.ppt. Properties of FDR Control u The B&H sequential procedure controls the FDR at u FDR < FWER and equality holds if K=K 0. u The Hochberg (1988) stepwise procedure compares while the FDR procedure compares u FDR is potentially more powerful than FWER controlling procedures. for independent hypotheses.

Graybill.ppt. Comparing FDR and FWER FDR adjusted P-values < FWER adjusted P-values Example (K=4) Unadjusted P-values FDR-adjusted P-values FWER-adjusted P-values

Graybill.ppt. Modified FDR for Discrete Data u Adjusted P-values for FDR u For discrete data Where is largest P-value achievable for hypothesis i that is less than or equal to P (j).

Graybill.ppt. FDR for Discrete Data u Gain in power for the discrete data FDR comes from the difference. u If endpoint i is not able to achieve a P-value ≤ P (j) then and the dimensionality is reduced. u If endpoint i is able to achieve a P-value ≤ P (j) then and a smaller quantity adds to.

Graybill.ppt. Other Approaches for Discrete Data u Tarone (1990) proposed a modified Bonferroni procedure for discrete data by removing those endpoints unable to reach that level of statistical significance. u Gilbert (2005) proposed a 2 step FDR method for discrete data. 1.Apply Tarone’s method to identify endpoints suitable for adjustment. 2.Apply B-H FDR to those endpoints. u Calculating the FDR adjusted P-value is expected to improve upon these approaches by using the complete exact distribution.

Graybill.ppt. Example: Genetic Variants of HIV Gilbert (2005) compared the mutation rates at 118 positions in HIV amino-acid sequences of 73 patients with subtype C to 73 patients with subtype B. The B-H FDR procedure identified 12 significant positions. The Tarone modified FDR procedure reduced the dimensionality to 25 and identified 15 significant positions. The fully discrete FDR identified 20 significant positions.

Graybill.ppt. Simulation Study for Independent Hypotheses u A simulation study was conducted to evaluate the statistical properties of the FDR controlling methods for discrete data using Fisher’s Exact Test. u Simulation parameters –Number of Hypotheses: K = 5, 10, 15, 20 –Varying numbers of false hypotheses (K-K 0 ) –Background rates chosen randomly from U(.01,.5) –Odds Ratios for Effect Size: OR = 1.5, 2, 2.5, 3 –Sample sizes: N = 10, 25, 50, 100 –  = Tailed

Graybill.ppt. Rate of Rejecting True Hypotheses When All Hypotheses are True (K 0 =K)

Graybill.ppt. Rate of Rejecting True Hypotheses When Some Hypotheses are False (K 0 <K)

Graybill.ppt. Rate of Rejecting False Hypotheses

Graybill.ppt. Other Applications u Analysis of Tier II clinical trial adverse experiences u Trend test analysis of rodent carcinogenicity data u Similar modification applied to Bonferonni adjustment for discrete endpoints

Graybill.ppt. Rodent Carcinogenicity Studies u Long-term carcinogenicity studies typically test candidate drugs in several graded doses and use a vehicle control group. u 50 male and 50 female rodents are randomly assigned to each drug treated group with 100 rodents of each sex assigned to control. Male and female studies are considered separately. u Treatment is administered daily and a terminal necropsy is performed at the end of the study.

Graybill.ppt. Rodent Carcinogenicity Studies (cont’d) u Each individual tumor site encountered is described by a combination of organ or tissue with tumor type. u A statistical analysis of trend is performed for all tumor sites encountered. u An exact test uses the permutation distribution of the trend statistic. u Exact tests can account for age at necropsy and lethality of tumor. u For illustration purposes, this analysis only considered dose and presence of tumor.

Graybill.ppt. Summary of Statistical Results from a Long-Term Carcinogenicity Study in Male Mice Tumor Site Liver, Hepatocellular Carcinoma P.S.U., Hemangiosarcoma Adrenal Cortex, Adenoma P.S.U., Sarcoma P.S.U., Lymphoma Lung, Adenoma Liver, Hepatocellular Adenoma Liver, Hemangiosarcoma Harderian Gland, Adenoma Skin, Fibroma Thyroid, Follicular Cell Carcinoma P.S.U., Leukemia Lung, Adenocarcinoma Testes, Interstitial Cell Tumor Stomach, Papilloma Number of Mice on Study Control Trend P-value N.41N.16N Trend P-value is reported 1-tailed using exact permutational distribution. N indicates 1-tailed P-value for negative trend. Test Agent Dose

Graybill.ppt. Available Methods u Adjusting P-values to account for multiple tumor types. (Mantel 1980, Brown and Fears 1981, Mantel et al. 1982)  Adapting  for interpreting unadjusted P-values (Haseman 1983 and 1990, Lin and Rahman 1998) u Resampling methods to adjust P-values (Heyse and Rom 1988, Westfall and Young 1989, Westfall and Soper 1998) u Bayesian methods using historical control priors (Westfall and Soper 2001)

Graybill.ppt. Multiplicity of Statistical Tests u Liver, hepatocellular carcinoma was only 1 of K=15 tumor sites encountered. u P (1) = was the most extreme individual trend P- value. u Interest is in the likelihood of observing P (1) = as the most extreme P-value among the K=15 in this study. u Need to consider the discrete nature of the data since several tumor sites may not be able to achieve significance levels of P (1).

Graybill.ppt. P-value Adjustment Methods u Mantel (1980) attributed to J.W. Tukey Where k=number of tumor sites that could yield P-values as extreme as P (1). u Mantel et al. (1982) Where is largest P-value achievable for tumor site i that is less than or equal to P (1). u Discrete FDR adjustment: = 0.264

Graybill.ppt. Conclusions u FDR control provides higher power than FWER control when some hypotheses are false.  Proposed procedure based on exact analysis of binomial data controls FDR at  u Discrete nature of data results in slightly conservative FDR control. FDR is less conservative for increasing sample size and increasing numbers of hypotheses. u Accounting for discrete endpoints increases the power of FDR.