1 OTC-TFM Monograph: Statistical Issues of Study Design and Analyses Thamban Valappil, Ph.D. Mathematical Statistician OPSS/OB/DBIII Nonprescription Drugs AC Meeting March 23, 2005

2 Outline Introduction Summary of Statistical Issues Current TFM trial Design and Analyses with surrogate endpoints Statistical Issues of Study Design and Analyses Options for Trial Design and Efficacy Criteria using surrogate endpoints

3 Introduction Previous presentations on issues involved in validating surrogate endpoints. In the absence of clinical trials data, FDA still needs to address current products under review. This talk discusses issues related to analysis of data obtained on surrogate endpoints. Does not address clinical relevance of statistical findings or differences in analysis of data based on surrogate endpoints.

4 Summary of Statistical Issues Primary Endpoint of Log reduction in Bacterial Counts from baseline. Data Analyses & Variability Issues –Binary Outcomes –Log Reduction (Mean vs. Median) outcomes Variability in methodology –Study Design and Controls Active Vehicle

5 IndicationControlsEndpointsStatistical tests Sample size /arm Surgical HandScrub Active, Vehicle or Placebo Log Reduction, Binomial t-tests66 PreOperative Skin Prep Active Control Log Reduction, Binomial Paired t-test96 Healthcare Personnel Handwash Active control Log Reduction t-tests54 Current TFM Recommendations

6 Current TFM Recommendations: Issues TFM Recommends: –Randomized –Blinded (to persons determining counts only) –Use of Active Control –Use of Vehicle or Placebo Control (role not clearly specified) However, in the current TFM A non-comparative study design is used in which the test product is not directly compared to the Active Control. Mean log reduction meeting the threshold log reduction has been used to demonstrate efficacy.

7 Although vehicle and placebo controls are mentioned in the TFM, majority of the NDAs only have test product and active control arms. Active controls have only been used for internal validation of study methods. Efficacy assessment does not include a direct comparison of Test product performance to Active control, vehicle or placebo. Current TFM Recommendations: Issues

8 Statistical Issues of Study Design and Analyses Primary Endpoint: Log Reduction –Mean Log Reduction can be influenced by few extreme observations. ( Suggestion: Median log reduction may be another option. Median Log Reduction is less sensitive to few subjects with extreme log reductions or outliers. )

9 The efficacy criteria in the current TFM are based on point estimates and do not include confidence intervals to evaluate variability. Consequently, a few extreme observations can potentially drive the efficacy results. Statistical Issues of Study Design and Analyses

10 Example: Log Reduction: Mean = 2.0, Median = 1.7 Below 2-log: 78%

11 Active Control Test Vehicle Example: Log Reduction in Bacterial Counts Threshold.

12 Active Control Test Vehicle. Threshold Upper Limit Mean Lower Limit

13 Active Control Test Vehicle. Threshold Upper Limit Mean Lower Limit

14 Subject will be classified as a ‘success’ or a ‘failure’ based on meeting the threshold reduction. Advantages: Outcome centered on number of subjects and not on organisms may be more clinically relevant. Effect of Variability is reduced. Disadvantage: This method does not differentiate the magnitude of log reductions among those who meet the criteria for “success”. Primary Endpoint Using Binary Response

15 Example:

16 Study #1 (Surgical Hand Scrub) Treatment Day Mean Median Success (%) Test 1 (1 log) /21 (95%) 2 (2 log) /21 (100%) 5 (3 log) /20 (65%) Active #1 1 (1 log) /12 (75%) 2 (2 log) /12 (67%) 5 (3 log) /12 (75%) Active #2 1 (1 log) /22 (64%) 2 (2 log) /22 (45%) 5 (3 log) /21 (10%)

17 Sample Size Issues In current TFM, sample size is estimated based on allowing a test product to be as much as 20% worse than active control in the mean log reductions. However, the basis for 20% margin is not clearly stated. Majority of the current submissions do not follow the recommended sample size as specified in the TFM and only use a sample size of ~30 subjects per treatment arm.

18 Options for Trial Design and Efficacy criteria Issue 1 –How to analyze the data obtained on the surrogate endpoint of log reductions in bacteria? Issue 2 –How to take into account the variability in the data collected, when measuring effect of the product? Issue 3 –How to take into account the variability in the test methodology?

19 Options for Trial Design and Efficacy criteria Issue 1: How to analyze the data obtained on the surrogate endpoint of log reductions in bacteria? –Mean log reduction It can be influenced by few extreme observations. –Median log reduction Less sensitive to outliers or extreme observations. –Percentage of subjects who meet log reduction criteria Outcome centered on number of subjects who meet the threshold and may provide incentive to study “conditions of use” that provides highest success rates.

20 Options for Trial Design and Efficacy criteria Issue 2: How to take into account the variability in the data collected? –Examine confidence intervals around the “outcomes” as defined on previous slide with a threshold for lower bound of confidence interval PRO: Improvement over examination of point estimates alone. CON: Does not take into account the variability in test method. –Examine confidence intervals around treatment difference between test product and some control PRO: Allows for examination of variability in methodology across treatment arms. CON: May require larger sample size for products with lower success rates.

21 Options for Trial Design and Efficacy criteria Issue 3: How to take into account variability in the test methodology? –“Equivalence”/non-inferiority showing test product is no worse than active control by some margin PRO: Allows comparison with an active treatment to rule out loss of effect relative to active control CON: Lack of constancy of effect of active control in previous studies, possible overlap of effect of active and test product with vehicle, hence no basis to select a non-inferiority margin –Superiority of test product to vehicle AND superiority of active control to vehicle PRO: Given lack of constancy of effect with both active and vehicle controls, allows internal validity of comparisons CON: May require larger sample size than current TFM standards (how much larger depends on product efficacy over vehicle)

22 Test Product Active Control Vehicle S S S = Superiority Controlling Variability in Test Methodology ?

23 Sample Size: Superiority Test

24 Thank you!