Webinar Overview – Host HOSTED BY: Ronan Fitzpatrick Head of Statistics FDA Guest Speaker nQuery Lead Researcher Guest Lecturer

Webinar Overview – Content Sample Size and Sensitivity Analysis Introducing Assurance with nQuery Demo Assurance Discussion and Recommendations Content> nQuery Assurance Plans

Part 1 Sample Size & Sensitivity Analysis

Sample Size Determination Review Sample Size Determination (SSD) is the process for finding the appropriate sample size for your study. Common metrics: statistical power, interval width or cost SSD seeks to balance ethical and practical issues This has made it Standard Trial Design Requirement SSD is crucial to arrive at valid conclusions Yet Many Studies Still have Insufficient Sample Size Biggest SSD challenge is dealing with uncertainty

What Is Statistical Power? The most commonly used metric for sample size determination is statistical power The power is the probability that the study will be able to detect a true effect of a specified size. In other words, Power is the probability of rejecting the null hypothesis when it is false Traditional statistical power and assurance are intimately related concepts

5 Essential Steps for Sample Size 1 Formulate the study Study question, primary outcome, statistical method 2 Specify analysis parameters Standard deviation, ICC, dispersion 3 Specify the Effect Size for Test Expected/targeted difference or ratio 4 Compute Sample Size or Power N for specified power or power for fixed N 5 Perform Sensitivity analysis Some parameters more uncertain and/or influential

Issues With Sensitivity Analysis Sensitivity analysis provides an overview of the effect of varying the effect size and/or analysis parameters Important to understand robustness of sample size estimate and dispel over-confidence in fixed estimates But traditionally only involves assessing a small number of potential alternative scenarios No rules for choosing scenarios and how to pick between them How to improve upon or assist with this process?

Part 2 Assurance Introduction & Demo

Bayesian Sample Size Sample Size for Bayesian Methods Sample size for specific values of Bayesian parameters e.g. Bayes Factors, Credible Intervals, Utility/Cost function Bayesian Approaches to Improve Sample Size Integrating Bayesian methods into current methods to add greater context for parameter uncertainty e.g. Assurance, Predictive Power, Adaptive One Sample Credible Interval with Known Precision One Sample Credible Interval with Unknown Precision One Sample Mixed Bayesian Likelihood Criterion Two Sample Credible Interval with Known Precision Two Sample Credible Interval with Unknown Precision Assurance for Superiority Trial Comparing Two Means Assurance for Equivalence Trial Comparing Two Means Assurance Non-inferiority Comparing Normal Means

Assurance for Clinical Trials Assurance (sometimes called “Bayesian power”) is the unconditional probability of significance given a prior Focus on methods proposed by O’Hagan et al. (2005) Assurance is the expectation of the power averaged over a prior distribution for the effect and/or parameters Often framed the “true probability of success” of a trial Will focus on simple two-sample normal case in our demonstration

Assurance and Sensitivity Analysis In a sensitivity analysis, a number of scenarios are chosen by the researcher and assessed individually for power or N This gives a clear indication of the merits of the individual cases highlighted but no information on other scenarios With assurance, we have the average power over all plausible values by assigning a prior to one/more parameters This provides a summary statistic for the effect of parameter uncertainty but less information on specific scenarios.

Introducing nQuery Over 20 Years of Experience in Sample Size Determination and Power Analysis for Clinical Trials Latest Release had Methods for >200 Trial Designs Used by 45/50 Top Pharma and Biotech Companies nQuery’s over 20 Years of Success is Based on: Being Easy to Use and Accessible to All Users Being Fully Validated and an Industry Standard Quickly Integrating Innovations and Customer Requests

Assurance Example 1 “The outcome variable … is reduction in CRP after four weeks relative to baseline, and the principal analysis will be a one-sided test of superiority at the 2.5% significance level. The (two) population variance … is assumed to be … equal to 0.0625. … the test is required to have 80% power to detect a treatment effect of 0.2, leading to a proposed trial size of n1 = n2 = 25 patients … For the calculation of assurance, we suppose that the elicitation of prior information … gives the mean of 0.2 and variance of 0.0625. If we assume a normal prior distribution, we can compute assurances with m = 0:2, v = 0.06 … With n = 25, we find assurance = 0.595 Source: Wiley.com Parameter Value Significance Level (One-Sided) 0.025 Prior Mean Difference 0.2 Prior Difference Variance 0.06 Posterior Standard Deviation √0.0625=0.25 Sample Size per Group 25

Assurance Example 2 “We estimated that we would need to enrol 163 patients, given an expected mean (±SD) annual decline in the FVC of 9±16 percent of the predicted value and a dropout rate of 15 percent, to achieve a two-sided alpha level of 0.05 and a statistical power of 90 percent.” Source: nejm.com (2006) Parameter Value Significance Level (Two-Sided) 0.05 Mean Difference (%) -9 Standard Deviation (%) 16 N per Group (post drop-out) 0.85(163/2)= 68 Target Power 90%

Part 3 Assurance Discussion and Recommendations

Technical Issues With Assurance How to decide on a prior for the assurance calculation? Previous studies, educated guess, expert elicitation Sheffield Elicitation Framework (SHELF) has industry interest Able to use a non-normal prior? Yes, but no analytical solution. Requires usage of simulation More flexible methods available in nQuery in Spring 2018 Assurance for alternative data types or decision criteria? Methods in literature for proportions and survival (Spring 2018) Assurance concept technically generalizable to any criteria

Recommendations For Assurance Assurance instead of power to find the original sample size? Would usually recommend “no” due to the nature of chosen effect size If the effect size is the “expected” size it might make sense but are we interested in non-clinically relevant powers? Does assurance measure for the question of interest or just the p-value? Does Assurance replace sensitivity analysis completely? Has many advantages but sensitivity analysis more “tactile”? Assurance is vital contextual tool in the planning toolbox Places uncertainty at the heart of the sample size determination process

Part 4 nQuery Assurance Plans

nQuery Assurance Plans Additional assurance methods for more complex probability distributions for prior (parametric, empirical) Assurance methods for other data types (proportions, TTE etc.) and statistical hypotheses (equivalence etc.) Assurance methods for joint probability distributions for multiple parameters and where multiple hypotheses exist Greater access and plotting options to explore assurance results

nQuery Features & Capabilities nQuery 100s of sample size tables to cover your trial design Installation & Operational Qualification Tools Expert support available from our Sample Size Experts Bayes Bayesian Assurance – Get an informative answer on how likely it is to see a “positive” result from a trial Posterior Credible Intervals – Use prior information from pilot studies and other sources to make quicker and better decisions

New nQuery Tables Survival (time-to-event) Analysis The two major area of focus in this release is on: Survival (time-to-event) Analysis Bayesian Statistics New Survival Tables New Bayes Tables Bonus Tables - n-of-1 Trials - Gamma Regression

nQuery Advanced For further details about any item discussed today, email us at info@statsols.com Thanks for listening!

References O'Hagan, A., Stevens, J. W., & Campbell, M. J. (2005). Assurance in clinical trial design. Pharmaceutical Statistics, 4(3), 187-201. Dallow, N., Best, N., & Montague, T. (2017). Better Decision Making in Drug Development Through Adoption of Formal Prior Elicitation. arXiv preprint arXiv:1708.09823. O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., ... & Rakow, T. (2006). Uncertain judgements: eliciting experts' probabilities. John Wiley & Sons. Ren, S., & Oakley, J. E. (2014). Assurance calculations for planning clinical trials with time‐to‐event outcomes. Statistics in medicine, 33(1), 31-45.