Students Opportunities: Conferences: Free Opportunities: Free webinars for members Free student membership Professional Growth: Affordable trainings at conferences Get mentoring from professionals Students Opportunities: Scholarships Travel grants Student paper awards Conferences: Joint Statistical Meeting Regulatory-Industry Stat. Workshop Nonclinical Biotatistics Conference Learn more by visiting the website: https://community.amstat.org/biop/home

Descending from the theoretical to the practical Our journey from clear definition to messy inference and back Michael O’Kelly

Objectives of a phase III clinical trial Outline Objectives of a phase III clinical trial The practice of deciding what to estimate and then estimating it, in order to fulfil those objectives As it was As envisioned As it really is and as it may be in the future Michael O’Kelly IQVIA JSM Aug 2019 Estimands: from the theoretical to the practical

Objectives of a phase III clinical trial Given the logic of commercial development, objective of clinical trial is assumed here to be to obtain approval for as wide a use as possible of a new treatment, insofar as this is consistent with patient safety.

Objectives of a phase III clinical trial Given the logic of commercial development, objective of clinical trial is assumed here to be to obtain approval for as wide a use as possible of a new treatment, insofar as this is consistent with patient safety. A business-like approach might be to identify objective decide what we want to estimate in order to achieve the objective find or invent an estimator that will estimate what we decided to estimate

Objectives of a phase III clinical trial Given the logic of commercial development, objective of clinical trial is assumed here to be to obtain approval for as wide a use as possible of a new treatment, insofar as this is consistent with patient safety. A business-like approach might be to identify objective decide what we want to estimate in order to achieve the objective find or invent an estimator that will estimate what we decided to estimate maybe we can’t find or invent exactly the estimator required – then we might revise our ideas of what we want to estimate a little, until we find an estimator that will estimate what we have decided to estimate, and will fulfill the trial objective

Objectives of a phase III clinical trial Over the last decade and more, regulators have stated at conferences that protocols/analysis plans are not clear on what is estimated, and how what is estimated is linked to the trial objective…

The practice of deciding what to estimate and then estimating it As it was Objective

The practice of deciding what to estimate and then estimating it As it was Objective Estimand

The practice of deciding what to estimate and then estimating it As it was Objective Estimand

The practice of deciding what to estimate and then estimating it As it was Objective Estimand

Objective Estimand Sensitivity analysis The practice of deciding what to estimate and then estimating it As it was Objective Estimand Sensitivity analysis

Practice as it was – assessment “As it was”, i.e. the last fifteen years Assumptions with regard to missing data (or data not used by the estimator) were often stated in the protocol or statistical analysis plan (SAP) e.g. “outcomes were assumed to be missing at random”. Exactly what was estimated was not defined explicitly. It was often recognized that the primary estimate was “shaky” to the extent that not all subjects contributed directly to that estimate. This “shakiness” or potential lack of credibility in rejecting the null hypothesis was “fixed” by sensitivity analyses. Sensitivity analyses assessed the robustness of the conclusion to the missing data (to subjects not directly included).

Practice as it was – assessment ICH E9 (R1) now sees what we want to estimate as inextricably tied to the entire subject experience, of all randomized subjects with the nature and extent of subjects’ contribution to the estimate being part of the definition of that estimand and estimate. Inference from an ad hoc “convenience” sample from a randomized experiment, bolstered by assessment of robustness to the ad hoc nature of the selection via sensitivity analyses, is implicitly strongly deprecated by ICH E9 (R1). Accepting (R1) has far-reaching consequences for the scientific process, perhaps even for epistemology in general (or maybe the logic of (R1) will only hold sway in the realm of clinical trials?)

Practice as it was – assessment Example: MMRM estimator only covers subjects modelable by observations from their own treatment group; sensitivity analyses might attempt to extend inference to all subjects, by assessing dependence of conclusion on that modelability. In the logic of (R1), use of such sensitivity analyses attempts to extend the inference of the primary analysis informally, i.e. attempts to allow the estimate to cover populations that could not have been covered by the planned estimator due to the study design primary analysis framework and/or assumptions study execution (e.g. subjects were lost to follow-up) …and is not acceptable.

The practice of deciding what to estimate and then estimating it As envisioned Objective

The practice of deciding what to estimate and then estimating it As envisioned Objective Estimand

Objective Estimand Estimator The practice of deciding what to estimate and then estimating it As envisioned Objective Estimand Estimator

Sensitivity analysis Objective Estimand Estimator The practice of deciding what to estimate and then estimating it As envisioned Sensitivity analysis Objective Estimand Estimator

The practice of deciding what to estimate and then estimating it As envisioned Or maybe

The practice of deciding what to estimate and then estimating it As envisioned Objective Estimand 1 Estimand 2 Estimator 1 Estimator 2

Effect of treatment as prescribed Treatment adherence The practice of deciding what to estimate and then estimating it As envisioned Objective Estimand 1 Estimand 2 Effect of treatment as prescribed Treatment adherence Estimator 1 Estimator 2

Effect of randomized treatment policy The practice of deciding what to estimate and then estimating it As envisioned Objective Estimand 1 Estimand 2 Effect of randomized treatment policy Effect of treatment taken as prescribed Estimator 1 Estimator 2

The practice of deciding what to estimate and then estimating it As envisioned Objective Estimand 1 Estimand 2 Estimator 1 Estimator 2

The practice of deciding what to estimate and then estimating it As envisioned Supplementary analysis Objective Estimand 1 Estimand 2 Estimator 1 Estimator 2 Supplementary analysis

Practice as envisioned – assessment “A clinical trial protocol and analysis plan should include a ‘golden thread’ linking clear trial objectives with selection and prioritisation of endpoints and hypotheses for statistical testing or targets for estimation.” Revised draft Concept Paper on choosing appropriate estimands and defining sensitivity analyses in confirmatory clinical trials, July 25, 2014 Sounds like good science: we are clear as to what we are making inference about. Note that clear definition implies precision, implies circumscription. The more aspects of an inference we define, the more limited the inference.

“Absolute confidence” needed in making inference from cervical cancer screening – Judge Kevin Cross

Attaining absolute confidence

Practice as envisioned – assessment: clear definition of population and scenario about which inference is made The more precisely defined the population to whom our inference applies, all things being equal, the narrower the inference. Under (R1), we can infer with reference to a wider population, or to a wider variety of scenarios, at a cost, e.g. via a very inclusive estimand or by multiple well-defined estimands.

Practice as envisioned – assessment: clear definition of population and scenario about which inference is made Consistent with the aspiration towards clear definition of estimand and inference, for any estimand, (R1) only allows for sensitivity analyses “focused on the same estimand” (p. 14). So, no extending (formally or informally) of the inference of the primary estimand via sensitivity analyses, as in the past. Analyses that vary the scenario or the population defined by the primary estimand informally, can only be presented in a way that is divorced from the primary inference, and cannot directly support the primary inference (R1) calls these supplementary analyses.

The practice of deciding what to estimate and then estimating it Now and in the future (maybe) Objective

The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Objective

Objective Multiple estimands? The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Multiple estimands? Objective Estimand 3 Estimand 2 Estimand 4 Estimator 3 Estimand 1 Estimator 2 Estimator 4 Estimator 1 Estimand 5 Estimator 5

Objective Multiple estimands? The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Multiple estimands? Objective Estimand 3 Estimand 2 Estimand 4 Attributable effect Treatment adherence Estimator 3 Causal effect in completers Estimand 1 Estimator 2 Estimator 4 Estimator 1 Effect as prescribed Estimand 5 Estimator 5 Causal effect in per protocol

Objective Multiple estimands? The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Multiple estimands? Objective Estimand 3 Estimand 2 Estimand 4 Attributable effect Treatment adherence Estimator 3 Causal effect in completers Estimand 1 Estimator 2 Estimator 4 Estimator 1 Effect as prescribed Estimand 5 Estimand 6 Estimator 5 Causal effect in per protocol Estimator 6 Safety

Objective Estimand Estimator The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but precisely defined inference by… Treatment policy strategy, using all outcomes no matter what interfering events happen? Objective Estimand Estimator

Objective Estimand Estimator The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Treatment policy strategy, using all outcomes no matter what interfering events happen? Objective Estimand Estimator

Objective Estimand Estimator The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Treatment policy strategy, using all outcomes no matter what interfering events happen? Objective Estimand Estimator

Objective Estimand Estimator The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Treatment policy strategy, using all outcomes no matter what interfering events happen? Objective Estimand Estimator

What treatment effect have we measured here, exactly? The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Treatment policy strategy, using all outcomes no matter what interfering events happen? Objective Estimand What treatment effect have we measured here, exactly?

Objective Estimand Estimator The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Coarsening a clinical measure to incorporate the fact that an interfering event occurred? [Composite strategy] Objective Estimand Estimator

Objective E t m n Estimator The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Coarsening a clinical measure to incorporate the fact that an interfering event occurred? [Composite strategy] Objective E t m n Estimator

[Hypothetical strategy] The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Positing a trajectory for the measure after events that interfere with it? [Hypothetical strategy] Objective Estimand Estimator

[Hypothetical strategy] The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Positing a trajectory for the measure after events that interfere with it? [Hypothetical strategy] Objective Estimand Once upon a time…

[Hypothetical strategy] The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Objective Positing a trajectory for the measure after events that interfere with it? [Hypothetical strategy] Once upon a time… Estimand

[Hypothetical strategy] The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Objective Positing a trajectory for the measure after events that interfere with it? [Hypothetical strategy] Use the rich data of a trial to model outcomes for the estimand – estimate treatment effect under a clinically meaningful scenario, for all subjects. Estimand

[Hypothetical strategy] The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Objective Positing a trajectory for the measure after events that interfere with it? [Hypothetical strategy] (R1) “prediction model may use data from other subjects who discontinued treatment but for whom data collection has continued” Estimand

[Hypothetical strategy] The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Objective Positing a trajectory for the measure after events that interfere with it? [Hypothetical strategy] …or model outcomes after rescue using similar subjects who did not receive rescue Estimand

[Hypothetical strategy] The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by… Objective Positing a trajectory for the measure after events that interfere with it? [Hypothetical strategy] …or model outcomes after treatment discontinuation using similar subjects in the control arm…. Estimand

In conclusion: a question Does (R1) describe a scientific principle …or a convention that suits the organisations, the “stakeholders”, the clinical process? …or neither?

In conclusion: a question Does (R1) describe a scientific principle …or a convention that suits the organisations, the “stakeholders”, the clinical process? …or neither? We can at least be grateful to those who have worked on (R1) for stimulating us to think more rigorously about the epistemology of the clinical trial process.