1. Students Opportunities: Conferences:
Free Opportunities:
Free webinars for members
Free student membership
Professional Growth:
Affordable trainings at conferences
Get mentoring from professionals
Students Opportunities:
Scholarships
Travel grants
Student paper awards
Conferences:
Joint Statistical Meeting
Regulatory-Industry Stat. Workshop
Nonclinical Biotatistics Conference
Learn more by visiting the website:

2. Descending from the theoretical to the practical
Our journey from clear definition to messy inference and back
Michael O’Kelly

3. Objectives of a phase III clinical trial
Outline
Objectives of a phase III clinical trial
The practice of deciding what to estimate and then estimating it, in order to fulfil those objectives
As it was
As envisioned
As it really is and as it may be in the future
Michael O’Kelly IQVIA JSM Aug 2019 Estimands: from the theoretical to the practical

4. Objectives of a phase III clinical trial
Given the logic of commercial development, objective of clinical trial is assumed here to be
to obtain approval for as wide a use as possible of a new treatment, insofar as this is consistent with patient safety.

5. Objectives of a phase III clinical trial
Given the logic of commercial development, objective of clinical trial is assumed here to be
to obtain approval for as wide a use as possible of a new treatment, insofar as this is consistent with patient safety.
A business-like approach might be to
identify objective
decide what we want to estimate in order to achieve the objective
find or invent an estimator that will estimate what we decided to estimate

6. Objectives of a phase III clinical trial
Given the logic of commercial development, objective of clinical trial is assumed here to be
to obtain approval for as wide a use as possible of a new treatment, insofar as this is consistent with patient safety.
A business-like approach might be to
identify objective
decide what we want to estimate in order to achieve the objective
find or invent an estimator that will estimate what we decided to estimate
maybe we can’t find or invent exactly the estimator required – then we might revise our ideas of what we want to estimate a little, until we find an estimator that will estimate what we have decided to estimate, and will fulfill the trial objective

7. Objectives of a phase III clinical trial
Over the last decade and more, regulators have stated at conferences that protocols/analysis plans are not clear on what is estimated, and how what is estimated is linked to the trial objective…

8. The practice of deciding what to estimate and then estimating it As it was
Objective

9. The practice of deciding what to estimate and then estimating it As it was
Objective
Estimand

10. The practice of deciding what to estimate and then estimating it As it was
Objective
Estimand

11. The practice of deciding what to estimate and then estimating it As it was
Objective
Estimand

12. Objective Estimand Sensitivity analysis
The practice of deciding what to estimate and then estimating it As it was
Objective
Estimand
Sensitivity analysis

13. Practice as it was – assessment
“As it was”, i.e. the last fifteen years
Assumptions with regard to missing data (or data not used by the estimator) were often stated in the protocol or statistical analysis plan (SAP)
e.g. “outcomes were assumed to be missing at random”.
Exactly what was estimated was not defined explicitly.
It was often recognized that the primary estimate was “shaky” to the extent that not all subjects contributed directly to that estimate.
This “shakiness” or potential lack of credibility in rejecting the null hypothesis was “fixed” by sensitivity analyses.
Sensitivity analyses assessed the robustness of the conclusion to the missing data (to subjects not directly included).

14. Practice as it was – assessment
ICH E9 (R1) now sees what we want to estimate as inextricably tied to the entire subject experience, of all randomized subjects
with the nature and extent of subjects’ contribution to the estimate being part of the definition of that estimand and estimate.
Inference from an ad hoc “convenience” sample from a randomized experiment, bolstered by assessment of robustness to the ad hoc nature of the selection via sensitivity analyses, is implicitly strongly deprecated by ICH E9 (R1).
Accepting (R1) has far-reaching consequences for the scientific process, perhaps even for epistemology in general
(or maybe the logic of (R1) will only hold sway in the realm of clinical trials?)

15. Practice as it was – assessment
Example: MMRM estimator only covers subjects modelable by observations from their own treatment group; sensitivity analyses might attempt to extend inference to all subjects, by assessing dependence of conclusion on that modelability.
In the logic of (R1), use of such sensitivity analyses attempts to extend the inference of the primary analysis informally, i.e. attempts to allow the estimate to cover populations that could not have been covered by the planned estimator due to
the study design
primary analysis framework and/or assumptions
study execution (e.g. subjects were lost to follow-up)
…and is not acceptable.

16. The practice of deciding what to estimate and then estimating it As envisioned
Objective

17. The practice of deciding what to estimate and then estimating it As envisioned
Objective
Estimand

18. Objective Estimand Estimator
The practice of deciding what to estimate and then estimating it As envisioned
Objective
Estimand
Estimator

19. Sensitivity analysis Objective Estimand Estimator
The practice of deciding what to estimate and then estimating it As envisioned
Sensitivity analysis
Objective
Estimand
Estimator

20. The practice of deciding what to estimate and then estimating it As envisioned
Or maybe

21. The practice of deciding what to estimate and then estimating it As envisioned
Objective
Estimand 1
Estimand 2
Estimator 1
Estimator 2

22. Effect of treatment as prescribed Treatment adherence
The practice of deciding what to estimate and then estimating it As envisioned
Objective
Estimand 1
Estimand 2
Effect of treatment as prescribed
Treatment adherence
Estimator 1
Estimator 2

23. Effect of randomized treatment policy
The practice of deciding what to estimate and then estimating it As envisioned
Objective
Estimand 1
Estimand 2
Effect of randomized treatment policy
Effect of treatment taken as prescribed
Estimator 1
Estimator 2

24. The practice of deciding what to estimate and then estimating it As envisioned
Objective
Estimand 1
Estimand 2
Estimator 1
Estimator 2

25. The practice of deciding what to estimate and then estimating it As envisioned
Supplementary analysis
Objective
Estimand 1
Estimand 2
Estimator 1
Estimator 2
Supplementary analysis

26. Practice as envisioned – assessment
“A clinical trial protocol and analysis plan should include a ‘golden thread’ linking clear trial objectives with selection and prioritisation of endpoints and hypotheses for statistical testing or targets for estimation.”
Revised draft Concept Paper on choosing appropriate estimands and defining sensitivity analyses in confirmatory clinical trials, July 25, 2014
Sounds like good science: we are clear as to what we are making inference about.
Note that clear definition implies precision, implies circumscription.
The more aspects of an inference we define, the more limited the inference.

27. “Absolute confidence” needed in making inference from cervical cancer screening – Judge Kevin Cross

28. Attaining absolute confidence

29. Practice as envisioned – assessment: clear definition of population and scenario about which inference is made
The more precisely defined the population to whom our inference applies, all things being equal, the narrower the inference.
Under (R1), we can infer with reference to a wider population, or to a wider variety of scenarios, at a cost, e.g. via a very inclusive estimand or by multiple well-defined estimands.

30. Practice as envisioned – assessment: clear definition of population and scenario about which inference is made
Consistent with the aspiration towards clear definition of estimand and inference, for any estimand, (R1) only allows for sensitivity analyses “focused on the same estimand” (p. 14).
So, no extending (formally or informally) of the inference of the primary estimand via sensitivity analyses, as in the past.
Analyses that vary the scenario or the population defined by the primary estimand informally, can only be presented in a way that is divorced from the primary inference, and cannot directly support the primary inference
(R1) calls these supplementary analyses.

31. The practice of deciding what to estimate and then estimating it Now and in the future (maybe)
Objective

32. The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Objective

33. Objective Multiple estimands?
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Multiple estimands?
Objective
Estimand 3
Estimand 2
Estimand 4
Estimator 3
Estimand 1
Estimator 2
Estimator 4
Estimator 1
Estimand 5
Estimator 5

34. Objective Multiple estimands?
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Multiple estimands?
Objective
Estimand 3
Estimand 2
Estimand 4
Attributable effect
Treatment adherence
Estimator 3
Causal effect in completers
Estimand 1
Estimator 2
Estimator 4
Estimator 1
Effect as prescribed
Estimand 5
Estimator 5
Causal effect in per protocol

35. Objective Multiple estimands?
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Multiple estimands?
Objective
Estimand 3
Estimand 2
Estimand 4
Attributable effect
Treatment adherence
Estimator 3
Causal effect in completers
Estimand 1
Estimator 2
Estimator 4
Estimator 1
Effect as prescribed
Estimand 5
Estimand 6
Estimator 5
Causal effect in per protocol
Estimator 6
Safety

36. Objective Estimand Estimator
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but precisely defined inference by…
Treatment policy strategy, using all outcomes no matter what interfering events happen?
Objective
Estimand
Estimator

37. Objective Estimand Estimator
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Treatment policy strategy, using all outcomes no matter what interfering events happen?
Objective
Estimand
Estimator

38. Objective Estimand Estimator
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Treatment policy strategy, using all outcomes no matter what interfering events happen?
Objective
Estimand
Estimator

39. Objective Estimand Estimator
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Treatment policy strategy, using all outcomes no matter what interfering events happen?
Objective
Estimand
Estimator

40. What treatment effect have we measured here, exactly?
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Treatment policy strategy, using all outcomes no matter what interfering events happen?
Objective
Estimand
What treatment effect have we measured here, exactly?

41. Objective Estimand Estimator
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Coarsening a clinical measure to incorporate the fact that an interfering event occurred?
[Composite strategy]
Objective
Estimand
Estimator

42. Objective E t m n Estimator
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Coarsening a clinical measure to incorporate the fact that an interfering event occurred?
[Composite strategy]
Objective
E t m n
Estimator

43. [Hypothetical strategy]
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Positing a trajectory for the measure after events that interfere with it?
[Hypothetical strategy]
Objective
Estimand
Estimator

44. [Hypothetical strategy]
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Positing a trajectory for the measure after events that interfere with it?
[Hypothetical strategy]
Objective
Estimand
Once upon a time…

45. [Hypothetical strategy]
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Objective
Positing a trajectory for the measure after events that interfere with it?
[Hypothetical strategy]
Once upon a time…
Estimand

46. [Hypothetical strategy]
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Objective
Positing a trajectory for the measure after events that interfere with it?
[Hypothetical strategy]
Use the rich data of a trial to model outcomes for the estimand – estimate treatment effect under a clinically meaningful scenario, for all subjects.
Estimand

47. [Hypothetical strategy]
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Objective
Positing a trajectory for the measure after events that interfere with it?
[Hypothetical strategy]
(R1) “prediction model may use data from other subjects who discontinued treatment but for whom data collection has continued”
Estimand

48. [Hypothetical strategy]
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Objective
Positing a trajectory for the measure after events that interfere with it?
[Hypothetical strategy]
…or model outcomes after rescue using similar subjects who did not receive rescue
Estimand

49. [Hypothetical strategy]
The practice of deciding what to estimate and then estimating it Under (R1): achieve wide but well defined inference by…
Objective
Positing a trajectory for the measure after events that interfere with it?
[Hypothetical strategy]
…or model outcomes after treatment discontinuation using similar subjects in the control arm….
Estimand

50. In conclusion: a question
Does (R1) describe a scientific principle
…or a convention that suits the organisations, the “stakeholders”, the clinical process?
…or neither?

51. In conclusion: a question
Does (R1) describe a scientific principle
…or a convention that suits the organisations, the “stakeholders”, the clinical process?
…or neither?
We can at least be grateful to those who have worked on (R1) for stimulating us to think more rigorously about the epistemology of the clinical trial process.