Why we don’t like protocol violations Yuko Y. Palesch, PhD Medical University of South Carolina

Humans are the worst experimental units.

Because…  They are very heterogeneous

Because…  They are very heterogeneous  They need to be respected and treated fairly and with dignity

Because…  They are very heterogeneous  They need to be respected and treated fairly and with dignity  They have a mind of their own

In addition, those of us who design and conduct the trial are also humans

To Err = Human

Some Errors in ProTECT  Eligibility / Randomization  TBI mimic (e.g., alcohol intoxication) randomized (??%)  Study drug not given within 4 hours (16%)  Study drug administration  Wrong drug given / crossovers (<1%)  Infusion interruptions (12%)  Taper errors (30%)  Missing data  Lost to follow-up (<1%)  Outcomes assessed beyond the time window (13%)  Data quality and timeliness  Incorrect and/or late data entries

But why do we care about these errors? the ITT Analysis

ITT stands for… (choose one) 1.Incentive to Treat 2.Insure to Treat 3.Intent to Treat 4.International Treatment Trial 5.Incredibly Tedious Trial 6.Both 3 and 5

Scenario 1 (misdiagnosis) A patient, upon arrival in the ER was mis-diagnosed to have a TBI and was randomized and treated with the study drug. Shortly thereafter, it was discovered that she was just intoxicated, and terminated from the study. Should the subject be included in the ITT analysis of the primary outcome? A.Yes B.No

Scenario 2 (time delay) An eligible patient was randomized at 3.5 hours from injury, but the study drug was initiated at 5 hours. He died from his injury within two hours of study drug initiation. Should the subject be included in the ITT analysis of the primary outcome? A.Yes B.No

Scenario 3 (crossover) An eligible patient was randomized but, because of the study drug kit accounting error, the “wrong” study drug was administered. Nevertheless, the subject is followed through 6 months per the protocol. Should the subject be included in the ITT analysis of the primary outcome? A.Yes B.No

Scenario 4 (drug admin error) An eligible patient was randomized but, because of a variety of reasons, including infusion interruptions, only 1/2 of the study drug dose was administered. The subject was lost-to-follow- up at 3 months. Should the subject be included in the ITT analysis of the primary outcome? A.Yes B.No

Scenario 5 (non-EFIC Trial) An eligible patient is randomized and treated with study drug, but it is discovered upon site monitoring 2 months later that a signed and dated Informed Consent was not obtained prior to randomization. Should the subject be included in the ITT analysis of the primary outcome? A.Yes B.No

Scenario 6 (consent withdrawal) An eligible patient is randomized at 2 hours from injury and IC obtained from the LAR within the hour. But the family asks for DNR before the study drug is administered and withdraws consent. Should the subject be included in the ITT analysis of the primary outcome? A.Yes B.No

Definition of ITT Analysis Analysis that includes all randomized patients in the groups to which they were randomly assigned, regardless of their adherence with the entry criteria, regardless of the treatment they actually received, and regardless of subsequent withdrawal from treatment or deviation from the protocol. Fisher et al. In: Statistical issues in drug research and development. New York:Marcel Dekker, pp , 1990.

Effect of errors on the statistical test and interpretation of results

Primary outcome is dichotomous - good vs bad: * Using alpha (two-sided) = 0.05 and power = 85%; accounting for two planned interim analysis. Group Hypothesized % good outcome N* PRG60%462 PLC50%462 Tx effect∆=10% ProTECT Trial 1º Hypothesis

A Hypothetical Scenario Subject Types PRG GroupPLC Group ABCDEF How many? % Good outcome among them Col. A x Col. B How many? % Good outcome among them Col. D x Col. B Misdiagnose alcohol intoxication 1%90%0.0091%90%0.009 Time to study drug initiation >4 hrs 6%50%0.036%50%0.03 Crossovers1%50%0.0051%60%0.006 Drug infusion errors10%50%0.0510%50%0.05 Remainder (i.e., protocol compliers) 82%60% %50%0.41 Observed good outcome 58.6% 50.5% Observed difference 8.1% (instead of 10%) Hence, dilution of treatment effect Assume true PRG and PLC % good outcome are 60% and 50%, respectively

 With N=924, we only have 67% power to detect a difference of 8.1%, even if the true PRG effect is 10% better than PLC.  OR we’d need an additional 482 subjects to keep the original 85% power (or a total of N=1,404) (NOTE: Max total N planned in protocol = 1,140 to account for some dilution effect)  Therefore, because of the errors, we may fail to achieve statistical significance, and the ProTECT Trial will be deemed negative / neutral study, even if PRG is an effective treatment.  Dilution of Tx Effect

Scenario 4: Lost to follow-up Scenario 6: Withdrawal of consent  Missing data  Need to impute (make up) data for ITT  Will also contribute to the dilution of the tx effect  May cause biased results, especially if Missing NOT At Random (MNAR) Bias Effect

“…no statistical analyses can ever adequately adjust for missing data, despite many techniques that attempt to do so.” DeMets DL. Journal of Internal Medicine 2004; 255:529–537 Bias Effect (cont’d)

So why do the ITT analysis? Why not do Per Protocol analysis?

Problem with PP Analysis  Assumes that the non-compliers do not differ in their state of health from those who comply.  And that the decision to comply is not itself influenced by treatment.  If assumptions are incorrect, we have subset selection bias which causes increase in false positive errors.  In practice, EXTREMELY difficult to determine who belongs in PP analysis.

Anturane Infarction Trial Example Temple R, Pledger GW. The FDA's critique of the Anturane Reinfarction Trial. New Engl J Med 1980; 303: 1488–92. Compared sulfinpyrazone vs placebo in post-heart attack patients. Original study results only used those deemed “eligible” post-hoc with p-value=0.07. Results of re-analysis by the FDA:

Anturane Infarction Trial Example Temple R, Pledger GW. The FDA's critique of the Anturane Reinfarction Trial. New Engl J Med 1980; 303: 1488–92. Compared sulfinpyrazone vs placebo in post-heart attack patients. Original study results only used those deemed “eligible” post-hoc with p-value=0.07. Results of re-analysis by the FDA:

Conclusion To ensure correct statistical inference from the study:  Minimize randomization and implementation errors.  Avoid missing data.  Enter data in a timely manner so that the interim analysis can be performed with complete and accurate data.