Kenneth F. Schulz, PhD, MBA Triangle Global Health Consortium Breakfast Discussion

Outcome Outcome Placebo 40% Non- compliance 60% Compliance Policy of no treatment 25% Non- compliance 75% Compliance Policy of treatment Treatment Randomize Participants with Infection

Outcome Outcome Treatment A 97% Treatment A 3% Treatment B Policy of Treatment A 33% Treatment A 67% Treatment B Policy of Treatment B Treatment B Randomize Participants with Infection

8,198 Placebo 22% Non-compliant  < 4 doses  Outside time period  24 HIV + 78% Compliant  50 HIV+ Policy of administering placebo 25% Non-compliant  < 4 doses  Outside time period  15 HIV+ 75% Compliant  36 HIV+ Policy of administering vaccine 8,197 Vaccine Randomize 16,395 Outcome Outcome Participants without HIV Infection

Per-protocol traditionally used in preventive vaccine trials  Completed immunization series  With the assigned vaccine  Within the allotted window of time  Moreover, follow-up for counting cases of disease typically begins a few weeks after the last dose –Vaccine-induced response is believed “adequate”

Per-protocol similar to ITT in vaccine trials  Levels of full compliance are frequently as high as 90-95% in vaccine trials  Moreover, compliance is easy to assess –Vaccinations given by health care providers  And instances of noncompliance usually appear to be unrelated to treatment –Conjecture... cannot be proven to be  Dropping out of a vaccine trial due to toxicity is uncommon –Vaccines extremely benign compared to therapies

Per-protocol similar to ITT in vaccine trials  If few cases of disease occur before the participants are “fully immunized” and the conditions are met from the prior slide  Then, “it may be possible to estimate the biological efficacy of a full vaccination series without a large magnitude of bias.”  “... There should be little risk that an ineffective preventive vaccine will be licensed if conclusions are drawn based on a per- protocol analysis.” Horne AD, Lachenbruch PA, Goldenthal KL. FDA

Excluding Post-Randomization, Before Series Completion, Outcomes  Intuitively attractive –However, the same argument could be used to exclude all outcomes, e.g., in a placebo group  At best, if a priori, cannot improve upon randomization –Only can worsen internal validity... Additional bias  If a posteriori, certainly biased –Investigators observe results, then conjure up theoretically justifiable rules that favor their hypotheses –Severely biased, although will appear logical in the paper Post hoc lucidity

Exclusions After Randomization (Per-protocol)  Damage internal validity –Can introduce bias –Avoided in design and conduct –Carefully scrutinized in reports  All randomized patients should be analyzed –and analyzed as part of the group to which they were initially assigned –ITT (intent-to-treat) analysis  Secondary, non-ITT can be performed –Labeled as non-randomized comparisons

800 Low Risk 2 HIV+ 1,000 Placebo 200 High Risk 40 HIV+ 800 Low Risk 8 HIV+ Per-protocol Placebo 160 High Risk 792 Low Risk 200 High Risk 10 HIV+ Per-protocol Vaccine 190 High Risk 798 Low Risk 1,000 Vaccine Randomize 2,000 Outcome Outcome Participants without HIV Infection

End Slides after this will only be used to clarify a point if brought up in discussion

RCT Compared the Effectiveness of Clofibrate in Preventing Cardiac Deaths in Men Who Had Survived a Myocardial Infarction ClofibratePlacebo 5 Year mortality 20.2%20.9%(p =.55) Authors state that: One can justify almost any conclusion, dependent upon the analysis chosen Manipulating deviates leads to severe bias Can you ever do so? Eliminating deviates from clofibrate (80% adherence) 15.0%20.9%(p <.05) Eliminating deviates from both groups 15.0%15.1%

Thank you for your thoughtful and comprehensive treatment of randomization issues in controlled trials. This comes with hope that you have a few minutes to respond to a query. I am currently an investigator at the end of a 4 year... trial, writing up a paper I would like to submit to JAMA. I prepared the randomization list and concealment according to procedures you recommended in the 1995 article... During the course of the study, however, my colleagues and I (naïve about applying an intention-to-treat analytic plan) deviated from the original generation list. When, three quarters of the way through sample recruitment, treatment participants were missing all sessions, we assumed this group could be treated as "controls," as they received no treatment. Thus, I took any new "control" envelopes out of the randomization sequence, leaving only the treatment assignments. Dear Dr Schulz ( )

Soon thereafter, realizing our mistake, but with only 8-10 slots left in enrollment, I put all the controls back in (all of the last assignments were control, in an attempt to re-balance the randomization). Again, my assignment staff were never aware of any deviations in the protocol. The overall effect is that there are approximately 8 fewer controls than there are for the two treatment groups, but we have worked closely with a statistician to take this deviation into account in the analysis. My question is to what extent (in how much detail) would you describe this problem to the JAMA reader, and how egregious a mistake is this, in your opinion? Many thanks in advance for your time and consideration. Dear Dr Schulz ( , continued)

Flow Chart: Withdrawals but ITT The Lancet 2004; 364: 772

Exclusions of LFU Damage Internal Validity  Without outcomes from those lost to follow- up, investigators have little choice but to exclude them from the analysis  Any losses damage internal validity – However, differential rates of loss among comparison groups cause major damage  Investigators must minimize their losses to follow-up

Summary: Per-protocol analyses and “Non-Analyzable” Outcomes  Per-protocol anlayses, if specified a priori, cannot improve upon randomization – But, could lead to bias  If instituted a posteriori, biased and unethical – Even though it may sound logical in paper  Critically, readers cannot determine if per- protocol analysis was really instituted before (protocol) or after – Therefore, using just a rule can taint an article