Is it True? Evaluating Research about a Therapy Allen F. Shaughnessy, PharmD, MmedEd Tufts University School of Medicine Department of Family Medicine David C. Slawson, MD The University of Virginia, Department of Family Medicine

2

Task 3: Clarification Now let’s say that the same patient has heard from a friend that there is a vitamin that will help prevent migraines. What study design could answer the question of whether there is a vitamin that is useful in preventing migraine headaches in this patient?

4 Study Methods to Answer This Question Epidemiology: Patients taking a vitamin are less likely to have migraines Pharmacology: Drug x affects cerebral vasculature in rat brain isolates Case report: “It worked on one patient” Case-series: “It worked on a bunch of patients” Randomized controlled trial: 1/2 get drug, 1/2 placebo. No one knows who ‘til the end who took what

The “Danger” of Natural History Studies For each 1% decline in hemoglobin A1c, decrease in mortality, stroke, CHD,.... Women taking HRT and CVD mortality.

The “Danger” of Natural History Studies Compliance Effect! Known as Factor X

8 Validity Internal validity: How well was the study done? Do the results reflect the truth? External validity: can I apply these results to MY patients?

9 Was it a randomized controlled trial? Randomization is the best protection against being mislead

10 32 controlled trials of anticoagulation in acute MI Results by type of study: Chalmers TC, et al. N Engl J Med 1977;297: The value of randomization

11 Was allocation assignment “concealed”? Did investigators know to which group the potential subject would be assigned before enrolling them?

12 Importance of concealed allocation Trials with unconcealed allocation consistently overestimate benefit by ~40% Schulz KF, Chalmers I, Hayes RJ, et al. JAMA 1995;273: Schulz KF, Grimes DA. Lancet 2002;359: Pildal J, et al. Int J Epidemiol 2007;36: Moher D, et al. Lancet 1998;352:

13 Was allocation assignment “concealed”? Concealed allocation  blinding Blinding can occur without concealed allocation Surfactant in the NICU Allocation can be concealed in an unblinded study PT vs surgery for knee DJD Moseley JB, O'Malley K, Petersen NJ, et al. N Engl J Med 2002; 347:81-8.

14 Potential Subjects Conducting a Study Actual Subjects A B Randomization Blinding, etc Trial starts Concealed Allocation

15 Importance of concealed assignment Meta-analysis of trials evaluating screening mammography In studies in which allocation wasn’t concealed Higher SE status, education level in screened group Age disparity (average 6 mo older in the unscreened group) Richer, smarter, younger Trials with concealed allocation = screening harmful! No effect or increased mortality 20% more mastectomies Lancet Jan 8, 2000; Oct 20, 2001

Potential Subjects Conducting a Study Actual Subjects A B Randomization Blinding, etc Trial starts Concealed Allocation Mammography Study Sign-up Number Group Patient name 1 Mamm. 2No Mamm 3Mamm 4No Mamm 5No Mamm 6Mamm 7Mamm 8No Mamm 9Mamm 10No Mamm 11Mamm 12Mamm 13No mamm 14No mamm 15Mamm Sara Smith Jill Jones Wendy Walsh Linda Lucky

17 Technical Nitpicking? Could this really make a difference? Cumulative database: ~500,000 women Current policy is based on very small differences: Deaths in unscreened women902 Deaths in screened women837 Death difference (of 456,349) 65! Systematic bias is not “random error” for which meta- analysis can compensate

18 Mundus Vult Decipi “The world wishes to be deceived” People would rather be deceived than have the truth cause anxiety Caleb Carr, Killing Time

19 “YOU WANT ANSWERS??!!! “I WANT THE TRUTH!!” “YOU CAN’T HANDLE THE TRUTH!!” Jack Nicholson and Tom Cruise “A Few Good Men”

20 Nonfebrile Seizure Incidence

21 Were all the patients properly accounted for at its conclusion? Complete follow-up? “Intention to treat” analysis? Patients are analyzed in the groups to which they are assigned Attempts to reflect “real world” clinical situations in which not all patients follow treatment recommendations Watch when they compare only compliers with compliers and non- compliers Compliant subjects always do better overall

22 Was study “double-blinded”? Did the patients know to which group they were assigned? Did the treating physician know? Did investigators assessing outcomes know (“triple- blinding” – up to 7 levels!)? Judicial assessor blind + allocation concealment = surgery RCTs Schulz KF. Ann Int Med 2002;136:254-9.

23 Were intervention and control groups similar? See Table 1 of most studies Randomization is best way to avoid bias, though imbalances still can occur (especially if allocation was not concealed) Small differences sometimes are important

24 Task 4.