EBM Workshop: Treatment Abdolmehdi Baghaei MD. Resident of Internal Medicine Oct - 2016

OPTIONAL COMPONENTS TO BE ADDED BY THE PHYSICIAN PATIENT Values, Preferences Concerns, Expectations Life predicament HUMILITY Non-authoritarian practice CHARITY EBM is not a required practice (yet) EBM Charity – EBM not mandatory (not yet) Humility – EBM is not for those who prefer an authoritarian practice of medicine Enthusiasm – have fun with the challenge, variety and change!!! PHYSICIAN Training Expertise Continued Learning Demand for proof INFORMATION Clinically relevant Proven by research Current, up to date ENTHUSIASM Challenge, Variety, Change

THE FIVE BASIC STEPS OF EBM Clinical Question Patient-focused, problem-oriented 2. Find Best Evidence Literary Search 3. Critical Appraisal Evaluate evidence for quality and usefulness 4. Apply the Evidence Implement useful findings in clinical practice 5. Evaluate The information, intervention, and EBM process

Objectives: To define characteristics of a suitable report of clinical trials To define the items of paper evaluation in clinical practice To demonstrate peer review check lists for papers that report clinical trials

EBM QUESTION: Should include multiple factors (Examples) P PATIENT type of patient or population Ex: 47 yr male w/DM2 and cellulitis toe, 25 yr female w/DVT and chest pain E EXPOSURE environmental, personal, biological Ex: TB, tobacco, drug, diet, pregnancy or menopause, MRSA, allergy I INTERVENTION clinical intervention Ex: medication, procedure, test, surgery, radiation, drug, vaccine C COMPARISON compare alternative treatment Ex: other prior, new or existing therapy O OUTCOME clinical outcome of interest Ex: Reduced death rate in 5 yrs, decreased infections, fewer hospitalizations

FRAMING THE QUESTION (Example: PICO) ELEMENT PROMPTS THE QUESTION: Patient How would I describe a group of patients similar to mine? Intervention What main action am I considering? Comparison What is/are the other options? Outcome What do I (or the patient) want to happen (or not happen)? Example: P: In kids under age 12 with poorly controlled asthma on metered dose inhaled steroids… I: would the addition of salmetrol to the current therapy C: compared to increasing the dose of current steroid O: lead to better control of symptoms without increasing side effects?

CATEGORY OF QUESTION MAJOR CATEGORIES Diagnosis Prognosis Therapy/ Treatment PICO Harm (iatrogenic, other) PEO MISCELLANEOUS Quality of care Health economics Office Management Etc.

CRITICAL APPRAISAL Interpreting the evidence How to read a paper How to do the math

CRITICAL APPRAISAL IMPORTANT! You do NOT have to become a researcher, epidemiologist, or statistician to practice EBM. Focus on how to USE research reports – not on how to generate them!

Four levels of evidence: To help clinicians rank quality between evidence sources David Sackett, MD, popularized the evidence-based medicine pyramid. Four levels of evidence: Most desirable at top

Internal validity Components of internal and external validity of controlled clinical trials Internal validity Extent to which systematic error (bias) is minimized in clinical trials How to Evaluate Assessment Index Biased allocation to comparison groups Selection bias Unequal provision of care apart from treatment under evaluation Performance bias Biased assessment of outcome Detection bias Biased occurrence and handling of deviations from protocol and loss to follow up Attrition bias

External validity Components of internal and external validity of controlled clinical trials External validity Extent to which results of trials provide a correct basis for generalization to other circumstances. How to Evaluate Assessment Index Age, sex, severity of disease and risk factors, comorbidity Patients Dosage and route of administration, type of treatment within a class of treatments, concomitant treatments Treatment Regimen Level of care (primary to tertiary) and experience and specialization of care provider Setting Type or definition of outcomes and duration of follow up be assessed Modalities of outcomes

Quantifying Treatment Effects

Rationale Any treatment involves tradeoffs Weigh benefits against risks/costs Benefit $$ Harm

Rationale Sometimes the decision is difficult! $$ Harm Benefit

Rationale And this one? How big is this box? $$ Harm Benefit

Rationale Tests can help us understand who is most likely to benefit from a treatment And this one? How big is this box? $$ Harm Benefit

Rationale Tests can help us understand who is most likely to benefit from a treatment Rapid strep to decide who will benefit from penicillin BNP to decide who will benefit from furosemide CRP to decide who will benefit from statins

Rationale The utility of a test depends on: How beneficial the treatment is How harmful the treatment is How much the test tells us about these benefits and harms in a given individual Risk of harm from the test itself

Rationale The topic for this lecture The utility of a test depends on: How beneficial the treatment is How harmful the treatment is How much the test tells us about these benefits and harms in a given individual Risk of harm from the test itself The topic for this lecture

Is the intervention beneficial? Randomized trials compare an outcome in treated to untreated persons MI in 10% vs. 15% Duration of flu symptoms 3 vs. 5 days

Is the intervention beneficial? Randomized trials compare an outcome in treated to untreated persons MI in 10% vs. 15% Duration of flu symptoms 3 vs. 5 days *Statistics* are used to decide if should reject the “null hypothesis” and accept that the intervention is beneficial

Is the intervention beneficial? But statistics cannot help us interpret effect size

Quantifying the Benefit Effect size How do we summarize and communicate this? What is really important for clinicians and policymakers? Clinical Significance Vs Statistical Significance

Quantifying the Benefit Effect size How do we summarize and communicate this? What is really important for clinicians and policymakers? Example: MI in 10% vs. 15% Q: What do we do with these two numbers?

Quantifying the Benefit Two simple possibilities: 10% / 15% = 0.66 15% - 10% = 5%

Quantifying the Benefit Two simple possibilities: 10% / 15% = 0.66 15% - 10% = 5% Relative Risk (RR) Absolute Risk Reduction (ARR)

Quantifying the Benefit Relative risk as a measure of effect size RR = 0.66 – is this big or small?

Quantifying the Benefit Relative risk as a measure of effect size RR = 0.66 – is this big or small? MI: 10% vs. 15% in 10 years Death: 50% vs. 75% in 3 years Basal Cell CA: 2% vs. 3% in lifetime

Quantifying the Benefit Relative risk as a measure of effect size RR = 0.66 – is this big or small? MI: 10% vs. 15% in 10 years Death: 50% vs. 75% in 3 years Basal Cell CA: 2% vs. 3% in lifetime Medium Big Small

Quantifying the Benefit Relative risk as a measure of effect size RR = 0.66 – is this big or small? MI: 10% vs. 15% in 10 years Death: 50% vs. 75% in 3 years Basal Cell CA: 2% vs. 3% in lifetime RR is NOT the best measure of effect size

Quantifying the Benefit Absolute risk reduction (ARR) is better ARR = Risk difference = Risk2 – Risk1

Quantifying the Benefit Absolute risk reduction (ARR) is better RR ARR MI: 10% vs. 15% in 10 years .66 5% Death: 50% vs. 75% in 3 years .66 25% Basal Cell CA: 2% vs. 3% in lifetime .66 1%

Q: What does the 34% reduction mean?

Nimotop® Ad Graph 33% 22% Risk1 = 61/278 = 21.8% Risk2 = 92/276 = 33% Nimotop bar is ½ the height of the control group bar. The control group bar looks like it’s 68% of the way to the top, so the implication is that the RRR = 50% and the ARR = 0.5 x 68% = 34%, but of course, it’s the RRR that’s 34%, and the ARR is 0.11 33% 22% Risk1 = 61/278 = 21.8% Risk2 = 92/276 = 33% RR = 22%/33% = .66 ARR = 33% - 22% = 11%

Nimotop® Ad Graph What is 34%? 33% 22% Risk1 = 61/278 = 21.8% Nimotop bar is ½ the height of the control group bar. The control group bar looks like it’s 68% of the way to the top, so the implication is that the RRR = 50% and the ARR = 0.5 x 68% = 34%, but of course, it’s the RRR that’s 34%, and the ARR is 0.11 33% 22% Risk1 = 61/278 = 21.8% Risk2 = 92/276 = 33% RR = 22%/33% = .66 ARR = 33% - 22% = 11% What is 34%?

Nimotop® Ad Graph 33% 22% Risk1 = 61/278 = 21.8% Risk2 = 92/276 = 33% Nimotop bar is ½ the height of the control group bar. The control group bar looks like it’s 68% of the way to the top, so the implication is that the RRR = 50% and the ARR = 0.5 x 68% = 34%, but of course, it’s the RRR that’s 34%, and the ARR is 0.11 33% 22% Risk1 = 61/278 = 21.8% Risk2 = 92/276 = 33% RR = 22%/33% = .66 ARR = 33% - 22% = 11% Relative risk reduction (RRR) = 1 – RR = 1-.66 = .34 or 34%

Quantifying the Benefit RRR is no better than RR RR RRR MI: 10% vs. 15% in 10 years .66 34% Death: 50% vs. 75% in 3 years .66 34% Basal Cell CA: 2% vs. 3% in lifetime .66 34%

Quantifying the Benefit RRR is ALWAYS bigger than ARR (unless untreated risk is 100%)

Quantifying the Benefit BEWARE of risk reduction language!! ARR or RRR? “We reduced risk by 34%” “Risk was 34% lower”

Quantifying the Benefit BEWARE of risk reduction language!! ARR or RRR? “We reduced risk by 34%” can’t tell “Risk was 34% lower” can’t tell Very hard to be unambiguous!

Quantifying the Benefit Another reason that ARR is better: Translate it into “Number Needed to Treat” NNT = 1/ARR

Why is NNT = 1/ARR? 100 SAH patients treated 67 no stroke anyway 33 strokes with no treatment R2 11 strokes prevented R1 22 strokes with treatment 22 strokes with Nimotop®

Why is NNT 1/ARR? Treat 100 SAH patients  prevent 11 strokes Ratio manipulation: 100 treated 1 treated 9.1 treated 11 prevented .11 prevented 1 prevented = =

Why is NNT 1/ARR? Treat 100 SAH patients  prevent 11 strokes Ratio manipulation: 100 treated 1 treated 9.1 treated 11 prevented .11 prevented 1 prevented = = 1/ARR = NNT

Why is NNT 1/ARR? NNT best expressed in a sentence: “Need to treat 9.1 persons with SAH using nimodipine to prevent 1 cerebral infarction”

Quantifying the Benefit NNT calculation practice RR ARR NNT? MI: 10% vs. 15% in 10 years .66 5% Death: 50% vs. 75% in 3 years .66 25% Basal Cell CA: 2% vs. 3% in lifetime .66 1%

Quantifying the Benefit NNT calculation practice RR ARR NNT? MI: 10% vs. 15% in 10 years .66 5% 20 Death: 50% vs. 75% in 3 years .66 25% Basal Cell CA: 2% vs. 3% in lifetime .66 1%

Quantifying the Benefit NNT calculation practice RR ARR NNT? MI: 10% vs. 15% in 10 years .66 5% 20 Death: 50% vs. 75% in 3 years .66 25% 4 Basal Cell CA: 2% vs. 3% in lifetime .66 1%

Quantifying the Benefit NNT calculation practice RR ARR NNT? MI: 10% vs. 15% in 10 years .66 5% 20 Death: 50% vs. 75% in 3 years .66 25% 4 Basal Cell CA: 2% vs. 3% in lifetime .66 1% 100

Quantifying the Benefit NNT expression practice RR ARR NNT? MI: 10% vs. 15% in 10 years .66 5% 20 Death: 50% vs. 75% in 3 years .66 25% 4 Basal Cell CA: 2% vs. 3% in lifetime .66 1% 100 Statins Chemo Sunscreen every day

Quantifying the Benefit NNT expression practice “Need to treat 20 patients with statins for 10 years to prevent 1 MI” “Need to treat 4 patients with chemo for 3 years to prevent 1 death” “Need to treat 100 patients with sunscreen every day for their whole life to prevent 1 basal cell”

Population: hypertensive 60-year-olds Outcome: stroke over 5 years Depiction of Results in Control Group Ref: http://www.nntonline.net/ Ref: http://www.nntonline.net/

Population: hypertensive 60-year-olds Outcome: stroke over 5 years Depiction of Results in Treatment Group Ref: http://www.nntonline.net/

Example 1 Randomized controlled trial of the effects of hip replacement vs. screws on re-operation in elderly patients with displaced hip fractures. Parker MH et al. Bone Joint Surg Br. 84(8):1150-1155.

Example 1 Re-operation No Re-operation Hip Replacement 12 217 229 Internal Fixation with Screws 90 136 226 Parker MH et al. Bone Joint Surg Br. 84(8):1150-1155.

Example 1 Re-operation No Re-operation Risk Hip Replacement 12 217 229 12/229 = 5.2% Internal Fixation with Screws 90 136 226 90/226 = 39.8%

Example 1 Re-operation No Re-operation Risk Hip Replacement 12 217 229 12/229 = 5.2% Internal Fixation with Screws 90 136 226 90/226 = 39.8% RR = R1/R2 = 5.2% / 39.8% = .13 RRR = 1-RR = 1-.13 = 87% ARR = R2 – R1 = 39.8% - 5.2% = 34.6% NNT = 1/ARR = 1/.346 = 3 “Need to treat 3 patients with hip replacement instead of screws to prevent 1 from needing a re-do operation”

Example 2 JUPITER: Randomized controlled trial of high dose rosuvastatin in patients with LDL<130 and CRP>2.0 Ridker et al. NEJM 2008; 359:2195-207

Example 2 Ridker et al. NEJM 2008; 359:2195-207

Example 2 Ridker et al. NEJM 2008; 359:2195-207

Example 2 HR = (R1/R2) (from regression) = .56 RRR = 1-HR = 1-.56 = 44% ARR = R2 – R1 = 1.36 - 0.77 = .59 / 100py* = .0059 / py NNT = 1/ARR = 1/.0059 = 100/.59 = 169 pys “Need to treat 169 patients for a year to prevent 1 CVD event” Or better: “Need to treat 85 patients for 2 years to prevent 1 CVD event” (average treatment duration in trial was 1.9 years) * py = person-years

Example 4 Warfarin vs. placebo for atrial fibrillation Warfarin Placebo Risk of major bleed (/yr) 1.2% 0.7% Ann Intern Med 1999; 131:492-501

Example 4 Warfarin vs. placebo for atrial fibrillation RR = R1/R2 = 1.2% / .7% = 1.7 RR (flipped) = R2/R1 = .7% / 1.2% = .59 RRR (flipped) = 1-RR = 1 - .59 = 41% ARR = R2 – R1 = .7% - 1.2% = -.5% “ARI” – Absolute risk increase = 0.5% NNT = 1/ARR = 1/-.5% = -200 “NNH” – Number needed to harm = -NNT = 1/ARI = 200 “If you treat 200 Afib patients with warfarin, you will cause 1 major bleed”

Circling back to test utility… Tests help determine: If the RR applies Treatment for a disease doesn’t help if you don’t have the disease! Interactions (RR is higher or lower than average) Statins more effective if CRP is high? Patients with gene XYZ more likely to have a side effect Baseline risk The higher the risk, the larger the ARR, the smaller the NNT

Key Concepts Test utility depends on how good the treatment is RR and p-values good for hypothesis testing/statistics ARR and NNT (and NNH) better for interpreting clinical importance ARR = risk difference NNT = 1/NNT Beware RRR and ambiguous language

Where To Start

Where To Start Trials are usually published in a set format: abstract; introduction; methods; results and discussion plus a conclusion. Use a check list when you read a RCT (e.g. CONSORT, CASP). But always start by asking two basic question: Where is the trial published? Is the trial sponsored?

Patient Intervention Comparator Outcome. Critical appraisal starts with a well-formulated question. This typically has four parts and the mnemonic is PICO: Patient Intervention Comparator Outcome.

METHODS Is the precise aim of the study described? Dose the section contains of all steps taken to avoid bias? Randomization Blindness Baseline data etc. What were the inclusion and exclusion criteria of the trial?

METHODS Was the group size and duration of the study sensible? Is the new drug being compared with the gold standard? Are realistic comparative doses being used? Is the variable selected for measurement really related to the answer being sought?

RESULT Statistical tests and analysis (beta error, alpha error, p- value, etc.) Confidence intervals How are the results expressed? (relative or absolute measures; NNT) Are all patients accounted for? (intention to treat versus pre-protocol analysis) Surrogate markers versus hard clinical end point. Statistical significance versus clinical significance

DICUSSION Can the results be applied to the local patient population? What is the authors’ conclusion? What are the main defects of the study?

FINAL CONSIDERATION: “STEP acronym” SAFETY TOLERABILITY EFFICACY PRICE

Judging the quality of research JAMA articles http://www.cche.net/user Checklists e.g. CASP NHS Critical Appraisal Skills Programme http://www.phru.nhs.uk/casp/appraisa.htm A framework for thinking about the quality of research

Intent to Treat Analysis People in a trial still sometimes do what they want instead of what you told them to do Some people assigned to placebo will go to their private doctor and be treated with the experimental drug Some people assigned to the active drug will not take their medicine How do you analyze these people?

Intent to Treat Analysis People should be analyzed in the groups that they were assigned to If assigned to placebo, analyze as placebo If assigned to active drug, analyze as active drug, even if you have evidence that they did not take the drug

Intent to Treat Analysis: Example Coronary Drug Project Men given clofibrate vs. placebo Outcome was 5-year mortality

Intent to Treat Analysis: Example Intent-to-treat analysis of 5 year mortality 20.0% in clofibrate group 20.9% in placebo group On-treatment analysis of the men who took more than 80% of the clofibrate over 5 years 15.0% 5-year mortality in clofibrate compliant

Intent to Treat Analysis An individual assigned to a particular intervention group is included in that group’s outcome statistics even if he/she never receives the intervention Preserves the full value of randomization

Example: Experimental Designs Coronary Primary Prevention trial, 1985 Treatment groups: men with high serum cholesterol randomized to receive either cholestyramine or an equally foul-tasting placebo Outcome was cardiac death after 7-10 years

Grades of Recommendations for a Specified Level of Baseline Risk RCTs, no heterogeneity, CIs all on one side of threshold NNT A1 RCTs, no heterogeneity, CIs overlap threshold NNT A2 RCTs, heterogeneity, CIs all on one side of threshold NNT B1 RCTs, heterogeneity, CIs overalp threshold NNT B2 Observational studies, CIs all on one side of threshold NNT C1 Observational studies, CIs overlap threshold NNT C2