Critical Appraisal of Systematic Reviews Douglas Newberry

Systematic Reviews — or How to make a Monkey out of EBM without hardly trying!

Systematic Reviews: Objectives: Appraise a systematic review for validity Discuss Meta Analysis / use Odds Ratios Obtain Number Needed to Treat (NNT) from Odds Ratios Consider clinical implications of a Systematic Review {including when to bin it instead!}

We can see further than our forbearers because we stand on the shoulders of Giants {and have better spectacles} these ideas are cribbed unashamedly from friends, books & previous courses

Systematic Reviews: What are your Objectives : What do you want to cover? Please interject with helpful questions!

Did I really want a systematic review? (but please do not pretend) admit your ignorance — expert review or consensus guidelines > broad introduction, cover many areas (class C evidence). if the question is important > formulate it! Systematic review > narrow but rigorous focus.

Systematic Reviews — Where do I start: Start with your 4 (or 3) part clinical question! Is a systematic review a sensible approach? Does THIS systematic review address MY question? Is it a systematic review at all?

Is it a systematic review? does it: define a four part (answerable) clinical question? combine Randomized Controlled Trials (RCT’s)? describe PRE-DEFINED search methods? PRE-DEFINED inclusion criteria? PRE-DEFINED methodological exclusion criteria?

Sceptical View? Take it with a grain of salt: transparent declaration of funding of work? Drug Company sponsorship of Reviews vs. Methodological quality>Cochrane review! who employs the authors? open discussion of existing controversy & commercial gain? Don’t waste salt on your food, keep it for your reading!

Meta analysis — combine what with what? Low Molecular Weight Heparin (LMWH) in hip surgery — begin before or after the operation? meta analysis of placebo controlled RCT’s of heparin in hip surgery >> pre-op & post-op LMWH vs. placebo post-op LMWH Vs placebo pre-operative >> less intra-op bleeding??

Can we believe it ? bias free search & inclusion criteria? appraisal of methodology of primary studies? consistent results from all primary studies? –if not, are the differences sensibly explained? are the conclusions supported by the data?

If we believe it — does it apply to our patient? Is our patient (or population) so different from those in the primary studies that the results may not apply? consider differences in: –time — many things change. –culture — both treatments and values of outcomes can be different –stage of illness or prevalence can effect results.

We believe it ! but —>> does it matter? Is the benefit worthwhile to our patient? Ask the patient about cultural values. Think about Relative Risk Reduction vs. Absolute Risk to our patient. Potential benefit is the Absolute risk avoided in our patient = Absolute Risk Reduction (ARR)!

Absolute Risk—> The risk our patient is facing! How likely is our patient to die (or have the outcome of interest) without intervention? = Control Event Rate (CER) consider this individual patient’s risk factors to estimate Patient Expected Event Rate = PEER. Absolute Risk usually increases with age. Improvement measured as Absolute Risk Reduction (ARR)

Relative Risk Reduction: Usually reported in studies. Ratio of the improvement of outcome over outcome without intervention (Rx): {Control Event Rate (CER) — Experimental Event Rate (EER)} / CER i.e. {CER-EER}/CER often independent of prevalence! often similar at different ages!

Our patient wants an absolute Risk Reduction (ARR): is a 40% reduction in Cardiac Risk worth taking pills daily for 10 years?? >vote! if I have a 30% risk of MI or death {30 out of 100 people like me will suffer MI or death} in next 10 years > 40% RRR >> only 18 out of 100 will have MI or death. ARR = 12 out of 100! >> I like that! BUT if I have a 1% risk in 10 years, 40% less is a 0.6% risk >> hardly different!

Number Needed to Treat (NNT) (very trendy but tricky): only defined for specific prevalence- Patient’s Expected Event Rate=PEER! only defined for a specific intervention! only defined for a specific outcome! – eg. Pravastatin™ 40 mg nocte x10 years, in a 65 year old male, ex-smoker with high BP and Diabetes, to reduce MI or Death. NNT is the inverse of Absolute Risk Reduction: i.e. NNT = 1/ARR

Number Needed to Treat (NNT) for previous example: 12 fewer MI or death in 10 years per 100 persons treated: ARR=12/100 NNT = 1/(12/100)=100/12= 8.3 But the same Relative Risk Reduction (RRR) of 40% with a low prevalence: 0.4 fewer MI/death per 100 treated, ARR=0.4/100. NNT = 1/(0.4/100) = 100/0.4 = 250!

Why Odds Ratios? > compare results of different studies. consider 2x2 table: RRR is (a-b/a) — but you can only go in rows within same study! Odds ratio is (a/c)/(b/d) = ad / bc — the individual ratios are in columns, and therefore are independent of the prevalence which is different in different studies. must use odds ratios to combine RCT’s

Odds Ratio (OR) to NNT — is the improvement worth the trouble? 1>OR>0, lower the OR = better the treatment (Rx) >> lower NNT. for any OR, NNT is lowest when PEER=0.5 estimate the PEER (patient’s risk) apply the OR to get patient's NNT.

Convert PEER & OR to NNT:

Formula used in the table:

Table induced nausea! lower OR >> lower NNT Patient needs to be at risk (non-trivial PEER) in order for risk reduction to be worth the effort. for any OR, NNT lowest when PEER=0.5 more effective treatment > lower NNT BUT are your patient’s values satisfied by the intervention and its sequelae?

Subgroup analysis: Sceptical unless: the subgroups make biological and clinical sense? the differences are both clinically & statistically significant? was a-priori hypothesis (before this data)? other evidence supports these sub-groups? few (OK) or many (nix) sub-group analyses?

Any Questions?

Summary 1: Set your goals. define your 4 (or 3) part question. do you want a true systematic review? does this narrow review address my question? PRE-DEFINED search, inclusion, exclusion!

Summary 2: Be Sceptical! look for bias, conflict of interest. critical appraisal of primary studies? consistent results? if not, why not? does our patient fit the groups studied? does it matter to our patient?

Summary 3: Risks that matter. Absolute risk > estimate the Patient Expected Event Rate (PEER) obtain Relative Risk Reduction (RRR) or Odds Ratio (OR) from a Meta-analysis plug into a table to estimate Number Needed to Treat (NNT)

Summary 4: Sceptical & common sense! beware of post-hoc sub-group analysis, especially if multiple. step back and consider if the systematic review really related to our patient’s situation (PEER), culture and expectations? do not loose sight of common sense!

Coffee Now! Small Groups Afterwards