Critical appraisal Systematic Review กิตติพันธุ์ ฤกษ์เกษม ภาควิชาศัลยศาสตร์ มหาวิทยาลัยเชียงใหม่

Systematic review vs Meta-analysis Systematic review: a systematic approach to minimising bias and error Meta-analysis: a statistical analysis, which aim to produce a single estimate of a treatment effect Systematic review may or may not include Meta-analysis

Why do we need systematic review? The early 1980 s uncomplicated MI “Should pt receive a beta-blocker for secondary prevention before discharge?” Library:4 randomised controlled trials (RCT)

Beta-blocker vs placebo RCT 1.Mortality and hospital readmission is not different RCT 2. Not conclusive RCT 3. Beta-blocker not shown benefit RCT 4. Long term beta-blocker reduces the mortality and rate of re-infarction

A review in BMJ 1981 There is no clear evidence that beta- blocker improves long tem survival after MI despite almost 20 yrs of clinical trials Good enough!!

Another review in European Heart Journal 1981 “it seems perfectly reasonable to treat patients who have survived an infarction with beta-blocker”

Limitation of a single study Too small sample size false negative

Problem of Conventional review Prone to bias and error –Select only evidence support the author’s view –Not specify methodological quality of studies –Finally choose most vote ignore sample size, and design

Meta-analysis = combining all available data Attractive alternative to such large, expensive and problematic study Weight average of the result large > small trial

Meta-analysis Beta-blocker trials - MI Beta-blocker betterPlacebo better

Cumulative meta-analysis of beta-blocker trials

Cumulative meta-analysis Significant effect from 1980 onwards (OR not across 1

Meta-analysis Beta-blocker trials - MI Beta-blocker betterPlacebo better Maybe Unnecessary trials

Benefit Estimate the overall effect Examine different result between studies (heterogeneity) Identified insufficient data

Cochrane collaboration International organisation of health care profession Promoting accessibility of systematic review Foster development of systematic review 50 collaborative review groups

Potentials of systematic review Good Bad

Systematic review Basic structure and types กิตติพันธุ์ ฤกษ์เกษม ภาควิชาศัลยศาสตร์ มหาวิทยาลัยเชียงใหม่

1. Basic structure

Like primary research Why- Introduction, background How-method What we found-result What it mean-discussion

Basic structure Abstract Introduction –Background –Objectives Method “treat a paper like a patient in 1  reseach” –Type of studies –Inclusion criteria type of participants –Exclusion criteria –Type of intervention –Type of outcome measures –Search strategy for identification of study –Method of analysis

Basic structure Result (special diagram) Conclusion Reference

Black square = OR, horizontal line = 95% confidence interval Area of black square = weight, diamond = combined OR with 95% CI LA better GA better Forest plot Stroke rate ‘LA vs GA in carotid sx trial’

2. types

Types Systematic review of primary research –Observational studies –Diagnostic screening –RCT

3. Method “treat a paper like a patient in 1  reseach”

The process (1) Research question Writing protocol Searching Article retrieval Literature review

The process (2) Inclusion/ exclusion criteria Validity and quality of articles Data extraction/ synthesis Interpretation

The question Is local anesthesia is better than general anesthesia during carotid endarterectomy?

Writing the protocol Background Objectives Type of studies Inclusion criteria Type of participants Exclusion criteria Type of intervention Type of outcome measures Search strategy for identification of study Method of analysis Reference

Searching Medline Other database Hand searching the literature Writing to people

Getting the article Which ones to get? It takes time Libraries Inter-library Loans

Literature review You don’t have to read the whole paper yet! Translation

Validity and quality of articles Do read the paper and see what the author thought was wrong Unequal intervention/control size Hidden loss to follow up

Data extraction Read method carefully Design a form

Synthesis/ Interpretation Estimates and confidence intervals pool effect make by statistic method e.g. Peto method (fix method) give more weight effect for large study than small study (P value) Difference between studies (Heterogeneity) Chi-squared test (P value)

Small RCTs show LA is marginal lower mortality than GA

Critical Appraisal 1. Are the result valid? 2. What are the results?

1. Are the result valid? Did this review address a sensible clinical question Was the search for relavant studies detailed and exhaustive? Were selection and assessment of studies reproducible? Were the primary studies of high methodological quality?

Publication bias “A (significant) beneficial treatment effect are published, but an equal result remain unpublished” In general medical journal and public heath journal reported statistically significant 85.4% In psychological journal 95.6%

Time lag bias “Positive result will dominate the literature for several year until the negative will report later” HIV trial in USA, median time to publish of positive result 4.2 years, but negative result 6.4 years

Duplicate publication bias “ one study presents and reports several times” “ include this lead to overestimation of treatment effect” Ondersetron to prevent postoperative nausea vomitting 16 studies 3 duplicated papers Sometimes difficult to say, since not share single common authors !!!!!!

Language bias “Authors tend to report positive result in international papers, English language journal, but if negative result are published in local journal”

Outcome reporting bias In trials many outcome is recorded but only favorable finding will be reported Clinical trials by drug companies, unpublished trials gave information on adverse effect > published trials

Selection Bias Tend to happen in non Randomised controlled trials (non RCT) For example select low risk group to new treatment group

2. What are the result? Were the results similar from study to study? If yes, the credit of single estimates is OK. –Point estimates similar? –Overlapping confidence interval –Test for heterogeneity? (Chi square test) –Percentage of variability (I 2 ) good 50%

Look overlapping confidence interval Rerkasem Cochrane Database Syst Rev 2008; (4):CD

Small RCTs show LA is marginal lower mortality than GA Rerkasem Cochrane Database Syst Rev 2008 (4):CD

2. What are the result? Were the results similar from study to study? What are the overall results of the reviews? How precise were the results?