Objectives  Identify the key elements of a good randomised controlled study  To clarify the process of meta analysis and developing a systematic review  To use this to critically evaluate a Cochrane review  Answer a clinically relevant problem using information from a systematic review

RCT – key issues Allocation – method of randomisation (to minimise confounding) Blinding – patient, clinician and study personnel (to minimise observer and recall bias) Follow-up (adequate; complete; differences between those followed up and those lost to follow-up) Intention to treat analysis

Checklist for appraising RCTs Was assignment of patients to treatment randomised? Was randomisation concealed? Were the groups similar at the start of the trial? Was follow-up of patients sufficiently long and complete? Were all patients analysed in the groups to which they were randomised (intention to treat)? Were patients, clinicians and study personnel kept blind to treatment? Were groups treated equally, apart from the experimental therapy?

Overview of a systematic review ASK a clear and focussed question –How easy is this? SEARCH for relevant studies –ALL studies, anywhere, ever? Does systematic have to be exhaustive? APPRAISE the quality and relevance –Structure depends on the type of studies SYNTHESISE –Meta-analysis or narrative synthesis INTERPRET –In relation to the question asked (policy or clinical)

Correspondingly… critical appraisal Did the review ask a clearly focussed question? Are the findings valid? –Did they find all the relevant studies? –Was quality appraised? –Were the study findings combined appropriately? –Were the study results consistent? What are the findings? Will the findings help me with my patient (or policy problem)?

Finding studies… Electronic databases –General Medline Embase –Methodologically specific Cochrane Library (Reviews and CCTR) DARE, HTA database, INAHTA, NHS EED –Disease area specifics Psychlit Cancerlit –“Grey literature” Hand searching Citation searching

Getting the right studies… PUBLICATION BIAS: Systematic exclusion of studies with (usually) negative findings DUPLICATE PUBLICATION BIAS: Multiple publications of positive trials leads to double- counting in reviews CITATION BIAS: Positive studies are cited more frequently LANGUAGE BIAS: English and positive results

TIME LAG BIAS: Negative studies take longer to be published RETRIEVAL BIAS: Method of searching produces a systematic error – high impact journals are more likely to report positive findings and to be listed in databases

Assessing publication (and similar) bias(es) Were the searches comprehensive? Evidence of “missing studies” Look at the FUNNEL PLOT

Study size Effectiveness Treatment betterControl better Biased pooled effect size

Critical appraisal Not going to cover in detail Some explicit method is required –General “systematicity” Transparency Reducing errors and bias –Lots of issues! One or two reviewers? Measure agreement? Scores or not? A tension: –Identify all studies? But… –Include only the best quality?? Methodological quality is important as possible reason for differences between studies

Findings of review Forest Plot of Amenorrhoea following Microwave Endometrial Ablation

Heterogeneity Are the studies all measuring the same thing? If not, then the pooled estimate may not be informative Heterogeneity can due to a variety of reasons

Sources of heterogeneity Chance (i.e. sampling error) Clinical –Different doses or duration of treatment or co-interventions –Different populations (i.e. risk of events) –Different outcome measures Methodological –Blinding –Methods of analysis –Publication bias

Statistical significance and heterogeneity Studies can be tested statistically for heterogeneity to see if results are significantly different P value of 0.05 or lesser indicates statistical significance for heterogeneity If significant for heterogeneity, then need to consider whether results can be pooled

Summary of critical appraisal of a systematic review Was there a clear question? Did they find all the relevant studies? Was the quality of the studies considered? Was the quality so poor that the review is suspect? What are the results? Is there important heterogeneity? What does it mean? What are the implications of the review findings for my clinical or policy question?

Are they good questions?  Appropriate to compare with non opiate analgesics or sedatives?  Outcome measures appropriate?  Which outcomes most relevant (in order)?  Which outcomes easiest to measure?  Any difficulties with some of these outcomes?

Appropriate search strategy? Were the Cochrane neonatal group guidelines followed in relation to  Blinding of randomisation  Blinding of intervention  Completeness of follow up  Blinding of outcome measurement – see handout

Appropriate search strategy?  What is a quasi randomised study?  What is Cochrane’s central register of controlled studies?  Were non English studies included?  Was there risk of publication bias? How to overcome this.  Was more than 1 reviewer involved? How were differences resolved?

Data Analysis/ Results  Look at meta-analysis of PIPP scores (figure 01.01) Do you think the all studies result shows a satisfactory forest plot?  What is your interpretation of the results of the other methods of assessing pain (figure 01.04)  What is the relevance of a reduction of 2 in the PIPP? (see handout of PIPP)  What are the differences in execution and reporting of trials which authors say will interfere with applying this?

Data Analysis/ Results  What do they mean that heterogeneity was high in all analyses of pains?  What are sources of heterogeneity for PIPP?  What are the other sources of heterogeneity?  Do you think there is sufficient evidence to recommend morphine over midazolam for sedation?

Application  How important are the results?  How applicable are the results?  What do you do?