R. Heshmat MD; PhD candidate Systematic Review An Introduction

R. Heshmat MD; PhD candidate Review The general term for all attempts to synthesize the results and conclusions of two or more publications on a given topic. A review may or may not be systematic. The general term for all attempts to synthesize the results and conclusions of two or more publications on a given topic. A review may or may not be systematic.

R. Heshmat MD; PhD candidate Overview=Systematic Review A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review. A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.

R. Heshmat MD; PhD candidate Meta-analysis When an systematic review incorporates a specific statistical strategy for assembling the results of several studies into a single estimate. When an systematic review incorporates a specific statistical strategy for assembling the results of several studies into a single estimate.

R. Heshmat MD; PhD candidate A systematic review may, or may not, include meta-analysis. A systematic review may, or may not, include meta-analysis. Systematic review is always appropriate and desirable, but it may sometime inappropriate to statistically pool results from separate studies. Systematic review is always appropriate and desirable, but it may sometime inappropriate to statistically pool results from separate studies.

R. Heshmat MD; PhD candidate A single study may produce a false negative results, due to inadequate sample size. A single study may produce a false negative results, due to inadequate sample size. In the meta-analysis the necessary number of participants can be reached, and small effects can be detected or excluded with confidence. In the meta-analysis the necessary number of participants can be reached, and small effects can be detected or excluded with confidence.

R. Heshmat MD; PhD candidate Systematic Review: Epidemiology of results Where the findings of an original study replace the individual as the unit of analysis.

R. Heshmat MD; PhD candidate Narrative reviews The classical review is subjective and therefore prone to bias and error. The classical review is subjective and therefore prone to bias and error. Selective inclusion of studies that support the author’s view is common. Selective inclusion of studies that support the author’s view is common. It ignores sample size, effect size, and research design. It ignores sample size, effect size, and research design. Systematic reviews allow a more objectives appraisal. Systematic reviews allow a more objectives appraisal.

R. Heshmat MD; PhD candidate Historical notes The statistical basis of meta-analysis reaches back to the 17 th century in astronomy and geodesy. The statistical basis of meta-analysis reaches back to the 17 th century in astronomy and geodesy. In 1976 the psychologist Glass coined the term “meta-analysis”. In 1976 the psychologist Glass coined the term “meta-analysis”. In the 1980s meta-analysis became increasingly popular in medicine. In the 1980s meta-analysis became increasingly popular in medicine.

R. Heshmat MD; PhD candidate Finally: Cochrane Collaboration was held in Oxford Cochrane Collaboration was held in Oxford in October in October The Collaboration aims to help people make well-informed decision about health care by preparing, maintaining and promoting the accessibility of systematic review. The Collaboration aims to help people make well-informed decision about health care by preparing, maintaining and promoting the accessibility of systematic review.

R. Heshmat MD; PhD candidate Principles of and procedures for systematic review Systematic reviews should be as carefully planned as any other research project, with a detailed written protocol in advance.

R. Heshmat MD; PhD candidate Stage 1 Planning the review Stage 2 Conducting a review Stage 3 Reporting & dissemination

R. Heshmat MD; PhD candidate Stage 1 Planning the review Phase 0 Identification of the need for a review Phase 1 Preparation of a proposal for a review Phase 2 Development of a review protocol

R. Heshmat MD; PhD candidate Stage 1 Planning the review Phase 0 Identification of the need for a review Phase 1 Preparation of a proposal for a review Phase 2 Development of a review protocol

R. Heshmat MD; PhD candidate Stage 1 Planning the review Phase 0 Identification of the need for a review Phase 1 Preparation of a proposal for a review Phase 2 Development of a review protocol

R. Heshmat MD; PhD candidate Stage 1 Planning the review Phase 0 Identification of the need for a review Phase 1 Preparation of a proposal for a review Phase 2 Development of a review protocol

R. Heshmat MD; PhD candidate Stage 2 Conducting a review Phase 3 Identification of research Phase 4 Selection of studies Phase 5 Study quality assessment Phase 6 Data extraction and monitoring progress Phase 7 Data synthesis

R. Heshmat MD; PhD candidate Stage 2 Conducting a review Phase 3 Identification of research Phase 4 Selection of studies Phase 5 Study quality assessment Phase 6 Data extraction and monitoring progress Phase 7 Data synthesis

R. Heshmat MD; PhD candidate Stage 2 Conducting a review Phase 3 Identification of research Phase 4 Selection of studies Phase 5 Study quality assessment Phase 6 Data extraction and monitoring progress Phase 7 Data synthesis

R. Heshmat MD; PhD candidate Stage 2 Conducting a review Phase 3 Identification of research Phase 4 Selection of studies Phase 5 Study quality assessment Phase 6 Data extraction and monitoring progress Phase 7 Data synthesis

R. Heshmat MD; PhD candidate Stage 2 Conducting a review Phase 3 Identification of research Phase 4 Selection of studies Phase 5 Study quality assessment Phase 6 Data extraction and monitoring progress Phase 7 Data synthesis

R. Heshmat MD; PhD candidate Stage 2 Conducting a review Phase 3 Identification of research Phase 4 Selection of studies Phase 5 Study quality assessment Phase 6 Data extraction and monitoring progress Phase 7 Data synthesis

R. Heshmat MD; PhD candidate Stage 3 Reporting & dissemination Phase 8 The report and recommendations Phase 9 Getting evidence into practice

R. Heshmat MD; PhD candidate Stage 3 Reporting & dissemination Phase 8 The report and recommendations Phase 9 Getting evidence into practice

R. Heshmat MD; PhD candidate Stage 3 Reporting & dissemination Phase 8 The report and recommendations Phase 9 Getting evidence into practice

R. Heshmat MD; PhD candidate Developing a Protocol for a Systematic Review

R. Heshmat MD; PhD candidate Role of the protocol A written document that forms the ’plan’ for the review A protocol helps to avoid or minimise bias Bias may occur in the retrieval, selection, extraction of data and evaluation of results A protocol can (and should be) sent for external peer review

R. Heshmat MD; PhD candidate Components of a protocol Background Background Review questions/ Objectives Review questions/ Objectives Search strategy Search strategy Study selection criteria and procedures Study selection criteria and procedures Study quality assessment checklists and procedures Study quality assessment checklists and procedures

R. Heshmat MD; PhD candidate Components of a protocol Data extraction strategy Data extraction strategy Data synthesis strategy Data synthesis strategy Project timetable Project timetable

R. Heshmat MD; PhD candidate Background section Patients / disease characteristics Patients / disease characteristics Course of disease Course of disease Pathophysiology Pathophysiology Interventions Interventions

R. Heshmat MD; PhD candidate Research questions/ objectives Population/ participants Population/ participants Interventions Interventions Outcomes Outcomes Study designs Study designs

R. Heshmat MD; PhD candidate Literature search Search strategy should specify which databases and other sources will be searched Search strategy should specify which databases and other sources will be searched Based on components of review questions Based on components of review questions Do not need to present detailed search strategies in protocol Do not need to present detailed search strategies in protocol

R. Heshmat MD; PhD candidate Possible sources of literature Electronic databases Electronic databases Medline, Embase, PsycLIT, CINAHL Medline, Embase, PsycLIT, CINAHL specialist trial registers e.g. CCTR specialist trial registers e.g. CCTR Handsearching Handsearching Checking reference lists Checking reference lists Personal communication Personal communication Pharmaceutical companies Pharmaceutical companies Grey literature Grey literature

R. Heshmat MD; PhD candidate Study selection criteria Should follow from research questions Should follow from research questions i.e. population, interventions, outcomes, study design i.e. population, interventions, outcomes, study design Inclusion and exclusion criteria Inclusion and exclusion criteria Details of selection process should be set out here (how many reviewers, how are disagreements resolved?) Details of selection process should be set out here (how many reviewers, how are disagreements resolved?)

R. Heshmat MD; PhD candidate Study quality assessment Purpose of quality assessment? Purpose of quality assessment? For selection? For data synthesis? For implications of results? For selection? For data synthesis? For implications of results? Choose appropriate checklist (related to study design) Choose appropriate checklist (related to study design) Details of assessment process (how many reviewers, disagreements etc) Details of assessment process (how many reviewers, disagreements etc)

R. Heshmat MD; PhD candidate Data extraction strategy Think about what data you need to extract from included studies to answer the questions Think about what data you need to extract from included studies to answer the questions Pilot a draft data extraction form Pilot a draft data extraction form Agree process (how many reviewers, disagreements etc) Agree process (how many reviewers, disagreements etc) Any manipulation of study data to be reported here Any manipulation of study data to be reported here

R. Heshmat MD; PhD candidate Data synthesis Will results be pooled? How? Will results be pooled? How? How will differences between studies be taken into account? How will differences between studies be taken into account? Proposed sensitivity analyses? Subgroups? Proposed sensitivity analyses? Subgroups? How will results be displayed? How will results be displayed? May not be possible to be specific at protocol stage May not be possible to be specific at protocol stage

R. Heshmat MD; PhD candidate Dissemination How will you publish it? In what format? How will you publish it? In what format?

R. Heshmat MD; PhD candidate Timescale Set out key milestones (e.g. searching, study selection, data extraction, draft report to peer review, final report) Set out key milestones (e.g. searching, study selection, data extraction, draft report to peer review, final report) Some stages may overlap Some stages may overlap An end date for the review is the most important! An end date for the review is the most important!

R. Heshmat MD; PhD candidate Peer review Best to find out about big problems before you start the review! Best to find out about big problems before you start the review! A panel of topic experts (including consumers) will be able to comment on the relevance of your research questions A panel of topic experts (including consumers) will be able to comment on the relevance of your research questions A panel of methodological experts will be able to comment on whether your proposed methods will answer the proposed review questions A panel of methodological experts will be able to comment on whether your proposed methods will answer the proposed review questions

R. Heshmat MD; PhD candidate Methods of peer review Circulation of draft protocol to expert panel and receipt of comments Circulation of draft protocol to expert panel and receipt of comments Expert panel meeting Expert panel meeting Commissioners’ meeting Commissioners’ meeting Publish draft protocol on web site Publish draft protocol on web site Submit draft protocol to relevant Cochrane Review Group Submit draft protocol to relevant Cochrane Review Group

R. Heshmat MD; PhD candidate Protocol modifications May need to change protocol at study selection stage (e.g. if there are no studies which meet inclusion criteria) May need to change protocol at study selection stage (e.g. if there are no studies which meet inclusion criteria) Other research questions may arise Other research questions may arise Modifications to the protocol should be documented and justified Modifications to the protocol should be documented and justified

R. Heshmat MD; PhD candidate Limitations of Systematic Review Reporting bias and the inadequate quality of primary research are potentially serious problems for systematic reviews. Reporting bias and the inadequate quality of primary research are potentially serious problems for systematic reviews. The quality of component studies is of crucial importance. The quality of component studies is of crucial importance. The dissemination of research findings is not a random process; rather it is strongly influenced by the nature and direction of results. The dissemination of research findings is not a random process; rather it is strongly influenced by the nature and direction of results.

R. Heshmat MD; PhD candidate Type of reporting bias Publication bias Publication bias The publication or non-publication of research findings, depending on the nature and direction of the results The publication or non-publication of research findings, depending on the nature and direction of the results

R. Heshmat MD; PhD candidate Time lag bias Time lag bias The rapid or delayed publication of research finding, depending on the nature and direction of the results The rapid or delayed publication of research finding, depending on the nature and direction of the results

R. Heshmat MD; PhD candidate Multiple (duplicate) publication bias Multiple (duplicate) publication bias The multiple or singular publication of research finding, depending on the nature and direction of the results The multiple or singular publication of research finding, depending on the nature and direction of the results

R. Heshmat MD; PhD candidate Citation bias Citation bias The citation or non- citation of research The citation or non- citation of research finding, depending on the nature and direction of the results finding, depending on the nature and direction of the results

R. Heshmat MD; PhD candidate Language bias Language bias The publication of research finding in a particular language, depending on the nature and direction of the results The publication of research finding in a particular language, depending on the nature and direction of the results

R. Heshmat MD; PhD candidate Outcome reporting bias Outcome reporting bias The selective reporting outcomes but not of others, depending on the nature and direction of the results The selective reporting outcomes but not of others, depending on the nature and direction of the results

R. Heshmat MD; PhD candidate The inclusion of data from unpublished studies can itself introduce bias. The inclusion of data from unpublished studies can itself introduce bias. Unpublished studies may be of lower methodological quality than published studies. Unpublished studies may be of lower methodological quality than published studies.

R. Heshmat MD; PhD candidate Biased inclusion criteria If, as is usual, the inclusion criteria are developed by an investigator familiar with the area under study, they can be influenced by knowledge of the results of the set of potential studies. If, as is usual, the inclusion criteria are developed by an investigator familiar with the area under study, they can be influenced by knowledge of the results of the set of potential studies. Manipulating the inclusion criteria could lead to selective inclusion of positive studies and exclusion of negative studies. Manipulating the inclusion criteria could lead to selective inclusion of positive studies and exclusion of negative studies.

R. Heshmat MD; PhD candidate Why do we search widely? Savoie et al estimated that 29.2% of items in their review were uncovered by: Savoie et al estimated that 29.2% of items in their review were uncovered by: searching the web searching the web handsearching handsearching scanning reference lists scanning reference lists personal communication personal communication searching specialised databases and web sites. searching specialised databases and web sites.

R. Heshmat MD; PhD candidate Why search widely…. Allen & Hanburys found that: Allen & Hanburys found that: only 51% of the clinical trials relating to their respiratory products were published in journals indexed by MEDLINE, EMBASE or CINAHL (46% are in grey lit) only 51% of the clinical trials relating to their respiratory products were published in journals indexed by MEDLINE, EMBASE or CINAHL (46% are in grey lit) Wallace et al. Wallace et al. 11 of 65 trials in end stage renal disease reviews were found by searching beyond major databases. 11 of 65 trials in end stage renal disease reviews were found by searching beyond major databases.

R. Heshmat MD; PhD candidate Why search widely.... Long lead times before publication: Long lead times before publication: publication gaps after conference presentation publication gaps after conference presentation indexing publication lag before recorded in databases indexing publication lag before recorded in databases Cheng et al, 1998 Cheng et al, % of a set of conference papers achieved publication within 12 months 8.1% of a set of conference papers achieved publication within 12 months 29% within 2 years and 40% within 5 years 29% within 2 years and 40% within 5 years

R. Heshmat MD; PhD candidate Delay and non publication Non-publication Non-publication Cheng et al, 1998 Cheng et al, 1998 only 32% of abstracts presented at chronic fatigue conferences were subsequently published in full only 32% of abstracts presented at chronic fatigue conferences were subsequently published in full Petticrew et al, 1999 Petticrew et al, % of oral paper presented at the Society for Social Medicine 1996 achieved publication 50.6 % of oral paper presented at the Society for Social Medicine 1996 achieved publication