Workshop on VHL and HEN, Sao Paulo, April 2006 HEN Methodology Step by step

Workshop on VHL and HEN, Sao Paulo, April 2006 Reviewing evidence Systematic reviews are increasingly common Cochrane, Campbell collaboration, DARE, HTA… These are often done by experts Can be slow (6 months plus) Expensive: $130,000 average Probably needs 2+ people per review We don’t have time! We don’t have the resources!

Workshop on VHL and HEN, Sao Paulo, April 2006 Types of methods to produce evidence

Workshop on VHL and HEN, Sao Paulo, April 2006 The starting point for HEN is the policy makers’ question Policy makers’ question Synthesis of evidence 10 page reports & 1 page summary

Final Version POLICY QUESTION HEN – TUs Refine questions Identify authors HEN Possible reject Drafting Report Initial Review Author External / Internal Reviewers TUs Peer Review Draft revision Possible reject HEN Quality Control Possible reject Author Final Synthesis HEN Communication INFORMATION NEED Identify sources Description Mapping Documents/DB Selection Edit Resources HEN Consultant HEN Freelance editor HEN Member States’ Evidence Needs Synthesis Update Authors Re-phrase question Immediate answers Resources HEN Copy-editor

Workshop on VHL and HEN, Sao Paulo, April 2006 Collection of questions A) Proactive Call for topics once a year - Ministries of health (using WHO channels) - Technical Units of WHO - Network Members Systematically review the work of HEN Members B) Reactive HEN electronic mailbox Phone questions from policy makers Questions or policy concerns identified by WHO Technical Units Issues raised at regional committees and ministerial conferences

Workshop on VHL and HEN, Sao Paulo, April 2006 Prioritization of the policy concerns Highest priority: questions from Ministries of Health Priority areas of WHO Europe –health systems –mental health –child health –environment –HIV/AIDS –nutrition –non-communicable diseases –ageing –poverty

Workshop on VHL and HEN, Sao Paulo, April 2006 Selection criteria Availability of evidence (after preliminary literature searches, and discussions with technical staff) Feasibility (is it practically possible to produce a synthesis report within a reasonable time and budget frame?) Relevance for audience Coverage (whether the proposed public health questions are of interest to a number of Member States or only one) Timeliness (how long it will take to produce the answer)

Workshop on VHL and HEN, Sao Paulo, April 2006 Selection of authors Proven ability to undertake a systematic review (participation in systematic reviews conducted by the Cochrane Collaboration, HTA agencies) Proven international record of publication in the field of public health (international scientific papers, indexed in Medline/PubMed or another scientific bibliographical database) Proven record of communication with policy makers (indicated by the topics of the experts publications, conference presentations or teaching areas) (Medline, CV) Availability and possibility to produce a paper in a given time table

Workshop on VHL and HEN, Sao Paulo, April 2006 Standard structure for synthesis reports The report: Introduction Findings from health related research Other knowledge Current debate Discussion Conclusions References The summary: Issue Findings Policy considerations Type of evidence

Workshop on VHL and HEN, Sao Paulo, April 2006 Quality Control 1) Author has conducted a proper and systematic search and is transparent with the search strategy 2) Inclusion/exclusion criteria and the methods of analysis are described properly 3) List of references, adding those relevant to the paper (and to our Member States) 4) Findings/results reported critically but objectively 5) Style suitable for policy makers (not too clinical or technical) 6) Paper and summary follow the standard format 7) Policy considerations are clear, based on findings and provide concrete support for decision making or action

Workshop on VHL and HEN, Sao Paulo, April 2006 Grading of evidence and strength of policy options Strong evidence – consistent findings in two or several scientific studies of high quality Moderate evidence – consistent findings in two or several scientific studies of acceptable quality Limited evidence – only one study available or inconsistent findings in several studies No evidence – no study of acceptable scientific quality available

Workshop on VHL and HEN, Sao Paulo, April 2006 Methodological appropriateness different types of questions best answered by different types of study Muir Gray, 1997 (on social interventions in children)

Workshop on VHL and HEN, Sao Paulo, April 2006 (Randomised) Controlled Trials Was the assignment to the treatment groups random? Was relatively complete follow-up achieved? Were the outcomes of people who withdrew described and included in the analysis? Were the control and intervention groups similar at the start of the study? Were the groups treated identically (other than the intervention/s of interest? How big is the study? How big is the effect? Do the numbers add up?

Workshop on VHL and HEN, Sao Paulo, April 2006 Cohort Studies (that follow cohorts of people over time) Is the sample representative? What else happened? (What factors other than the intervention may have affected the outcome, and were the cohorts being compared comparable on these important confounding factors? Are the outcomes meaningful? Was there adequate statistical adjustment, or matching for the effects of these confounding variables? How big is it? How big is the effect? Do the numbers add up?

Workshop on VHL and HEN, Sao Paulo, April 2006 Surveys (cross-sectional - one point in time) Is the study based on a representative (random) sample? Was follow-up long enough for important events to occur? Are the measures meaningful? Do we know how people got into the survey? Do we know how many were surveyed and how many refused? How big is the study? (Big surveys are not necessarily better) Who dropped out? Surveys: gold mines for data-dredgers

Workshop on VHL and HEN, Sao Paulo, April 2006 Appraisal of qualitative studies How credible are the findings? How clear is the basis for evaluation? How defensible is the research design? How well was the data collection carried out? Hoe well has diversity of perspective and content been explored? How adequately has the research process been documented? Etc.

Workshop on VHL and HEN, Sao Paulo, April 2006 Assessing the validity of literature reviews Look for sources of bias. The main ones are: A poorly defined question A limited search for literature Poorly defined inclusion/exclusion criteria Lack of assessment of the validity of the included studies Lack of investigation of heterogeneity Inappropriate pooling of studies

Workshop on VHL and HEN, Sao Paulo, April 2006 The “systematic-ish” review Looks a bit systematic... Includes details of some databases that were searched... Uses the right jargon... Doesn’t appraise the included studies... The conclusions are not consistent with the results of the studies...

Workshop on VHL and HEN, Sao Paulo, April 2006 Reading other people’s reviews Are they answering the right question? Are they using the right evidence to answer that question? Is it likely that they missed relevant evidence? Do you suspect that they are using evidence selectively? Have they paid attention to the quality of that evidence? E.g. Its methodological soundness, and relevance Who pays them? Who paid for the review? Are these, or other sources of funding likely to introduce bias?

Workshop on VHL and HEN, Sao Paulo, April 2006 How to be more evidence-based: Presentation is everything Be clear about what questions you are answering Be clear about what sort of evidence you believe is admissible, and why Be clear about what you have excluded/included Reference well-conducted research, in preference to opinion pieces, editorials, general reviews, general WHO or World Bank commentaries Reference relevantly Show them 1. methods and 2. the evidence

Workshop on VHL and HEN, Sao Paulo, April 2006 Always check the evidence - whatever the source “Those who can make you believe absurdities...can make you commit atrocities” Voltaire