Critical appraisal of the literature Michael Ferenczi Head of Year 4 Head of Molecular Medicine Section, National Heart and Lung Institute

Trends in MEDLINE journal articles, 1978–2001 (n = 8,123,392)

Appraisal: increases the effectiveness of reading by enabling you to exclude research studies that are too poorly designed to inform practice. This frees time to concentrate on a more systematic evaluation of those studies that cross the quality threshold and then to extract their salient points.

Why is this necessary? So much information is now easily available, or forced upon us, that it is not possible to read or study it all. Selection is required Criteria for selection need to be established

Selection criteria: * Systematically evaluating scientific literature. * Sifting the "wheat" from the "chaff" when your literature search harvests conflicting studies * Filtering out original research or meta-analyses which are methodologically sound * Deciding which papers are going to be relevant * Breaking down barriers between research (pure science) and practice (applied science) (e.g. in medical practice) * Supporting the development of Evidence Based Practice (EBP).

Selection criteria: 1. Relevance - the topic of the study, how important it is to you at the moment, the similarity of the setting to the one you work in, the level at which it is written, the professional group or discipline for whom it is written, etc. 2. Intrinsic factors - (i.e. those factors that relate to the study itself) - the appropriateness of the study design to the question being asked, the suitability of the sample, the methods used to recruit the sample, methods used to obtain the results, etc. 3. Extrinsic factors - (i.e those external factors which are assumed (but are not necessarily) to be associated with the quality of the article) - who wrote it, where they work, what their job or qualifications are, whether you have heard of them, who paid for the study, which journal it is written in, whether they have written on the subject before, etc. Are extrinsic factors acceptable, or are they too subjective?

Evidence Based Medicine Working Group A. Are the results of the study valid? [Validity] Are the conclusions justified by the description of the methodology and the findings? Is the methodology sound, have the authors made reasonable assumptions, are there confounding factors they have failed to consider? If they are using a sample, have they selected this to avoid bias? B. What are the results? [Reliability] What are the findings of this article? Is the effect demonstrated large enough to be of significance? How confident are we that the results fall within the bounds of reasonable expectation and are not fluke? C. Will the results help locally? [Applicability] Are the problems I deal with sufficiently like those in the study to extrapolate the findings? Can I generalise from this study to my work place, my experimental conditions, my patients?

Appraisal of reviews 1. Does the review set out to answer a precise question? (i.e. it is something more scientific than an attempt to gather everything written about malaria). 2. Has a thorough search been carried out? Anything missing? - Medline and other relevant bibliographic database - Cochrane controlled clinical trials register - Foreign language literature - "Grey literature" (unpublished or unindexed reports: theses, conference proceedings, internal reports, pharmaceutical industry) - Reference chaining from any articles found - Personal approaches to experts in the field - Hand searches of the relevant specialised journals. - (for meta-analysis, it may be important to track down the raw study data for re- analysis, rather than the final report) 3. Have the authors included explicit inclusion and exclusion criteria for studies, taking account of the patients in the studies, the interventions used, the outcomes recorded and the methodology? 4. Are results applicable locally

Evidence Based Medicine Working Group A. Are the results of the study valid? [Validity] Are the conclusions justified by the description of the methodology and the findings? Is the methodology sound, have the authors made reasonable assumptions, are there confounding factors they have failed to consider? If they are using a sample, have they selected this to avoid bias? B. What are the results? [Reliability] What are the findings of this article? Is the effect demonstrated large enough to be of significance? How confident are we that the results fall within the bounds of reasonable expectation and are not fluke? C. Will the results help locally? [Applicability] Are the problems I deal with sufficiently like those in the study to extrapolate the findings? Can I generalise from this study to my work place, my experimental conditions, my patients?

Evaluating websites Do not believe all what you read – be critical The URL.com,.biz and.co addresses suggest commercial bodies;.org suggests non-profit making sites;.edu and.ac suggest academic institutions;.gov,.doh etc. suggest government departments.

What is a paper? A critical report of a scientific study.

What factors will determine where you publish? Selecting a journal Is the work suitable for a general or specialist audience? –Which journal attract this readership –Impact factor of the journal The peer review process

The Impact Factor More than 6000 journals in Medicine and Life Sciences. Journals are ranked according to the number of citations for each article in the most recent 3 years The Impact factor, often abbreviated IF, is a measure of the citations to science and social science journals. It is frequently used as a proxy for the importance of a journal to its field. It is the average number of citations in a year given to those papers in a journal that were published during the two preceding years

The Impact Factor Abbreviated Journal Title ISSN Total Cites Impact Factor Immediacy Index Articles Cite Half-life

WHO CONTROLS WHAT IS PUBLISHED? The peer review system Editor: selects expert referees Referees: critically appraise the work and report back to the editor Editor: evaluates the referees’ reports and makes a decision based on Quality of the work Importance of the work and potential for citation Priority for publication.

Reading and analysing a scientific paper

What does a paper comprise? Abstract/Summary Introduction Materials and Methods Results Discussion References Acknowledgements

Approaching the paper Title and short title: these will give and indication of the broad area Abstract/summary: this is the overview. Ask yourself - is this of interest to me?

What is it? Background information which leads up to the hypothesis to be tested. Questions to ask Is it a fair reflection of the literature? Is the hypothesis sound? The introduction

Materials and Methods What should it include? A clear description of the design of the study and the methods used. Questions to ask Is the fundamental design appropriate? Are the methods suitable? Could you repeat the study with the information provided?

Results What should this section include? A clear description of the data, drawing attention to points of significance Presentation of the data as figures. Questions to ask Are the data expressed clearly? Are the appropriate control data included? Have the data been tested statistically? Do the data show what the authors claim?

Discussion This section should review the data critically in the light of the current literature limitations posed by the design of the study and the methods used.

Discussion Questions to ask Do the data justify the interpretations? Are there other possible interpretations? Are relevant published findings considered? Are the limitations of the study considered? Is the work a significant advance?

References Do these reflect fairly and appropriately the current state of knowledge? Is important work ignored (e.g. from competitors)? Is my relevant work cited?

Presentation Is the quality of English good enough Is the text clear and succinct Is the data presentation optimal: number and clarity of figures, and tables