1. The material was supported by an educational grant from Ferring How to critically appraise a study Nikolaos P. Polyzos M.D. PhD

2. Learning objectives At the end of this presentation you should appreciate that you should not accept the conclusions of any paper without question. This presentation will provide guidance of how you should question the conclusions of papers.

3. A research finding is less likely to be true when: the studies conducted in a field are smaller effect sizes are smaller there is greater flexibility in designs, definitions, outcomes, and analytical modes there is greater financial and other interest and prejudice Ioannidis JP, Plos Med 2005 Essay: Why Most Published Research Findings Are False - John P.A. Ioannidis

4. Why you should question the results in all papers How you should question the results

5. Top cited articles are not ALWAYS telling the truth Why you should question the results in all papers

6. Initial findings…might prove wrong in the future Highly cited studies (>1000 citations) with efficacy claims –16% were contradicted by subsequent research –16% were found to have initially stronger effects –44% were replicated also with a larger sample size in subsequent research compared with the original highly cited study) –24% had remained largely unchallenged Top cited articles are not always telling the truth Ioannidis JP, JAMA 2005

7. Data fabrication can occur even in highly esteemed journals Why you should question the results in all papers

8. Sudbø case (Lancet) Among 908 patients included in the study 250 had the same birth date Data fabrication can occur even in highly estimated journals

9. Low quality randomized trials may show inflated treatment effect Why you should question the results in all papers

10. Cochrane Pregnancy and Childbirth Database –250 controlled trials from 33 meta-analyses Greater apparent treatment effect in studies of poorer quality –Randomisation issues Odds Ratios exaggerated up to 41% –Inadequate blinding Odds Ratios exaggerated by 17% Low quality randomized trials may show spuriously inflated treatment effect Schulz. JAMA 1995; 273 (5): 408–412

11. META-ANALYSIS PLOT FOR PRETERM BIRTH <37 WEEKS OF GESTATION Study or subgroupTreatmentNo treatmentweightOdds ratio Events/toalEvents/total(%)(M-H, fixed) (95% CI) Low quality trails Subtotal (95% CI)114/996147/72535.20.52 (0.38 to 0.72) Test for heterogeneity: χ 2 = 4.95, df = 5, P = 0.42, I 2 = 0% Test for overall effect: z = 4.01, P<0.001 High quality trials Subtotal (95% CI)250/2303219/229064.81.15 (0.95 to 1.40) Test for heterogeneity: χ 2 = 4.02, df = 4, P = 0.40, I 2 = 1% Test for overall effect: z = 1.45, P = 0.15 Total (95% CI)364/3299366/30151000.93 (0.79 to 1.10) Test for heterogeneity: χ 2 = 25.94, df = 10, P = 0.004, I 2 = 61% Test for overall effect: z = 0.86, P = 0.39 M-H=Mantel-Haenszel fixed effects model 0.1 0.2 0.5 1 2 5 10 Favours treatmentFavours no treatment Odds ratio (M-H, fixed) (95% CI) Low quality trials showed treatment effect that was not confirmed in high quality studies Polyzos NP et al., BMJ 2010

12. Trials with negative results are not published or get published later Why you should question the results in all papers

13. Among studies presented in major scientific meetings, trials with positive results are more likely to get published compared to negative studies 0 20 40 60 80 100 1.0 0.8 0.6 0.4 0.2 0.0 Time to publication (months) Proportion of abstracts published in full-text Outcome in favour of the experimental arm Not-positive Positive Not-positive-censored Positive-censored Trials with negative results are not published or get published later Polyzos NP et al., Hum Reprod 2011

14. Why you should question the results in all papers Meta-analyses often rely on few or inconsistent evidence

15. 61 systematic reviews published during July 2012 in Cochrane –15% of the reviews included 1or 0 trials –Half included fewer than 1,000 patient randomized patients –31 were updated reviews –11 of these 31 updated reviews included the same number of trials and participants as the previous review they sought to bring up to date. Meta-analyses and systematic reviews may often rely on few or inconsistent evidence Humaidan & Polyzos Nat Med 2012

16. Industry supported research may demonstrate greater treatment effects Why you should question the results in all papers

17. Industry supported research may demonstrate greater treatment effects Lexchin J et al., BMJ. 2003.OR 4.05; 95% CI 2.98-5.51 Research funded by drug companies was 4x more likely to have outcomes that favour the sponsor's product than research funded by other sources Study (first author) Odds ratio 0.1 0.2 0.5 1 2 5 10 100 1000 10000 Azimi 12 Cho 14 Clifford 15 Davidson 16 Dieppe 18 Djulbegovic 19 Djulbegovic 20 Friedberg 23 * Friedberg 23┼ Kamal-Bahl 26╪ Kamal-Bahl 26§ Koep 30 Mandelkern 32 Sacristan 36¶ Sacristan 36 ** Thomas 38 Vandenbroucke 39 Yaphe 41

18. 82% of industry sponsored cost-effectiveness analyses show that drugs are cost effective Industry supported research may demonstrate greater treatment effects Valachis et al., J Clin Oncol 2012

19. Cost-effectiveness analyses with industry involvement were associated with lower baseline assumptions of the sensitivity of the Pap test. The “Straw-man” comparator 100 90 80 70 60 50 40 30 20 10 0 0 10 20 30 40 50 60 70 80 90 100 Author affiliation or funding or conflict of interest |with the manufacturer Estimates for CEA’s without authors affiliated, funding or conflict of interest with the manufacturer Esitmates for CEA’s with at least one author affiliated with the manufacturer or with funding or with conflict of interest with the manufacturer 1 - Specificity Sensitivity Industry supported research may demonstrate greater treatment effects Polyzos NP et al., CMAJ 2011

20. Why you should question the results in all papers Industry sponsored trials and even government sponsored trials have been published late in the past

21. Time to publication –Industry (24months) vs. non-industry (20 months) after study completion p<0.001 Ross JS et al., JAMA 2013 Publication rates within 2 years –industry (40%) vs. non-government/non-industry (56%) –RR = 0.73, 95% CI 0.61–0.87; p<0.001 –industry (40%) vs. government trials (47%); p = 0.22 Ross JS et al., Plos Med 2009 Industry sponsored trials may get published later than non sponsored trials

22. NIH trials NOT PUBLISHED after completion – 30 months - half unpublished – 51 months - 1/3 unpublished Even government sponsored trials fail to get published soon 80 60 40 20 0 0 20 40 60 80 100 Time from study completion (months) Percentage of studies published No at risk 635 635 635 635 493 330 220 153 95 54 44 No of unpublished studies Trial completed before 2007 269 264 259 235 221 197 175 Trial completed in 2007-8 366 356 324 282 244 215 176 80 60 40 20 0 0 5 10 15 20 25 30 Time from study completion (months) Percentage of studies published P<.001 Trial completed before 2007 Trial completed in 2007-8 Ross JS et al., BMJ 2011

23. Why you should question the results in all papers How you should question the results

24. Do not upfront reject a study based on its status as clinical trial governance and reporting have improved over time

25. Less likely to even want to read the full article ! Industry connection to a publication reduces its value to physicians Industry sponsorship: negatively influences physicians' perception of methodological quality reduces their willingness to believe and act on trial findings, independently of the trial's quality 1.25 1.00 0.75 0.50 Industry vs. NoneIndustry vs. NIH Funding Odds Ratio 0.68 (95% CI, 0.49-0.94) P =.02 0.52 (95% CI, 0.37-0.71) P<.001 Physician Willingness to Prescribe Drug Do not upfront reject a study based on its status Kesselheim NEJM 2012

26. The Good Clinical Practice Directive (Directive 2005/28/EC of 8 April 2005 of the European Parliament and of the Council) EudraCT vs. 9 (European trials registry) Results must be provided for all interventional clinical trials which commenced in the European Union from 01 May 2004 onwards The reporting and public disclosure includes: –Trials which have terminated early –Trials with positive as well as negative trial results –Trials of products with OR without a marketing authorisation within the community https://eudract.ema.europa.eu/index.html Why you should not upfront reject a study based on its status now

27. For Pharma to introduce a new drug the drug must pass Phase II studies to establish dosing and safety, followed by Phase III studies to confirm efficacy and further demonstrate safety which are externally monitored and must follow high methodological studies Phase II success rates for new development projects have fallen from 28% (2006–2007) to 18% (2008–2009) The combined success rate at Phase III and submission has fallen to ~50% from 2007 to 2010 Has the situation been improved over time? Arrowsmith. Nature Reviews Drug Discovery 2011 Do not upfront reject a study based on its status

28. How you should question the results Use a standard critical appraisal approach when evaluating a scientific article

29. Critically evaluate the purpose of the study What is the rationale for performing the study Is/are the research question(s) clearly defined and if not, should they be? Framework for How to Read and Critique a Research Study American Nurses Association http://www.nursingworld.org Use a standard critical approach when evaluating a scientific article

30. Critically evaluate the design of the study Is the design appropriate for the study? Does the sample fit with the research design and is the size sufficient? How were data collected? Is the analytical approach consistent with the study questions and research design? Use a standard critical approach when evaluating a scientific article Framework for How to Read and Critique a Research Study American Nurses Association http://www.nursingworld.org

31. Examine whether the study has been registered in a public registry Has the trial been registered prior starting the trial? Are the primary endpoints the same with the registered version of the trial? Use a standard critical approach when evaluating a scientific article

32. Always review the existing literature Is the literature review relevant to the study, comprehensive, and include recent research? Does the literature review support the need for the study? Are the findings consistent with existing literature? Use a standard critical approach when evaluating a scientific article Framework for How to Read and Critique a Research Study American Nurses Association http://www.nursingworld.org

33. Critically evaluate the results and conclusions Are the results presented clearly in the text, tables and figures? Are the statistics clearly explained? Are the results explained in relationship to the theoretical framework, research questions? Are the limitations presented and their implications discussed? Are there recommendations for clinical practice Use a standard critical approach when evaluating a scientific article Framework for How to Read and Critique a Research Study American Nurses Association http://www.nursingworld.org

34. No trial is without limitations, results may occur by chance There is even a risk results may be misleading The more robust the study the more likely it is to be true Always review all the literature and look for consistency and inconsistency in what is reported Do not upfront accept or reject the results of a study just based on the journal published, research group or industry involvement Conclusions

35. Thank you