Addressing missing participant data in systematic reviews: Part I – Dichotomous outcomes Elie Akl, Shanil Ebrahim, Bradley Johnston, Pablo Alonso, Matthias.

Slides:



Advertisements
Similar presentations
Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ)
Advertisements

Meta-analysis: summarising data for two arm trials and other simple outcome studies Steff Lewis statistician.
Systematic Review of Literature Part XIX Analyzing and Presenting Results.
Exploring uncertainty in cost effectiveness analysis NICE International and HITAP copyright © 2013 Francis Ruiz NICE International (acknowledgements to:
Reporting systematic reviews and meta-analyses: PRISMA
prognosis of patients with Acute Myocardial Infarction remains dismal.
ODAC May 3, Subgroup Analyses in Clinical Trials Stephen L George, PhD Department of Biostatistics and Bioinformatics Duke University Medical Center.
The following slides were presented at a meeting of potential editors and methods advisors for the proposed Cochrane review group in February The.
Elements of a clinical trial research protocol
Evidence-Based Medicine Week 3 - Prognosis Department of Medicine - Residency Training Program Tuesdays, 9:00 a.m. - 11:30 a.m., UW Health Sciences Library.
Chapter 7. Getting Closer: Grading the Literature and Evaluating the Strength of the Evidence.
Cohort Studies Hanna E. Bloomfield, MD, MPH Professor of Medicine Associate Chief of Staff, Research Minneapolis VA Medical Center.
Introduction to evidence based medicine
Critical Appraisal of an Article by Dr. I. Selvaraj B. SC. ,M. B. B. S
Felix I. Zemel, MPH DrPH Student Tufts University School of Medicine.
Making all research results publically available: the cry of systematic reviewers.
Are the results valid? Was the validity of the included studies appraised?
STrengthening the Reporting of OBservational Studies in Epidemiology
1 Efficacy Results NDA (MTP-PE) Laura Lu Statistical Reviewer Office of Biostatistics FDA/CDER.
Epidemiology The Basics Only… Adapted with permission from a class presentation developed by Dr. Charles Lynch – University of Iowa, Iowa City.
Systematic Reviews Professor Kate O’Donnell. Reviews Reviews (or overviews) are a drawing together of material to make a case. These may, or may not,
Do we understand our data? Evaluating comprehension and usefulness of statistical methods for continuous outcomes in meta-analyses Furqaan Sadiq 1, Reem.
Acute Bacterial Rhinosinusitis. Brief Background Typically follows viral infection Dx is by clinical manifestations Streptococcus pneumoniae, Haemophilus.
1 ICEBOH Split-mouth studies and systematic reviews Ian Needleman 1 & Helen Worthington 2 1 Unit of Periodontology UCL Eastman Dental Institute International.
How to Analyze Systematic Reviews: practical session Akbar Soltani.MD. Tehran University of Medical Sciences (TUMS) Shariati Hospital
Evidence Based Medicine Meta-analysis and systematic reviews Ross Lawrenson.
A Systematic Review On The Hazards Of Aspirin Discontinuation Among Patients With Or At Risk For Coronary Artery Disease Giuseppe Biondi Zoccai Hemodynamics.
Systematic Review Module 7: Rating the Quality of Individual Studies Meera Viswanathan, PhD RTI-UNC EPC.
A Meta-Analysis of Interventions to Improve Chronic Illness Care Alexander Tsai 1 S.C. Morton 2, C.M. Mangione 3, E.B. Keeler 2 1 Case.
EBC course 10 April 2003 Critical Appraisal of the Clinical Literature: The Big Picture Cynthia R. Long, PhD Associate Professor Palmer Center for Chiropractic.
Criteria to assess quality of observational studies evaluating the incidence, prevalence, and risk factors of chronic diseases Minnesota EPC Clinical Epidemiology.
Conducting and Interpreting Systematic Reviews and Meta- Analyses July 12, 2007.
Landmark Trials: Recommendations for Interpretation and Presentation Julianna Burzynski, PharmD, BCOP, BCPS Heme/Onc Clinical Pharmacy Specialist 11/29/07.
Name:Shanil Ebrahim, MSc, MSc, PhD Departments of Clinical Epidemiology & Biostatistics, and Anesthesia McMaster University, Hamilton, Canada Stanford.
Christopher Eccleston Centre for Pain Research The University of Bath
Clinical Writing for Interventional Cardiologists.
RevMan for Registrars Paul Glue, Psychological Medicine What is EBM? What is EBM? Different approaches/tools Different approaches/tools Systematic reviews.
Lenalidomide Maintenance Therapy in Multiple Myeloma: A Meta-Analysis of Randomized Trials Singh PP et al. Proc ASH 2013;Abstract 407.
PH 401: Meta-analysis Eunice Pyon, PharmD (718) , HS 506.
EBM Conference (Day 2). Funding Bias “He who pays, Calls the Tune” Some Facts (& Myths) Is industry research more likely to be published No Is industry.
2nd Concertation Meeting Brussels, September 8, 2011 Reinhard Prior, Scientific Coordinator, HIM Evidence in telemedicine: a literature review.
Development and the Role of Meta- analysis on the Topic of Inflammation Donald S. Likosky, Ph.D.
Objectives  Identify the key elements of a good randomised controlled study  To clarify the process of meta analysis and developing a systematic review.
EBM --- Journal Reading Presenter :呂宥達 Date : 2005/10/27.
1 Lecture 10: Meta-analysis of intervention studies Introduction to meta-analysis Selection of studies Abstraction of information Quality scores Methods.
Research article structure: Where can reporting guidelines help? Iveta Simera The EQUATOR Network workshop 10 October 2012, Freiburg, Germany.
How Empty Are Empty Reviews? The first report on the Empty Reviews Project sponsored by the Cochrane Opportunities Fund and an invitation to participate.
Safety of Albumin Revisited Blood Products Advisory Committee Meeting March 17, 2005 Laurence Landow MD, FRCPC.
Systematic reviews and meta-analyses: when and how to do them Andrew Smith Royal Lancaster Infirmary 18 May 2015.
CONSORT 2010 Balakrishnan S, Pondicherry Institute of Medical Sciences.
1 Lecture 10: Meta-analysis of intervention studies Introduction to meta-analysis Selection of studies Abstraction of information Quality scores Methods.
Is a meta-analysis right for me? Jaime Peters June 2014.
Selenium supplementation for the primary prevention of cardiovascular disease: a Cochrane review Clinical
Characteristics of studies on lifestyle interventions included in review Gillies CL, et al. BMJ 2007;334:
Webinar May 25th METHYLPHENIDATE FOR CHILDREN AND ADOLESCENTS WITH ATTENTION DEFICIT HYPERACTIVITY DISORDER (ADHD)
Reference based multiple imputation;
Methods to Handle Noncompliance
for Overall Prognosis Workshop Cochrane Colloquium, Seoul
Reporting quality in preclinical studies Emily S Sena, PhD Centre for Clinical Brain Sciences, University of
Supplementary Table 1. PRISMA checklist
Randomized Trials: A Brief Overview
Fatimah Al-Ani 1,2,. MD MRCP, Jose Maria Bastida Bermejo3,
Heterogeneity and sources of bias
Challenges of statistical analysis in surgical trials
STROBE Statement revision
H676 Meta-Analysis Brian Flay WEEK 1 Fall 2016 Thursdays 4-6:50
JAMA Ophthalmology Journal Club Slides: Effect of Oral Voriconazole on Fungal Keratitis Prajna NV, TKrishnan T, Rajaraman R, et al; Mycotic Ulcer Treatment.
Q&A – studying medicine or health-related topics at university
A systematic survey shows that reporting and handling of missing outcome data in networks of interventions is poor Loukia M Spineli1; Juan Yepes-Nuñez2,3;
Use of Piecewise Weighted Log-Rank Test for Trials with Delayed Effect
Presentation transcript:

Addressing missing participant data in systematic reviews: Part I – Dichotomous outcomes Elie Akl, Shanil Ebrahim, Bradley Johnston, Pablo Alonso, Matthias Briel, Gordon Guyatt

Disclosures No conflicts of interest to disclose This work was funded by the Cochrane Methods Innovation Fund

Objective To describe how to use an innovative approach to addressing missing participant data for dichotomous outcomes in systematic reviews of randomized trials

Workshop plan Missing participant data at the RCT level Missing participant data at the SR level Practical issues Discussion

Workshop plan Missing participant data at the RCT level Missing participant data at the SR level Practical issues Discussion

Missing Participant Data MPD refers to: – participants excluded from the analysis of the effect estimate in the primary study because no data are available MDP does not refer to: – Missing studies (e.g., unpublished studies); – Missing outcome data (e.g., unreported outcomes); – Missing summary data (e.g., unreported SD); – Missing study-level characteristics (e.g., mean age, for subgroup or meta-regression analyses)

Dealing with MPD at the RCT level 87% of RCTs published in high impact medical journals had participants with missing data for the primary outcome The median percentage of participants with missing data was 6% (inter-quartile range 2% to 14%) Akl et al. BMJ May 18;344:e2809

STUDY DESIGN AND OBJECTIVE

CONCLUSION

Karl et al. BJU :24-6Primary outcome: overall success

Dealing with MPD at the RCT level Complete case analysis Make assumptions about the outcomes of participants with missing data: – None suffered the outcome – All suffered the outcome – Best case scenario – Worst case scenario

Karl et al. BJU :24-6

Dealing with MPD at the RCT level However, these assumptions are not plausible More plausible assumptions: based on RI MPD/FU – event incidence among those with MPD (not followed-up) relative to the event incidence among those followed up – RI MPD/FU = 2: event incidence among those with MPD is double the event incidence among those followed up Akl et al. BMJ May 18;344:e2809

Exercise (not followed-up) ? 980 (followed-up) 200 events (not followed-up) ? 990 (followed-up) 240 events

Exercise 1 solution

Dealing with MPD at the RCT level What are the advantages and disadvantages of: – Complete case analysis – Assuming none suffered the outcome – Assuming all suffered the outcome – Assuming best case scenario – Assuming worst case scenario – RI MPD/FU approach

Workshop plan Missing participant data at the RCT level Missing participant data at the SR level Practical issues Discussion

Dealing with MPD at the SR level What are the issues that systematic review authors need to deal with in relation to MPD?

Dealing with MPD at the SR level We will discuss how systematic reviewer authors need to: – Deal with MPD when producing the pooled effect estimate for the primary analysis – Assess risk of bias associated with MPD and the extent to which introduces confidence in results (quality of evidence)

Systematic review level - data analysis Systematic review level - data availability Trial level - data analysis Trial level - data collection Trial level - participant flow Trial level - participant entry Randomized Adherent and followed-up CollectedIncludedAvailableNonadherentCollectedIncludedAvailable According to trial analysis CollectedExcludedAvailable ITT, per protocol or as treated Not collected Excluded Not available Lost to follow-up Not collected ExcludedMissing CCA, or make assumptions Mistakenly randomized Appropriately excluded Exclude

Dealing with MPD at the SR level The Cochrane handbook encourages systematic reviewer authors to re-analyze a study’s effect estimate by including all randomized participants The handbook, however, fails to provide detailed guidance on how such analyses should be conducted

Proposal to handle MPD For the primary analysis: exclude participants with missing data (complete case analysis) When the primary analysis suggests important effect, we suggest sensitivity meta-analyses making different assumptions about the outcome of participants with missing data, to test the robustness of the results (the risk of bias) Akl et al. PLoS One. 2013;8(2):e57132

Handling dichotomous MPD

Judging RoB dichotomous MPD Results robust to a worst case scenario  missing data does not represent a risk of bias Results not robust to worst case scenario  test progressively more extreme assumptions culminating in a "worst plausible case” Important changes in results with such sensitivity analyses suggest serious RoB

Example 1 Meta-analysis assessing effects of probiotics for prevention clostridium difficile-associated diarrhea Johnston et al. Ann Intern Med Dec 18;157(12):878-88

Complete case analysis

Event rate: 1.5:1

Event rate: 3:1

Event rate: 5:1

Example 1 Based on these findings – Would you judge the risk of bias as: low or high? – Would you rate down the confidence in effect estimates (quality of evidence)?

Example 2 Meta-analysis comparing oral direct factor Xa inhibitors to low-molecular-weight heparin for thromboprophylaxis in patients undergoing total hip or knee replacement The primary analysis: – complete case analysis – factor Xa inhibitors reduced the incidence of symptomatic deep venous OR 0.46 ( ) Neumann et al. Ann Intern Med May 15;156(10)

Example 2 Two sensitivity analyses based on extreme but plausible assumptions : – RI LTFU/FU 2 and 3 for the intervention arm and 1 for control arm. – The effect estimates did not change appreciably: OR 0.54 ( ), 0.59 ( ) respectively Neumann et al. Ann Intern Med May 15;156(10)

Example 2 The results would lose statistically significance, OR 0.84 ( ), when we assumed: – the lowest incidence among intervention arms of all included trials for those with missing data in the control group – the highest incidence among control arms of all included trials for those with missing data in the intervention group Neumann et al. Ann Intern Med May 15;156(10)

Example 2 Based on these findings – Would you judge the risk of bias as: low or high? – Would you rate down the confidence in effect estimates (quality of evidence)?

Workshop plan Missing participant data at the RCT level Missing participant data at the SR level Practical issues Discussion

Practical issues Identifying in the RCT report, which participants were actually followed up, and which participants having data missing Automatic integration of MPD in the analysis

Identifying participants with missing data

Using the Excel sheet

Workshop plan Missing participant data at the RCT level Missing participant data at the SR level Practical issues Discussion

Is the proposed approach reasonable? Is the proposed approach feasible?

Thank you!

Intervention arm | Control arm a + e a (randomized) b (not followed-up) ? a-b (followed-up) c events e (randomized) f (not followed-up) ? e-f (followed-up) g events

Dealing with MPD Analytic methodIntervention armControl arm NumeratorDenominatorNumeratorDenominator Complete case analysisca-bge-f Worst case scenariob + cage Best case scenariocaf + ge None has eventcage All had eventb + caf + ge

Dealing with MPD Intervention armControl arm NumeratorDenominatorNumeratorDenominator Complete case analysisca-bge-f Assumption for MPD Worst case scenariob + cage Best case scenariocaf + ge None has eventcage All had eventb + caf + ge

Dealing with MPD Intervention armControl arm NumeratorDenominatorNumeratorDenominator Incidence same as observed in same arm [b x c/(a- b)] + c a [f x g/(e-f)] + g e Incidence in both arms same as observed in the trial control arm [b x g/(e- f)] + c a[f x g/(e-f) ] + g e

Missing Participant Data in trials When considering RCTs with statistically significant effect estimates: – 58% of RCTs lose statistical significance when applying a worse case scenario – up to 33% of RCTs lose statistical significance when applying plausible assumptions about the outcome of participants for whom data are missing Akl et al. BMJ May 18;344:e2809

Dealing with MPD at the SR level Cochrane SR (N=101) Non-Cochrane SR (N=101) Reported the number of participants with missing data 45 (45%)6 (6%) Described method for handling MPD None61 (61%)91 (92%) Complete case analysis14 (14%)2 (2%) Assumed none with MPD had the event7 (7%)1 (1%) Assumed all with MPD had the event4 (14%)1 (1%)