1. Addressing missing participant data in systematic reviews: Part I – Dichotomous outcomes Elie Akl, Shanil Ebrahim, Bradley Johnston, Pablo Alonso, Matthias Briel, Gordon Guyatt

2. Disclosures No conflicts of interest to disclose This work was funded by the Cochrane Methods Innovation Fund

3. Objective To describe how to use an innovative approach to addressing missing participant data for dichotomous outcomes in systematic reviews of randomized trials

4. Workshop plan Missing participant data at the RCT level Missing participant data at the SR level Practical issues Discussion

5. Workshop plan Missing participant data at the RCT level Missing participant data at the SR level Practical issues Discussion

6. Missing Participant Data MPD refers to: – participants excluded from the analysis of the effect estimate in the primary study because no data are available MDP does not refer to: – Missing studies (e.g., unpublished studies); – Missing outcome data (e.g., unreported outcomes); – Missing summary data (e.g., unreported SD); – Missing study-level characteristics (e.g., mean age, for subgroup or meta-regression analyses)

7. Dealing with MPD at the RCT level 87% of RCTs published in high impact medical journals had participants with missing data for the primary outcome The median percentage of participants with missing data was 6% (inter-quartile range 2% to 14%) Akl et al. BMJ. 2012 May 18;344:e2809

8. STUDY DESIGN AND OBJECTIVE

11. CONCLUSION

13. Karl et al. BJU. 2010.106:24-6Primary outcome: overall success

14. Dealing with MPD at the RCT level Complete case analysis Make assumptions about the outcomes of participants with missing data: – None suffered the outcome – All suffered the outcome – Best case scenario – Worst case scenario

15. Karl et al. BJU. 2010.106:24-6

18. Dealing with MPD at the RCT level However, these assumptions are not plausible More plausible assumptions: based on RI MPD/FU – event incidence among those with MPD (not followed-up) relative to the event incidence among those followed up – RI MPD/FU = 2: event incidence among those with MPD is double the event incidence among those followed up Akl et al. BMJ. 2012 May 18;344:e2809

19. Exercise 1 20000 1000 20 (not followed-up) ? 980 (followed-up) 200 events 1000 10 (not followed-up) ? 990 (followed-up) 240 events

20. Exercise 1 solution

21. Dealing with MPD at the RCT level What are the advantages and disadvantages of: – Complete case analysis – Assuming none suffered the outcome – Assuming all suffered the outcome – Assuming best case scenario – Assuming worst case scenario – RI MPD/FU approach

22. Workshop plan Missing participant data at the RCT level Missing participant data at the SR level Practical issues Discussion

23. Dealing with MPD at the SR level What are the issues that systematic review authors need to deal with in relation to MPD?

24. Dealing with MPD at the SR level We will discuss how systematic reviewer authors need to: – Deal with MPD when producing the pooled effect estimate for the primary analysis – Assess risk of bias associated with MPD and the extent to which introduces confidence in results (quality of evidence)

25. Systematic review level - data analysis Systematic review level - data availability Trial level - data analysis Trial level - data collection Trial level - participant flow Trial level - participant entry Randomized Adherent and followed-up CollectedIncludedAvailableNonadherentCollectedIncludedAvailable According to trial analysis CollectedExcludedAvailable ITT, per protocol or as treated Not collected Excluded Not available Lost to follow-up Not collected ExcludedMissing CCA, or make assumptions Mistakenly randomized Appropriately excluded Exclude

26. Dealing with MPD at the SR level The Cochrane handbook encourages systematic reviewer authors to re-analyze a study’s effect estimate by including all randomized participants The handbook, however, fails to provide detailed guidance on how such analyses should be conducted

27. Proposal to handle MPD For the primary analysis: exclude participants with missing data (complete case analysis) When the primary analysis suggests important effect, we suggest sensitivity meta-analyses making different assumptions about the outcome of participants with missing data, to test the robustness of the results (the risk of bias) Akl et al. PLoS One. 2013;8(2):e57132

28. Handling dichotomous MPD

29. Judging RoB dichotomous MPD Results robust to a worst case scenario  missing data does not represent a risk of bias Results not robust to worst case scenario  test progressively more extreme assumptions culminating in a "worst plausible case” Important changes in results with such sensitivity analyses suggest serious RoB

30. Example 1 Meta-analysis assessing effects of probiotics for prevention clostridium difficile-associated diarrhea Johnston et al. Ann Intern Med. 2012 Dec 18;157(12):878-88

31. Complete case analysis

32. Event rate: 1.5:1

33. Event rate: 3:1

34. Event rate: 5:1

35. Example 1 Based on these findings – Would you judge the risk of bias as: low or high? – Would you rate down the confidence in effect estimates (quality of evidence)?

36. Example 2 Meta-analysis comparing oral direct factor Xa inhibitors to low-molecular-weight heparin for thromboprophylaxis in patients undergoing total hip or knee replacement The primary analysis: – complete case analysis – factor Xa inhibitors reduced the incidence of symptomatic deep venous OR 0.46 (0.30-0.70) Neumann et al. Ann Intern Med. 2012 May 15;156(10)

37. Example 2 Two sensitivity analyses based on extreme but plausible assumptions : – RI LTFU/FU 2 and 3 for the intervention arm and 1 for control arm. – The effect estimates did not change appreciably: OR 0.54 (0.37-0.80), 0.59 (0.40-0.87) respectively Neumann et al. Ann Intern Med. 2012 May 15;156(10)

38. Example 2 The results would lose statistically significance, OR 0.84 (0.59-1.20), when we assumed: – the lowest incidence among intervention arms of all included trials for those with missing data in the control group – the highest incidence among control arms of all included trials for those with missing data in the intervention group Neumann et al. Ann Intern Med. 2012 May 15;156(10)

39. Example 2 Based on these findings – Would you judge the risk of bias as: low or high? – Would you rate down the confidence in effect estimates (quality of evidence)?

40. Workshop plan Missing participant data at the RCT level Missing participant data at the SR level Practical issues Discussion

41. Practical issues Identifying in the RCT report, which participants were actually followed up, and which participants having data missing Automatic integration of MPD in the analysis

42. Identifying participants with missing data

44. Using the Excel sheet

45. Workshop plan Missing participant data at the RCT level Missing participant data at the SR level Practical issues Discussion

46. Is the proposed approach reasonable? Is the proposed approach feasible?

47. Thank you!

48. Intervention arm | Control arm a + e a (randomized) b (not followed-up) ? a-b (followed-up) c events e (randomized) f (not followed-up) ? e-f (followed-up) g events

49. Dealing with MPD Analytic methodIntervention armControl arm NumeratorDenominatorNumeratorDenominator Complete case analysisca-bge-f Worst case scenariob + cage Best case scenariocaf + ge None has eventcage All had eventb + caf + ge

50. Dealing with MPD Intervention armControl arm NumeratorDenominatorNumeratorDenominator Complete case analysisca-bge-f Assumption for MPD Worst case scenariob + cage Best case scenariocaf + ge None has eventcage All had eventb + caf + ge

51. Dealing with MPD Intervention armControl arm NumeratorDenominatorNumeratorDenominator Incidence same as observed in same arm [b x c/(a- b)] + c a [f x g/(e-f)] + g e Incidence in both arms same as observed in the trial control arm [b x g/(e- f)] + c a[f x g/(e-f) ] + g e

52. Missing Participant Data in trials When considering RCTs with statistically significant effect estimates: – 58% of RCTs lose statistical significance when applying a worse case scenario – up to 33% of RCTs lose statistical significance when applying plausible assumptions about the outcome of participants for whom data are missing Akl et al. BMJ. 2012 May 18;344:e2809

53. Dealing with MPD at the SR level Cochrane SR (N=101) Non-Cochrane SR (N=101) Reported the number of participants with missing data 45 (45%)6 (6%) Described method for handling MPD None61 (61%)91 (92%) Complete case analysis14 (14%)2 (2%) Assumed none with MPD had the event7 (7%)1 (1%) Assumed all with MPD had the event4 (14%)1 (1%)