Do we understand our data? Evaluating comprehension and usefulness of statistical methods for continuous outcomes in meta-analyses Furqaan Sadiq 1, Reem A. Mustafa 1,2, Bradley C. Johnston 2, Gordon H. Guyatt 2 1 UMKC School of Medicine, 2 McMaster University, Canada OBJECTIVE INTRODUCTION Clinicians rely on summary estimates from systematic reviews (SR) and meta- analyses for clinical decision making. Interpreting a treatment’s effects – large or small – can be difficult. When pooling results of trials, authors of SR report the differences between intervention and control groups in standard deviation units of Standardized Mean Difference (SMD). Presenting results as SMD is the longest standing and most widely used approach. SMD values 0.20 represents a small difference, 0.50 represents a moderate difference and 0.80 represents a large difference. To determine clinicians’ understanding and perceptions of 6 different approaches to the presentation of continuous outcome data in meta-analyses. The six include SMD, Minimal Important Difference Units, Natural Units, Relative Risk, Risk Difference and Ratio of Means METHODS DISCUSSION UMKC clinicians best understood continuous outcomes when presented as dichotomies (relative and absolute risk differences) and also found these presentations most useful. Presenting results as SMD, the longest standing and most widely used approach, was poorly understood and not perceived as useful. Strengths: This is one of the first studies to survey physicians about their understanding and perceived usefulness of different continuous outcome presentations. We used both perceived usefulness and percent of correct answer We surveyed both internal medicine and family medicine physicians We achieved an excellent response rate from the physicians surveyed Limitations: Data is from a single center, which may represent a biased view. However, the preliminary results from other centers support the same findings. There may have been a learning effect. However, we did not observe any systematic differences in responses based on forms with different orders. Surveys were distributed in an academic institution, so the results may not be generalized to practicing physicians outside academia. 63 clinicians responded (39 FM and 24 IM), all of which provided completed surveys (95.5% response rate) Data analysis entailed calculating proportion of participants answering correctly for small and large effects, along with number of participants who favored each approach. 95% Confidence Interval was constructed for each entity. Risk Difference was the approach best understood by clinicians, followed by the Ratio of Means and Relative Risk (Table 2). Clinicians generally found dichotomous presentation of continuous outcomes (Relative Risk; Risk Difference) very useful, and other approaches less useful (Table 3). ACKNOWLEDGEMENTS Funding: None We thank all participant clinicians who answered our survey References: Cochran Review Handbook 2011 RESULTS Table 3. Perceived Usefulness of the Presentation of Continuous Outcomes, n = 63 RESULTS Table 2. Understanding of the Presentation of Continuous Outcomes, n = 63 Survey design: Participants received paper-based, self-administered surveys presenting summary estimates of a hypothetical intervention versus placebo for chronic pain, with estimates demonstrating either a small effect or large effect for each of the 6 presentation approaches. We asked 6 questions addressing understanding and 6 questions addressing preferences. We randomized participants to size of effect and order. Participants: As part of a larger multicenter international study, we invited 66 staff, residents, and trainees in family medicine (FM) and internal medicine (IM) academic programs at University of Missouri-Kansas City (UMKC) to participate, 41 FM resident and staff and 25 IM residents. We evaluated 6 common presentation approaches found in systematic reviews and meta-analyses. (1) Standard deviation units – standardized mean difference (SMD) (2) Minimal important difference units (MID) (3) Conversion into natural units of the most commonly used instrument. (4) Conversion to relative effects, calculation of relative effects (e.g. relative risk) (5) Conversion to absolute effects, calculation of absolute effects (e.g. risk difference and the corresponding number needed to treat) (6) Ratio of means or ratio of change - ratio of means (RoM) or ratio of change (RoC) between the intervention and control groups ApproachMean (SD)(95%CI) SMD2.95 (1.43)[2.60 – 3.30] MID2.92 (1.53)[2.55 – 3.30] Natural units3.75 (1.61)[ ] Relative Risk4.16 (1.69)[ ] Risk Difference4.30 (1.73)[3.87 – 4.72] Ratio of Means3.95 (1.61)[ ] ApproachN (%) correct(95% CI) SMD16 (25.4%)(16.3% %) MID9 (14.3%)[7.71% %] Natural units10 (15.9%)[8.85% %] Relative Risk21 (33.3%)[22.9% %] Risk Difference24 (38.1%)[27.1% %] Ratio of Means22 (34.9%[24.3% %] Table 1. Demographic characteristics of respondents, n = 63 CharacteristicN(%) GenderMale Female (50.8%) (49.2%) SpecialtyInternal Medicine Family Medicine (38.1%) (61.9%) Professional statusStaff Trainee 6 57 (9.6%) (90.4%) Year graduated from medical school Before and after (4.8%) (1.6%) (19.0%) (74.6%) Training in Health Research Methods or Epidemiology Never completed a formal course Completed a formal course but no degree Have master/PhD degree in HRM (74.6%) (25.4%) (0%) 1-7 scale with higher numbers indicating higher preference