1 Xiaomei Yao, R. Brian Haynes, Nancy L. Wilczynski Health Information Research Unit, McMaster University Optimal sample sizes for testing search strategies for retrieving studies of treatment, diagnosis and prognosis from MEDLINE

2 Background MEDLINE covers 4800 journals and contains over 13 million articles in life sciences. Clinicians and researchers need to retrieve the target articles quickly.

3 Background con’d Unique optimal search filters (“hedges”) have been developed, based on the 161 core clinical journals published in 2000 from MEDLINE. ( ic/clinical.html). ic/clinical.html Determining optimal sample sizes for testing search strategies for retrieving studies of treatment, diagnosis and prognosis from MEDLINE.

4 Categories TypesOvid strategiesSensitivities (%)Specificities (%) TreatmentBest sensitivityclinical trial.mp. OR clinical trial.pt. OR random:.mp. OR tu.xs Best specificityrandomized controlled trial.mp. OR randomized controlled trial.pt Optimizationrandomized controlled trial.pt. OR randomized.mp. OR placebo.mp DiagnosisBest sensitivitySensitive:.mp. or diagnos:.mp. or di.fs Best specificitySpecificity.tw OptimizationSensitive:.mp. or predictive value:.mp. or accuracy:.tw PrognosisBest sensitivityIncidence.sh. OR exp mortality OR follow-up studies.sh. OR prognos:.tw OR predict:.tw. OR course:.tw Best specificityprognos:.tw. OR first episode.tw. OR cohort.tw Optimizationprognosis.sh. OR diagnosed.tw. OR cohort:.mp. OR predictor:.tw. OR death.tw. OR exp models, statistical Table 1 - Best search strategies for 3 categories retrieval from MEDLINE

5 Method The search terms were treated as “diagnostic tests” and hand searching as a “gold standard” articles were categorized as “pass” or “fail” studies for clinical topic areas. Precision analysis was used to calculate the sample size of “pass articles” needed to estimate sensitivity with 95% confidence intervals of ±0.01, ±0.02, ±0.025, ±0.03, ±0.04, ±0.05, ±0.10 (margin of error [ME]).

6 Method con’d The lowest sensitivity among 3 strategy types was chosen. Random samples of journals for size 100, 50, 30, 20, 10, etc. were generated. The order of the 161 journals was ranked in terms of the number of “pass articles”. STATA 9.0 was used as the statistical software.

7 Results sample size calculations based on search sensitivity were shown in Table 2. Table 3 shows the numbers of “pass articles” for the 3 purpose categories in different random samples of journals from the 161 database.

8 CategoriesME = ±0.01 ME = ±0.02 ME = ±0.025 ME = ±0.03 ME = ±0.04 ME = ±0.05 ME = ±0.10 Treatment Diagnosis Prognosis Table 2 – Sample size needed for different margins of error (MEs)

9 Random journal subsets from 161N for treatment (ME)N for diagnosis (ME)N for prognosis (ME) (±0.02)136 (±0.01)159 (±0.01) (±0.02)109 (±0.01)150 (±0.01) (±0.02)94 (±0.01)153 (±0.01) (±0.02)93 (±0.01)110 (±0.01) (±0.02)86133 (±0.01) (±0.02)83128 (±0.01) (±0.02)70125 (±0.01) (±0.02) (±0.025) (±0.02) (±0.02) (±0.025) (±0.03) (±0.04) (±0.04) (±0.05) (±0.05) (±0.10) (±0.10)01 Table 3 - The Numbers of “pass articles” (N) with their achieved MEs for random journal subsets

10 Results con’d Based on table 2 and table 3, we can find the smallest journal subset for any ME. The scatterplot of various MEs against random journal sample sizes for treatment is shown in Figure. For the diagnosis and prognosis categories, we only can achieve a ME = ±0.10 because of low prevalence of “pass articles”.

11 Figure

12 Results con’d We tested the previous optimal search strategies for the treatment category in different random journal subsets (Table 4). 1, 2, and 6 top journals can achieve a ME = ±0.10, ±0.05, and ±0.025 for treatment, respectively. The performances of the strategy in 1, 2, and 6 top journal subsets were very similar to the result of the 161 journals.

13 Ovid search strategies Sensitivities (%) of 100; 45; 15; 10 journal subsets, 161 journals (95% CI) Specificities (%) of 100; 45; 15; 10 journal subsets, 161 journals (95% CI) Precisions (%) of 100; 45; 15; 10 journal subsets, 161 journals (95% CI) Accuracies (%) of 100; 45; 15; 10 journal subsets, 161 journals (95% CI) Absolute differences: 100−161, 45−161, 15−161, Absolute differences: 100−161, 45−161, 15−161, Absolute differences: 100−161, 45−161, 15−161, Absolute differences: 100−161, 45−161, 15−161, Best sensitivity clinical trial.mp. OR clinical trial.pt. OR random:.mp. OR tu.xs (98.4, 99.6), 99.0 (98.0, 100.0), 98.4 (96.7, 100.2), (100.0, 100.0), 99.2 (98.7, 99.8) 71.5 (71.0, 71.9), 66.6 (65.7, 67.5), 69.3 (68.1, 70.5), 75.1 (73.6, 76.6), 70.4 (69.8, 70.9) 9.8 (9.2, 10.4), 9.2 (8.4, 10.1), 9.4 (8.1, 10.7), 9.9 (7.9, 11.9), 9.9 (9.3, 10.5) 72.3 (71.8, 72.8), 67.7 (66.8, 68.5), 70.2 (69.0, 71.3), 75.8 (74.3, 77.3), 71.3 (70.8, 71.8) −0.2, −0.8, , −3.8, −1.1, 4.7 −0.1, −0.7, −0.5, , −3.6, −1.1, 4.5 Table 4 - Performance characteristics in random sample 100, 45, 15, 10 journal subsets, and 161 journals for treatment category

14 Best specificity randomized controlled trial.mp. OR randomized controlled trial.pt (92.3, 95.3), 94.4 (92.2, 96.7), 90.1 (85.8, 94.3), 96.5 (92.6, 100.4), 93.1 (91.5, 94.8) 97.7 (97.5, 97.9), 97.3 (97.0, 97.6), 97.6 (97.2, 98.0), 97.4 (96.8, 97.9), 97.5 (97.3, 97.6) 56.0 (53.7, 58.4), 54.5 (50.8, 58.3), 55.0 (49.4, 60.5), 50.0 (42.4, 57.7), 54.4 (52.0, 56.8) 97.6 (97.4, 97.7), 97.2 (96.9, 97.5), 97.4 (97.0, 97.8), 97.3 (96.8, 97.9), 97.3 (97.1, 97.5) 0.7, 1.3, −3.0, , −0.2, 0.1, − , 0.1, 0.6, − , −0.1, 0.1, 0.0 Best optimization of sensitivity & specificity randomized controlled trial.pt. OR randomized.mp. OR placebo.mp (95.2, 97.5), 97.0 (95.3, 98.7), 93.2 (89.6, 96.8), 98.8 (96.5, 101.1), 95.8 (94.5, 97.1) 95.9 (95.7, 96.1), 92.7 (92.2, 93.2), 96.3 (95.8, 96.8), 95.1 (94.4, 95.9), 95.0 (94.8, 95.3) 42.2 (40.2, 44.2), 31.3 (28.7, 33.9), 45.0 (40.1, 49.9), 35.7 (29.6, 41.9), 38.5 (36.5, 40.5) 95.9 (95.7, 96.1), 92.8 (92.4, 93.3), 96.2 (95.7, 96.7), 95.2 (94.5, 96.0), 95.0 (94.8, 95.3) 0.5, 1.2, −2.6, , −2.3, 1.3, , −7.2, 6.5, − , −2.2, 1.2, 0.2 Table 4 con’d

15 Conclusion Our sample size calculations suggest that search strategies calibrated in small journal subsets will be as robust as those calibrated in larger collections for some categories that have relatively much more “pass articles”, such as treatment category with 3.24% of “pass articles”.

16 Conclusion con’d The "top yielding journal" approach seems most economical. We will develop and test the new search strategies in small random and top journal subsets to see how close the estimates are.

17 Acknowledgments The Hedges Team included Angela Eady, Brian Haynes, Susan Marks, Ann McKibbon, Doug Morgan, Cindy Walker-Dilks, Stephen Walter, Stephen Werre, Nancy Wilczynski, and Sharon Wong. Chris Cotoi (our computer programmer) helped us generate a journal randomizer.