1. Study Eligibility Criteria Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide www.ahrq.gov

2. Systematic Review Process Overview

3.  To understand the role of study eligibility criteria (inclusion/exclusion) in framing a systematic review  To know when and how to set study eligibility criteria  To understand the effect of study eligibility criteria on interpretation of a review Learning Objectives

4.  Determine which studies will be included in the analysis  Function the same in systematic reviews as in primary research  Should reflect the analytic framework and key questions  Are powerful tools for widening or narrowing the scope of a review  Provide information to determine whether reviews can be compared or combined Study Eligibility Criteria

5.  Population type (adult, community-dwelling females)  Intervention (screening tool to identify depression risk)  Country (United States only)  Setting (hospital-based studies only)  Study population size (N > 200)  Study design (randomized controlled trials) Some Sample Criteria

6.  Can be as broad as “all related studies”  Can be helpful for exploring “what is known”  May result in too much literature to feasibly review  Could pull in disparate literature that cannot be compared Using Broad Criteria

7.  Can be helpful in culling homogenous literature  Can reduce size of the literature to a manageable scope  Could be appropriate for a narrow research question  May reduce applicability of the review and sometimes risks introducing bias Using Narrow Criteria

8.  Choosing how broad or how narrow to set criteria requires a balance of obtaining adequate information to answer a given question without obscuring the results with irrelevant literature.  Review teams should work together to find this balance.  The overarching goal is to minimize bias related to which studies are selected. Refining Criteria

9.  Distortion of the estimate of effect that comes from how studies are selected for inclusion  Affects the applicability, or “external validity,” of the review itself Bias in This Context

10.  Inappropriate eligibilty criteria may limit applicability of the review.  Example: the use of studies of twin pregnancies in a review of preterm labor management for low-risk women  Eligibility criteria may result in the inclusion of more of a certain study type that either overestimates or underestimates effectiveness. Examples of Bias in This Context

11.  Review study goals  Assess analytic framework and key questions  Set criteria before beginning abstract review Selecting Criteria

12.  Population — condition, disease severity and stage, comorbidities, patient demographics  Intervention — dosage, frequency, method of administration  Comparator — placebo, usual care, or active control  Outcome — health outcomes, morbidity, mortality, quality of life  Timing — Duration of followup  Setting — Primary, specialty, inpatient, cointerventions Tie Criteria to PICOTS

13.  What study designs should be included?  Include studies in foreign languages?  Include studies conducted in other countries but published in English?  Include “grey” or “fugitive” literature?  Include year of publication? Other Considerations for Setting Criteria

14.  Limit to randomized controlled trials?  Include observational studies?  If so, what specific types?  What is the value of a case series?  How is the definition of case series operationalized? Types of Studies To Be Included

15.  There are four main types of observational studies:  Cohorts (with comparisons)  Case controls  Case series  Registries/databases Observational Studies (I)

16.  Well-conducted observational studies can address issues of applicability and the need for longer-term outcomes if they:  include more representative patient populations,  have relevant comparators, and  report more meaningful clinical outcomes over longer time frames.  Observational studies may be a better source of information about harms. Observational Studies (II)

17.  Positive findings may be more likely to be published in high-profile English-language journals.  Therefore, to include only English-language journals may overestimate the positive effect of an intervention.  Empirically, the bias associated with limiting a review to English-language reports has been shown to be small. Non-English–Language Study Reports Gregoire G, et al. J Clin Epidemiol 1995;48:159-63; Moher D, et al. J Clin Epidemiol 2000;53:964-72.

18.  Material that may be difficult to identify and retrieve:  Government reports and regulatory sites (Center for Drug Evaluation and Research/U.S. Food and Drug Administration)  Pharmacoepidemiologic databases, including postmarketing surveillance  Book chapters  Conference proceedings  Published dissertations  Scientific information packets (Agency for Healthcare Research and Quality)  Rationale for a search:  Because journals may publish positive or statistically significant results, finding grey literature of unpublished nonsignificant or null results may indicate the presence of publication bias. Grey or “Fugitive “Literature

19.  Literature searches typically have date parameters (i.e., earliest or latest date of publication).  Date parameters are particularly important when there has been a change in policy, practice, or formulation that makes older studies less applicable. Year of Publication

20.  The rest of this module contains a number of questions and some sample criteria.  Using your newly acquired knowledge of selection criteria and the guidelines described in the previous slides, work through the thought exercises and examples that follow. Using the Criteria

21. What would you do if you were asked to review the literature on transition support for adolescents with autism spectrum disorder who are entering adulthood? Before seeing the key questions, consider the categories of criteria that you will want to apply. Exercise 1

22.  What is the relevant population?  What is the intervention of interest?  To what exposure is the intervention being compared?  What outcomes are relevant?  What short-term or long-term outcomes should be considered?  In what setting would the results be applicable? Exercise 1: Basic PICOTS Questions PICOTS = population, intervention, comparators, outcomes, timing, and setting

23. Population What constitutes an adolescent? What constitutes a diagnosis of autism spectrum disorder (ASD)? InterventionHow is transition support defined? Comparator Do we compare to no transition support or directly compare types of support? Outcome What are the goals for adolescents with ASD as they transition to adulthood? TimingHow quickly should the outcomes be apparent? Setting Is transition support provided in multiple settings, such as schools, clinics, and the community? Exercise 1: PICOTS

24.  A study that included “individuals over 10 years of age”?  A paper about an intervention for individuals with a range of developmental disabilities, not only autism spectrum disorder (ASD)?  A study of children with Asperger’s syndrome but not other forms of ASD? Exercise 1: What Would You Do With...

25.  There are no “right” answers to these questions.  The team should consider how selection criteria may bias the review or affect applicability.  The team should clearly define how eligibility criteria are operationalized and record these decisions throughout the review. Exercise 1: No “Right” Answers

26. What is the efficacy of home uterine activity monitoring for preventing preterm birth among women at low risk of a preterm birth? Exercise 2: Selecting Criteria for a Narrow Research Question

27.  Efficacy: randomized controlled trials only  Low risk: no previous preterm birth  Yield: 11 studies Exercise 2: Implications of a Question That Requires Narrow Criteria

28. What is the best management strategy for overactive bladder (OAB) among women? consideration: OAB is a fairly difficult condition to define. The study of OAB is often combined with other types of urological conditions, such as stress incontinence or prostate issues, when studied. OAB is treated pharmacologically, behaviorally, and surgically and with complementary and alternative interventions — each area includes very different types of studies. Exercise 3: Selecting Criteria for a Broad Research Question

29.  The research team required a minimum sample size N ≥ 50 in each study at study start.  Implications:  Total excluded for N < 50: 79  Excluded for N < 20:36  Excluded for N 20 to 29:23  Excluded for N 30 to 39: 8  Excluded for N 40 to 49:12 Exercise 3: Impact of Study-Size Criteria on Excluded Studies

30.  Studies had to include at least 75 percent women.  This decision was based on expert opinion and the size and scope of the literature.  Based on this criterion, 40 studies were excluded.  Another 27 studies would have been excluded had the review been limited to studies of only women. Exercise 3: Impact of Gender on Study Selection Criteria

31.  Case series with N ≥ 2 were included.  Only 3 of 169 studies were randomized controlled trials; 122 were case series.  Because of the relative newness of this area of research, it was important to capture data even from studies without comparison groups. Example 1: A Maternal-Fetal Surgery Technical Brief

32.  A systematic review on outcomes of cesarean delivery on maternal request.  Conducted for the National Institutes of Health- Office of Medical Applications of Research (NIH- OMAR) State-of-the-Science Conference Example 2: Systematic Review of Cesarean Delivery

33.  No evidence on outcomes of cesarean delivery on maternal request (CDMR) versus other modes of delivery  Urgent need for actionable evidence  Need to recognize and account for confounders Example 2: The Challenge in the Systematic Review on Cesarean Delivery

34.  Expand search to include proxies  Weight rungs of evidence to account for confounding  Highest rung: trials of breech delivery, but only for maternal outcomes  Lower rung: planned cesarean versus planned vaginal delivery  Lowest rung: comparisons of maternal and neonatal outcomes from actual modes of delivery Example 2: The Solution in the Systematic Review on Cesarean Delivery

35.  Eligibility criteria:  Specify the scope of the literature to be included  Should be tied to the analytic framework, key questions, and PICOTS  Can reduce bias and support applicability, but run the risk of introducing bias if not chosen appropriately.  Must be applied consistently among individuals on the review team  Need to be recorded throughout the project and in the report for transparency and replicability Key Messages PICOTS = population, intervention, comparator, outcome, timing, and setting

36.  Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994;309:1286-91.  Gregoire G, Derderain F, Le L. Selecting the language of the publication included in a meta-analysis: is there a Tower of Babel bias? J Clin Epidemiol 1995;48:159-63.  McAuley L, Pham B, Tugwell P, et al. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 2000;356:1228-31.  McPheeters ML, Briss P, Teutsch SJ, et al. Systematic reviews in public health. In: Brownson RC and Petitti DB, eds. Applied epidemiology. New York: Oxford University Press; 2006. p. 99-124.  Moher D, Pham B, Klassen TP, et al. What contribution do languages other than English make on the results of meta-analyses? J Clin Epidemiol 2000; 53:964-72. References

37.  This presentation was prepared by Melissa L. McPheeters, Ph.D., M.P.H., a member of the Vanderbilt University Evidence-based Practice Center.  The information in this module is currently not included in Version 1.0 of the Methods Guide for Comparative Effectiveness Reviews (available at: http://www.effectivehealthcare. ahrq.gov/ehc/products/60/294/2009_0805_princi ples1.pdf). Author