1. Topic Refinement Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide www.ahrq.gov

2. Systematic Review Process Overview

3.  To describe Effective Health Care Program principles  To identify the importance of key questions  To describe the characteristics of a well- constructed key question  To describe the role of key informants in topic refinement  To describe a longitudinal and transparent topic refinement process Learning Objectives

4.  The Effective Health Care (EHC) Program aims to provide evidence-based information to health care stakeholders.  The work of the EHC Program is intended to be:  relevant and timely,  objective and scientifically rigorous, and  transparent with public participation. Effective Health Care Program Principles Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/ ?pageaction= displayproduct&productid=318.

5. Topic Refinement in the Comparative Effectiveness Review (CER) Process Develop Topic Identify and triage topics based on appropriateness, importance, desirability of new research/duplication, feasibility, and potential impact. Refine Topic Identify patient intervention, comparator, outcomes, timing, and setting of each topic. Review Topic Prepare topic, search for and select studies, abstract data, analyze and synthesize data, and present findings. Engage Stakeholders in Clarifying Areas for Future Research Identify future research needs and inform real world health care decisions.

6.  Topics for refinement will be:  appropriate, of high priority, feasible, not duplicative, and of high potential value.  A key feature of topic refinement is the formulation of key questions.  Key questions are objective demonstrations of Effective Health Care Program principles.  They should reflect uncertainty stakeholders have (decisionmakers, clinicians, patients, others). Topic Refinement Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/ ?pageaction=displayproduct&productid=318.

7. Does aliskerin reduce blood pressure in hypertensive patients more than placebo?  Is this key question reflecting uncertainties?  No. Aliskerin is approved by the U.S. Food and Drug Administration for hypertension, so it obviously reduces blood pressure more than placebo. A Question That Does Not Reflect Uncertainty

8. Does aliskerin reduce blood pressure in hypertensive patients more than other antihypertensives?  Clinicians and decisionmakers would be particularly interested. Is the antihypertensive effect of aliskerin impacted by ethnicity or genomic parameters?  African Americans and patients with angiotensin-converting enzyme (ACE) gene polymorphisms have less-robust blood pressure reductions with other renin-angiotensin-aldosterone blocking agents.  Patients, clinicians, and decisionmakers would be particularly interested. Questions That Reflect Uncertainty (I)

9. Can aliskerin reduce the risk of mortality, myocardial infarction, and stroke in patients with hypertension?  Because thiazide diuretics already reduce blood pressure and reduce final health outcomes, the impact of a new antihypertensive on final health outcomes is important in determining its role in therapy.  Clinicians and decisionmakers are particularly interested. What is the risk of renal dysfunction, hyperkalemia, cough, and angioedema associated with the use of aliskerin?  Harms need to be considered in order to determine the balance of benefits to harms.  Patients are particularly interested. Questions That Reflect Uncertainty (II)

10.  Key questions guide the entire systematic review process:  Literature search  Inclusion and exclusion criteria  Types of data extracted  Data synthesized and reported Importance of Getting the Questions Right Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318.

11.  Population — Who is being evaluated?  Intervention — What intervention is being evaluated?  Comparator — What is the intervention being compared with?  Outcome — What are the benefits and harms being evaluated?  Timing — What is the timing of outcomes or timing of followup?  Setting — What are the settings of interest? Key Questions Derived From Assessing PICOTS Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318.

12.  Key questions must be:  clear, precise, and relevant to stakeholders.  Key questions should not be posed because:  the answers are assumed to be known, or  there seems to be adequate literature to evaluate them.  Understand the topic before key questions are posed.  Background reading and key informants are critical to understanding the PICOTS components.  Identify appropriate PICOTS components, understand the topic. Defining Good Key Questions Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318. PICOTS = population, intervention, comparators, outcomes, timing, and setting

13. Devise an Analytic Framework* *See Analytic Frameworks module Phung OJ, et al. Effectiveness of Recombinant Human Growth Hormone (rhGH) in the Treatment of Patients With Cystic Fibrosis [draft report]. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction= displayproduct&productid=391.

14.  Key informants (stakeholders) serve a variety of important functions:  They help formulate key questions that address real-world dilemmas.  Key questions should be devised to address important health care dilemmas, not focus on interests pertinent only to researchers.  They provide context to help discern content areas and applicability.  They ensure transparency in the process. Roles of Key Informants Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/ ?pageaction=displayproduct&productid=318.

15.  Key informants provide context.  They explore their beliefs about advantages or disadvantages of alternative treatments based on  pragmatic trials,  indirect evidence,  understanding of pathophysiology, and  understanding of current practice.  They identify important outcomes or information that may affect decisionmaking. Gaining Context Through Key Informants Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/ ?pageaction=displayproduct&productid=318.

16.  Key informants are stakeholders.  Clinicians:  clarify (sub)populations to focus on,  identify areas in which studies differ that can affect applicability, and  identify what contemporary practice looks like.  Patients:  provide insight not usually appreciated by other stakeholders, and  may focus more on quality of life and harms. Types of Key Informants (I) Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/ ?pageaction=displayproduct&productid=318.

17.  Other key stakeholders that can be used as key informants include:  employers and business groups,  Federal and State partners,  health plans,  policymakers, and  researchers.  Health care industry representatives are key stakeholders but are not included on key informant panels. Types of Key Informants (II) Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/ ?pageaction=displayproduct&productid=318.

18.  After key informants provide insight, a final draft of the topic refinement document is devised and the key question posting document posted for public comment on the Effective Health Care Program Web site.  Public comment allows other stakeholders, including industry representatives, the opportunity to provide input.  Public comment is a critical final check of underlying assumptions about relevance and ensures transparency for all stakeholders. Public Comment Enhances Input and Transparency Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/ ?pageaction=displayproduct&productid=318.

19.  All public comments that can be answered with the existing Methods Guide are answered generically.  All other public comments are answered individually.  Changes to the key questions are made or further explanations of why certain aspects are not being changed are laid out.  Changes to key questions are incorporated into a draft protocol, which is reviewed at the first Technical Expert Panel meeting. Revising Key Questions After Public Comment Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/ ?pageaction=displayproduct&productid=318.

20.  Step 1: Core leadership team reads topic development document — with title, a preliminary PICOTS description, and a brief review of what systematic reviews and studies exist — and the topic refinement template.  Topic refinement on use of rhGH in cystic fibrosis  Step 2: Core leadership team selects a team from within the Evidence- based Practice Center that has knowledge, interest, and/or contacts within the topic area.  Lead investigator, senior scientist, research fellow identified  Step 3: Selected team performs general review of the topic area, focusing on tertiary texts describing the disease process or interventions of interest and then any practice guidelines.  General review conducted, presented by senior scientist Topic Refinement Process: UCONN/HH EPC Example (I) PICOTS = population, intervention, comparators, outcomes, timing, and setting; UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

21.  Step 4: The team reviews the topic development documents and continues more focused background reading.  Step 5: The team proposes the preliminary search strategy.  Approved by medical librarian and conducted  Step 6: The citations are reviewed, a body of literature is identified, and data on PICOTS are extracted; no results are reviewed.  Step 7: The team determines the PICOTS and the analytic framework and recruits key informants. Topic Refinement Process: UCONN/HH EPC Example (II) PICOTS = population, intervention, comparators, outcomes, timing, and setting; UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

22.  P — Patients with cystic fibrosis  I — Recombinant human growth hormone  C — Placebo, no therapy, another active therapy, or no control therapy  O — Pulmonary variables, anthropometrics, bone variables, IGF variables, hospitalization, bone fracture, health-related quality of life, glucose intolerance, malignancy, and mortality  T — At least 4 weeks of therapy with recombinant human growth hormone  S — Outpatient primary care, endocrine, or cystic fibrosis clinic Topic Refinement Process: UCONN/HH EPC Example (III) UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

23. Topic Refinement Process: UCONN/HH EPC Example (IV) Phung OJ, et al. Effectiveness of Recombinant Human Growth Hormone (rhGH) in the Treatment of Patients With Cystic Fibrosis [draft report]. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and- reports/?pageaction= displayproduct&productid=391. UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

24.  Step 8: Preliminary key questions are posed.  Step 9: List of questions for key informants is posed.  Step 10: Key informants are shown a draft of the topic refinement document.  Step 11: After 2 weeks, the topic refinement document is discussed with key informants and questions are posed. Topic Refinement Process: UCONN/HH EPC Example (V) UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

25.  Questions for key informants  Which of the end points available in the trials/studies are the most important to you as a researcher, clinician, payer, or patient?  Is growth accurately portrayed in this analytic framework as an intermediate outcome, or is it an important health outcome in its own right?  If the data suggest improvements in intermediate outcomes (growth velocity, exercise tolerance, pulmonary function tests, and protein turnover), how comfortable/confident are you that patients will derive the terminal benefits of interest (mortality, infections, fractures, etc.)?  Are there other health outcome benefits that may result from human growth hormone therapy, other than the ones already identified? Topic Refinement Process: UCONN/HH EPC Example (VI) UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

26.  Questions for Key Informants  How different is a study including people up to 23 years of age from a study limited to 18-year-olds? Consider this given the inverse linear relationship between human growth hormone efficacy and age, as well as the delayed growth and maturation inherent in this population.  How readily do you feel we can combine certain end points (e.g., lean body mass measured by different methods [DEXA, skin folds, BIA] in same units) or amino acid catabolism measures (leucine vs. glutamine)? Topic Refinement Process: UCONN/HH EPC Example (VII) BIA = bioelectric impedance analysis; DEXA = dual energy x-ray absorptometry; UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

27.  Questions for Key Informants:  In trials/studies that report multiple similar end points (like pulmonary function tests), which surrogate end points would be optimal (i.e., FEV1 percent vs. FEV1 absolute value)?  Given the paucity of harms data, could harms data from patients without cystic fibrosis be used to inform the potential risks associated with human growth hormone therapy (i.e., how similar are patients with hypogonadism to those with cystic fibrosis and poor growth progression)?  In particular, are you wary of using human growth hormone because of its risk of neoplasms? Topic Refinement Process: UCONN/HH EPC Example (VIII) FEV1 = forced expiratory volume in 1 second; UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

28.  Step 12: Based on feedback, the revised topic refinement and key question posting document is generated.  Step 13: The topic refinement and key question posting document is sent to AHRQ.  Step 14: After editorial review, the key question posting document is posted for public comment on the Effective Health Care Program Web site. Topic Refinement Process: UCONN/HH EPC Example (IX) UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

29.  Step 15: Public comments are sent to the Evidence-based Practice Center, and revisions are made in response to the comments.  Step 16: Changes to key questions are incorporated into a draft protocol that is reviewed at the first Technical Expert Panel meeting. Topic Refinement Process: UCONN/HH EPC Example (X) UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

30.  KQ 1: In patients with cystic fibrosis, does treatment with rhGH as an adjuvant to usual care improve intermediate outcomes, including pulmonary function (FEV percent and FEV1 absolute value), growth (height, weight, lean body mass, protein turnover), exercise tolerance, and bone mineralization, compared with usual care alone?  KQ 2: In patients with cystic fibrosis, does treatment with rhGH as an adjuvant to usual care improve health outcomes, including frequency of required intravenous antibiotic treatments, frequency of hospitalization, quality of life, bone fracture or development of osteoporosis/osteopenia, or mortality, compared with usual care alone?  KQ 3: In patients with cystic fibrosis, what is the strength of evidence that intermediate outcomes of pulmonary function, growth, and bone mineralization are associated with improvements in health outcomes of quality of life, bone fracture or development of osteoporosis/osteopenia, or mortality?  KQ 4: In patients with cystic fibrosis, what is the frequency of nonmalignant serious adverse effects resulting from treatment with rhGH? Adverse effects of interest include, but are not limited to: glucose intolerance, diabetes, and hypoglycemia. Key Questions (I)

31.  KQ 5: What is the risk of malignancy associated with rhGH use as determined by (a) markers of cancer risk with rhGH (IGF-I increases over 100 ng/ml or IGFBP-3 decreases over 1,000 ng/ml) from studies of rhGH in people with CF [cystic fibrosis], and (b) assessment of evidence on cancer incidence from non-CF patients receiving modest doses of rhGH (0.2 mg/kg per week to 0.6 mg/kg per week) for disorders such as growth hormone deficiency and idiopathic short stature?  KQ 6: In patients with CF, how are efficacy, effectiveness, safety, or adverse events impacted by rhGH dose, therapy duration, baseline nutritional status, and concurrent medical therapies?  KQ 7: In patients with CF, how do the efficacy, effectiveness, safety, or adverse events of treatment with rhGH differ between subgroups of patients? Subgroup characteristics of interest include, but are not limited to, age (prepubertal, pubertal, postpubertal); gender; baseline clinical status (height, weight, lean body mass, pulmonary function, exercise tolerance, nutritional status); and/or the nature, extent, and effectiveness of prior treatment. Key Questions (II)

32.  The topic refinement process is devised to ensure quality, clinical relevance, and transparency.  Understanding the topic area is important to describing the PICOTS and devising the analytic framework.  Studies are evaluated in this phase to review methods and demographics, not results.  Key informants provide context and ensure relevance and transparency.  Key informants cannot be limited to researchers.  Public comment is a final verification of relevance, and ensures transparency and a broad range of input. Key Messages

33.  Phung OJ, Coleman CI, Baker EL, et al. Effectiveness of Recombinant Human Growth Hormone (rhGH) in the Treatment of Patients With Cystic Fibrosis, Draft Comparative Effectiveness Review (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, Posted March 2010. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides- reviews-and-reports/?pageaction=displayproduct&productid=391.  Whitlock EP, Lopez SA, Chang S, et al. Identifying, selecting, and refining topics. In: Methods guide for comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality; posted April 2009. Available at: http://www.effectivehealthcare.ahrq. gov/index.cfm/search- for-guides-reviews-and-reports/?pageaction= displayproduct&productid=318. References

34.  This presentation was prepared by C. Michael White, Pharm.D., FCP, FCCP, a member of the University of Connecticut/Hartford Hospital Evidence-based Practice Center.  The module is based on chapter 2 in version 1.0 of the Methods Reference Guide for Effectiveness and Comparative Effectiveness Reviews (available at: http://effectivehealthcare.ahrq.gov/repFiles/20 07_10DraftMethodsGuide.pdf). Author