1. Systematic Review Module 1: Refining Key Questions C. Michael White, PharmD, FCP, FCCP Professor and Director University of Connecticut / Hartford Hospital Evidence-based Practice Center Speaker has no actual or potential conflicts of interest in relation to this activity

2. 1 Learning Objectives At the conclusion of this activity, the successful learner will be able to – – Describe what the Effective Healthcare Program is – – Describe how topics are selected for key question refinement – – Identify the importance of key questions – – Describe the role of key informants in topic refinement – – Describe a longitudinal and transparent topic refinement process

3. 2 Effective Healthcare Program Principles The work of the EHC program is intended to be – – Relevant and timely – – Objective and scientifically rigorous – – Transparent with public participation Whitlock EP, Lopez SA, Chang S, et al. Identifying, selecting, and refining topics. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

4. 3 Topic Refinement in the CER Process

5. 4 Topic Refinement Topics for refinement will be – – Appropriate, high priority, feasible, not duplicative, and of high potential value A key feature of topic refinement is the formulation of key questions Key questions are objective demonstrations of EHC program principles – – Should reflect uncertainty stakeholders have Decisionmakers, clinicians, patients, others Whitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

6. 5 Key Questions (I) Key questions guide the entire systematic review process – – Literature search – – Inclusion and exclusion criteria – – Types of data extracted – – Data synthesized and reported Key questions must be – – Clear and precise – – Relevant to stakeholders Whitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

7. 6 Key Questions (II) Key questions should not be posed because – – The answer is assumed to be known – – There seems to be adequate literature to evaluate it An understanding of the topic area is critical – – Understand PICOTS (explained in next slide), understand the topic – – Background reading and key informants are critical to understanding PICOTS Elements of PICOTS outlined in general form during topic development – – Comprehensive development of PICOTS is required during topic refinement before key questions are posed Whitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

8. 7 Key Questions Derived From Assessing PICOTS Population—Who is being evaluated? Intervention—What intervention is being evaluated? Comparator—What is the intervention being compared with? Outcomes—What are the benefits and harms being evaluated? Timing—What is the follow-up time? Setting—What are the settings of interest? Whitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

9. 8 Devise Analytic Framework (See Module 2) Intermediate Outcomes  Maximal concentrations  Minimal concentrations at steady state  Area under the curve  Average steady state concentrations  Dose needed to control seizure  Switchback rate from generic to innovator antiepileptic medication [ Pharmacokinetic data sought in epileptic patients only ] Final Health Outcomes  Mortality  Hospitalization  Office/emergency department visits  Composite of ambulance services, hospitalization, or emergency department visits for epilepsy  Health-related quality of life  Breakthrough seizures  Frequency of seizures  Secondary seizure injury (fracture, laceration, head injury, aspiration pneumonia)  Status epilepticus  Loss of driver’s license  Loss of employment  Total adverse events  Neurological adverse effects  Hypotension  Rash  Suicidal ideation  Dizziness (KQ 1) (KQ 3) (KQ 2) Patients with epilepsy using generic versus innovator antiepileptic medication White CM. AHRQ Topic Refinement Report, Submitted 7.15.09.

10. 9 Key Informants (I) Key informants (stakeholders) serve a variety of important functions: – – Help formulate key questions that address real-world dilemmas Key questions should be devised to address important health care dilemmas, not focus on interests pertinent to researchers – – Provide context to help discern content area and applicability – – Ensure transparency in the process Whitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

11. 10 Key Informants (II) Beliefs about advantages or disadvantages of various alternative treatments are an important target for exploration with key informants – – Some beliefs will be based on long-term RCTs – – Others will be based on dogma, extrapolations from surrogate end points, pathophysiology of disease, or pharmacologic profiles of drugs Key questions addressing these areas can profoundly affect practice Whitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

12. 11 Key Informants (III) Clinicians – – Clarify (sub)populations to focus on – – Identify areas in which studies differ that can affect applicability – – Identify what contemporary practice looks like Patients – – Provide insight not usually appreciated by other stakeholders May focus more on quality of life and harms Whitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

13. 12 Public Comment (I) After key informants provide insight, a final draft of the topic refinement document is devised This document will undergo public comment – – Anyone with an opinion and access to a computer can comment Public comment is a critical final check of underlying assumptions about relevance and ensures transparency for all stakeholders Whitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr.

14. 13 Public Comment (II) All public comments that can be answered with the existing Methods Guide are answered generically All other public comments are answered individually by the EPC – – Changes to the topic refinement document are made or further explanations of why certain aspects of the topic refinement project are not being changed are laid out Revised topic refinement document and dispensation of public comments are reviewed and approved Whitlock EP. Identifying, prioritizing, and developing research topics within the Effective Healthcare Program. Rockville, MD: Agency for Healthcare Research and Quality. Methods guide for comparative effectiveness reviews; 2009 Apr..

15. 14 Topic Refinement Process: UCONN/HH EPC Example (I) Step 1: Core Leadership team reviews topic development document with title, preliminary PICOTS, brief review of what systematic reviews and studies exist, and topic refinement template – – Topic refinement on treatments for epilepsy Step 2: Core Leadership team selects a team from within the EPC that has knowledge, interest, and/or contacts within the topic area – – Lead investigator, senior scientist, research fellow identified Step 3: Selected team performs general review of the topic area, focusing on tertiary texts describing the disease process or interventions of interest and then any practice guidelines – – General review on epilepsy conducted, presented by senior scientist as a lecture with a neurologist invited

16. 15 Topic Refinement Process: UCONN/HH EPC Example (II) Step 4: Team reviews the topic development documents and continues more focused background reading Step 5: Preliminary search strategy proposed – – Approved by medical librarian and conducted Step 6: Citations reviewed, body of literature identified, data on PICOTS extracted; NO RESULTS ARE REVIEWED Step 7: Team determines PICOTS and analytic framework

17. 16 Analytic Framework White CM. AHRQ Topic Refinement Report. Submitted 7/15/2009. Intermediate Outcomes  Maximal concentrations  Minimal concentrations at steady state  Area under the curve  Average steady state concentrations  Dose needed to control seizure  Switchback rate from generic to innovator antiepileptic medication [ Pharmacokinetic data sought in epileptic patients only ] Final Health Outcomes  Mortality  Hospitalization  Office/emergency department visits  Composite of ambulance services, hospitalization, or emergency department visits for epilepsy  Health-related quality of life  Breakthrough seizures  Frequency of seizures  Secondary seizure injury (fracture, laceration, head injury, aspiration pneumonia)  Status epilepticus  Loss of driver’s license  Loss of employment  Total adverse events  Neurological adverse effects  Hypotension  Rash  Suicidal ideation  Dizziness (KQ 1) (KQ 3) (KQ 2) Patients with epilepsy using generic versus innovator antiepileptic medication

18. 17 Topic Refinement Process: UCONN/HH EPC Example (III) Step 8: Preliminary key questions posed Step 9: List of questions for key informants posed Step 10: Key informants shown the background, PICOTS, analytic framework, and preliminary key questions Step 11: After 2 weeks, the topic refinement document discussed with key informants, questions posed

19. 18 Topic Refinement Process: UCONN/HH EPC Example (IV) Step 12: Based on feedback, revised topic refinement document generated Step 13: Topic refinement document sent to AHRQ Step 14: After editorial review, posted for public comment

20. 19 Topic Refinement Process: UCONN/HH EPC Example (V) Step 15: Public comment sent to the EPC, revisions made, comments responded to Step 16: EPC sent final version to AHRQ for final approval and official posting

21. 20 The topic refinement process is devised to ensure quality, clinical relevance, and transparency Understanding the topic area is important to describing the PICOTS and devising the analytic framework When studies are evaluated, the focus is on the methods and demographics, not the results Key informants provide context and ensure relevance and transparency Key informants cannot simply include researchers in the field Public comment is a final verification of relevance and ensures transparency Key Messages