1. Issues and challenges to scoping and focusing the question ESQUIRE Qualitative Systematic Review Workshop University of Sheffield 8 September 2011 Janet Harris Cochrane Qualitative Research Methods Group

2. Session aims To identify the issues and challenges in formulating different types of qualitative questions for qualitative evidence synthesis

3. The issue: What question? The challenges Utility Relevance Timeliness Explanatory value

4. How effectiveness questions are developed Refine definition of problem Identify possible causes and effects Identify problem Test solutions Margerunm-Leys http://www- personal.umich.edu/~jmargeru/conceptmap/types.htm http://www- personal.umich.edu/~jmargeru/conceptmap/types.htm Clinical expertise + research evidence are used to:

5. Effectiveness question: In adults with Low Back Pain does provision of individual education reduce pain, improve functioning and promote return to work? P: Adults with back pain (BP) O: Reduce pain, improve functioning, return to work I: Information on how to self-manage BP Assumed solution : Patients need written and/or verbal education delivered by a health professional Problem and solution (as defined by the researchers)

6. Possible results from a search using an informing question Problem Definition(s) Effects Causes Informing question (Background question) Single studies / evidence syntheses identifying factors that might make us rethink assumptions

7. Explore assumptions with informing questions P: Important characteristics relating to I and O O: Factors that may dilute potential effectiveness I: Factors that affect implementation Informing question (Background question) Refine P Consider subgroup analysis Consistency of implementation Such as participants’ views

8. Back pain patients’ experiences of receiving education for self management

9. Informing questions may identify theories Theories are explanations for a phenomenon or an event e.g. Why something happens in relation to another event They can be grand theories, they can be middle range theories, or they can be single-case theories Middle range theory goes beyond individual studies, providing a cross cutting explanation for patterns that are seen across settings and different types of people Whenever possible, interventions testing effectiveness should be based on theory

10. Issues raised from informing questions There may be ‘essential ingredients’ that need to be included in interventions For example, the search on back pain education identified an issue with appropriateness and relevance of information Informing questions need to produce convincing evidence about these ingredients to justify changing an effectiveness question. How much evidence is needed to change the original effectiveness question?

11. Revising the effectiveness question: An example In adults with Back Pain does provision of individual education by a health professional reduce pain, improve functioning and promote return to work? Revised effectiveness question: P = adults with Low Back Pain I = information explained by health professional C= information explained by a health professional with additional peer support O = reduce pain, improve functioning and promote return to work Additional ingredients that theoretically increase relevance

12. Issues with the refined question Is the quality of reporting for effectiveness studies adequate? For example, do the interventions describe how information is explained by the health professionals? Is the amount of time for the explanation documented? Does the comparison describe the type of peer support offered? Qualitative evidence can indicate how a question should be refined, but if effectiveness studies do not report the relevant information then the end product could be an empty review

13. Enhancing questions to explain the results of effectiveness reviews Monitors pathway from intervention to outcomes May highlight problems with Quan studies Enhancing question Within-study data Parallel- study data Similar- study data

14. Enhancing questions: Issues and challenges Adequate data within the study about PICO characteristics? Adequate data in embedded qualitative studies? Adequate data in parallel sibling studies? Can ‘similar’ qualitative studies be found? What does ‘similar’ mean – across embedded, sibling, and independently conducted studies? Comparability of data - how to synthesize data across studies?

15. Utility of findings: Used to critique design and PICO in included studies; subgroup analysis of theory-based interventions.. Extending question: Can explore an issue related to population, intervention, comparison or outcome Cause: Low awareness of educational approaches for behaviour change Extending question: How do people make sense of and use Back Pain information? What educational theories promote better use of BP information? Problem noted: Lack of theory-based education on how to self-manage BP Negative effect: Increased utilization of services and absenteeism

16. Extending questions: Issues and challenges Extending questions require a separate qualitative evidence synthesis, either conducted within or alongside an effectiveness review. Funding needs to be justified by considering: Timeliness: Is the information perceived to be critical to using the results of the effectiveness review? Relevance: Is the issue important to funders, policymakers and effectiveness reviewers ? Utility: did the scoping process specifically note issues that need to be informed by qualitative research? Explanatory value: Will the qualitative synthesis contribute to explaining the outcomes of effectiveness studies and/or the broader context that influences effectiveness?

17. Extending questions: Issues and challenges Extending questions require coordination of a team including qualitative evidence synthesis (QES) members working alongside effectiveness review (ER) members The parallel processes need to be considered because Findings from the QES may identify quality issues in the effectiveness studies. At what point are these quality issues brought into the effectiveness review – at abstract review? Development of inclusion criteria? Critical appraisal? Metaanalysis? Reporting QES process and findings: Approaches are still being developed. At the same time, the software for Cochrane ERs – RevMan – does not yet have the facility for including Extending reviews.

18. Effectiveness review Supplementing questions Supplementing review: Conducted independently Before or after an ER On a wider issue than effectiveness Effectiveness review

19. Supplementing question issues Supplementing questions: Used in stand alone reviews to produce information on a wider issue Does peer support help people with chronic conditions? Does peer support promote health literacy (help people to understand and use health information) Issues Timeliness: Will the theory be used over time? Relevance: Does the theory address issues noted by ER authors and those trying to implement review results (External validity) Utility: will the theory enable development of better effectiveness studies? Explanatory value: Can the theory be used to explain some of the differences in effect within and across existing ERs? Can the theory be used to inform development of future ERs?

20. Summary: What guides qualitative question formulation The need to ensure that effectiveness questions are informed by existing theory on what works (Informing questions) The need to provide real-time explanations for effectiveness studies (Enhancing questions) The need to further explore issues identified in the review by reviewing a broader range of research on effectiveness (Extending question) The need to explore the context surrounding a condition or an intervention, including situations and experiences that may not be directly related to effectiveness (Supplementing question)

21. Is it worth it? When scoping and focusing the question, we need to ask: Does this question pass the TRUE test? Timely: Will the question still be relevant when the review is finally completed? Relevant: How will this question contribute information that is relevant to health services policymakers? Useful: Which of (the many) findings from the qualitative scoping will be useful in focusing the review and/or explaining the results? Explanatory value: Does the synthesis contribute to explaining the outcomes of effectiveness studies and/or the broader context that influences effectiveness?

22. Further challenges Team skills Team coordination Informed and transparent decisions about when and how to useQES/ER data