K U Leuven Systematic Review Workshop 4-6 June 2012 Using qualitative research findings to inform question development in systematic reviews Janet Harris Cochrane Qualitative Research Methods Group K U Leuven Systematic Review Workshop 4-6 June 2012

Session aims To demonstrate how and when qualitative research findings can be used to inform the development of questions for reviews of effectiveness To identify the issues and challenges in formulating different types of qualitative questions for qualitative evidence synthesis

Developing questions for systematic reviews: When should qualitative research be considered? Qualitative research should be considered if your review question is about a complex intervention.

What is a complex intervention? Complex interventions in health care, whether therapeutic or preventative, comprise a number of separate elements which seem essential to the proper functioning of the interventions although the 'active ingredient' of the intervention that is effective is difficult to specify… Complex interventions are built up from a number of components, which may act both independently and interdependently. The components usually include behaviors, parameters of behaviors (e.g. frequency, timing), and methods of organizing and delivering those behaviors (e.g. type(s) of practitioner, setting and location).’ Medical Research Council: A framework for development and evaluation of complex interventions to improve health. London: Medical Research Council; 2000.

Most effectiveness questions identify a limited number of causes and effects Refine definition of problem Identify problem Test solutions Identify possible causes and effects Margerunm-Leys http://www-personal.umich.edu/~jmargeru/conceptmap/types.htm

The linear cause and effect model is reproduced in the effectiveness review Effectiveness reviews are conducted when a body of research has accumulated in a particular field, and findings need to be aggregated there is uncertainty about a common strategy or treatment there is uncertainty about relative effectiveness of two or more commonly offered interventions so a comparison is needed The first step in developing the review protocol is a scoping search to determine how much literature exists and what type of research has been conducted

Scoping searches A scoping search for a systematic review “should typically search for existing reviews and major trials and other studies. Results from these searches can refocus or focus the review.... This process can involve several iterations. Scoping searches are also used to estimate the size of the literature and by extension the cost of the review.” Centre for Reviews and Dissemination http://www.york.ac.uk/inst/crd/revsint.htm

Limiting your review to existing research is sometimes risky Published research reflects The predominant interest of the funder – the funder’s definition of the problem, the cause and the effect Responses from researchers to the requirements of the funder Researchers’ and funders’ formulation of the problem may be limited to a particular perspective, neglecting factors in the surrounding environment or characteristics of the participants that can affect outcomes The research question will also be bounded by the particular discipline of the researchers. Many health problems, for example, are investigated within the confines of a randomised controlled trial conducted under artificial conditions. As a result, it is difficult to successfully transfer the trial intervention to other contexts

Problems with replication Successful replication of the results of a trial depends on the amount of contextual information that was captured during the original trial In order to promote transferability, we need to know Important characteristics of participants Important characteristics of the setting Important characteristics of the people who delivered the intervention If these characteristics are similar to our situation, then we can be reasonably confident that we can obtain a similar effect

Collecting adequate data to explain the effects of an RCT Some studies collect qualitative information either within the study. This is called a ‘process evaluation’ In other cases, qualitative research is conducted alongside an RCT When looking for qualitative research to inform a review question, always look for this data first It is directly related to the trials that are going to be included in your review

Qualitative research that enhances a trial Enhancing question explores issues of patient acceptability, the process of implementing the intervention, and patient-defined outcomes Qualitative data may be collected Within the trial In a Parallel study In a Similar study

Enhancing data: Issues and challenges Adequate data within the study about PICO characteristics? Adequate data in embedded qualitative studies? Adequate data in parallel sibling studies? Can ‘similar’ qualitative studies be found? What does ‘similar’ mean – across embedded, sibling, and independently conducted studies? Comparability of data - how to synthesize data across studies?

What if there aren’t any process evaluations or parallel qualitative studies? We’ll use an example to illustrate how this can be done For people who suffer from low back pain, are back schools effective in reducing pain, improving functioning, and promoting return to work? A search for enhancing data showed that there were no qualitative studies on patients’ experiences of participating in back schools So, a qualitative scoping search, in areas related to the phenomenon of interest, needs to be conducted to inform development of the question

Back pain schools for non-specific back pain (Heymanns et al, 2010) This Cochrane review was unable to perform a meta analysis due to the heterogeneity of the studies included in the review. ‘A back school was defined as consisting of an educational and skills acquisition program, including exercises, in which all lessons were given to groups of patients and supervised by a paramedical therapist or medical specialist. Additional interventions were allowed.’ What are the possible sources of heterogeneity in this definition?

What do we know about possible sources of heterogeneity that could influence the results? The descriptions of participating patients and the content of interventions was ‘thin’ or non-existent ‘It is essential to be not only informed about effectiveness, but also about the characteristics of the included patients and the content of the programs, to determine the clinical relevance of the studies The majority of RCTs did not score sufficiently on the questions, especially Questions 2 and 4, which described the intervention and treatment settings and the clinically relevant effect size.’

Problems with poor reporting ‘RCTs either reported briefly about the content of the intervention or failed to report essential information about the type, intensity or performance of the exercises. ‘ If the components of the intervention are not described, then we cannot know which components were responsible for the effect

How important are the variations ‘Also apparent were the widely variations in the content and components of the interventions. This may explain the differences in interpretation of the items between the two authors, reflected by the disagreement score of 34% Can variations be statistically controlled? Or should heterogeneity be dealt with by ‘lumping and splitting’? These questions are especially important when dealing with complex interventions

How do we define important components of a complex intervention? Qualitative research can help to identify what is important in terms of Implementing the intervention Practitioners attitudes toward the condition and the treatment or prevention strategy Patients’ attitudes toward the intervention and experiences of the condition

Effectiveness question: In adults with Low Back Pain does provision of education via Back Schools reduce pain, improve functioning and promote return to work? P: Adults with back pain (BP) Assumed solution: Patients need written and/or verbal education and support delivered by a health professional Problem and solution (as defined by the researchers) I: Back School containing Information and exercises to promote self-management O: Reduce pain, improve functioning, return to work

Explore assumptions with informing questions S: Settings: in work; outside work Consistency of implementation; peer support Informing questions (Qualitative questions) P: Adults with LBP Relevant characteristics, for example literacy I: Receive information on managing LBP from doctor; from physiotherapist; from community health worker Who delivers; how delivered; credibility and relevance to everyday life

A qualitative informing question What are adults’ experiences of receiving information on how to manage Low Back Pain in different settings, from different providers?

Back pain patients’ experiences of receiving education for self management Can these quotes inform a review? Doesn’t apply to me = P Hard to read, difficult to understand = P and/or I & C – content of information, how delivered Time to discuss= C compare delivery e.g. Written information alone versus written information + discussion

Patients’ experiences of receiving back pain information in a physiotherapy-led back school ‘It is important to be taken seriously.’ ‘Before, I had pain so I would lie down. Then I had more pain. But now after the course, when I have pain, I no longer lie down. I see that I must continue to go on moving and the pain will be gone. if I lie down the pain will comeback. Before I was ill, now I am not.’   Here in the physiotherapy clinic, you can bring up all sorts of weird things, thoughts and theories because there’s no time limit’. ‘My husband wasn’t believed. [at the hospital].’

Quotes from patients with chronic pain ‘The doctor gave me some medication and told me to ‘pop along’ to the gym. Well, telling me what to do doesn’t mean I can just go and do it!’ ‘I’ve been stuck inside for 2 years.’ ‘I thought I was disabled.’ ‘[The Health Trainer] didn’t tell me what to do. I thought about what I’d like to be able to do, and she helped me figure out how to do it.’

Informing questions may identify theories Theories are explanations for a phenomenon or an event e.g. Why something happens in relation to another event They can be grand theories, they can be middle range theories, or they can be single-case theories Middle range theory goes beyond individual studies, providing a cross cutting explanation for patterns that are seen across settings and different types of people Whenever possible, interventions testing effectiveness should be based on theory

Emerging theory from qualitative research Sender: People need to receive health information from ‘someone like them’ e.g. a peer, someone who understands their situation and has something in common with them Message: People need an opportunity to discuss whether the health information applies to them. Time to reflect and opportunities to discuss are critical. Receiver: People need to accept that their condition is chronic, understand that a passive approach to treatment will not be effective, and be motivated to actively manage their own condition (Michie, 2008)

Can theories from qualitative research be used to inform the question? There may be ‘essential ingredients’ that need to be included in interventions For example, the search on back pain education identified an issue with appropriateness and relevance of information Informing questions need to produce convincing evidence about these ingredients to justify changing an effectiveness question. How much evidence is needed to change the original effectiveness question?

Revising the effectiveness question: An example Additional ingredients that theoretically increase relevance In adults with Back Pain does information plus peer support reduce pain, improve functioning and promote return to work? P = adults with Low Back Pain I = information explained using peer support C= information delivered by health professional O = reduce pain, improve functioning and promote return to work

Patient-defined outcomes? Most of the research defines outcomes as reduced pain, improved functioning and promoting return to work Qualitative research with patients who have chronic pain indicates that they define important outcomes as Better able to manage pain by recognising onset and timely use of medication – patients note that this is not the same as reducing pain Ability to manage some of the everyday things that I did before – this is not the same as improved physical functioning Able to participate in social activities, volunteer activities Able to make hob adjustments – this is different from returning to work The currently used biopsychosocial outcomes may not be measuring what is important

If you use a refined, theory-based question, what will you find? Is the quality of reporting for effectiveness studies adequate? For example, do the interventions describe how information is explained by the health professionals? Is the amount of time for the explanation documented? Does the comparison describe the type of peer support offered? Although there is a move toward requiring theory-based interventions in RCTs, this is a relatively new phenomenon Qualitative evidence can indicate how a question should be refined, but if effectiveness studies do not report the relevant information then the end product could be an empty review

Decisions about when to use qualitative research to refine a review question Weight of qualitative evidence Qualitative evidence synthesis? Large number of qualitative studies? Convergence of theory? Cross-paradigm evidence Qualitative research in more than one discipline produces complimentary findings In the Back Pain example, qualitative research in health psychology, chronic pain, peer support, and health literacy are all finding similar factors that promote self management Quality of reporting in the effectiveness literature Do effectiveness studies document any of the characteristics that qualitative research deemed important?

Using qualitative data to inform review questions: Summary of the process Search for enhancing studies – qualitative studies conducted within or alongside trials Search for ‘similar studies’ that are directly relevant to explaining the intervention If these studies don not exist, or the evidence is thin, consider searching on phenomena related to the components of the complex intervention If there is qualitative data on various components, what is the weight of the qualitative evidence? Can the qualitative evidence be used to further refine the PICO?

References Daykin AR & Richardson B (2004) Physiotherapists’ pain beliefs and their influence on the management of patients with chronic low back pain. Spine 29: 783-795. HeymansMW, van TulderMW, Esmail R, Bombardier C, Koes BW. Back schools for non-specific low-back pain.. Cochrane Database of Systematic Reviews 2004, Issue 4. Art. No.: CD000261. DOI: 10.1002/14651858.CD000261.pub2. Medical Research Council: A framework for development and evaluation of complex interventions to improve health. London: Medical Research Council; 2000. Michie S et al (2008) Walker J, Holloway I, Sofaer B. (1999) In the system: the lived experience of chronic back pain from the perspectives of those seeking help from pain clinics. Pain 80, 621.