Mixing different types of research in systematic reviews Methodological Challenges for the Twenty First Century Manchester, November 2007 James Thomas EPPI-Centre, Social Science Research Unit, Institute of Education, University of London

(2) RESEARCH SYNTHESIS/ SYSTEMATIC REVIEWS A particular type or class of reviews Bridge between research and policy and practice Usually question-driven Use explicit methods, taking steps to increase trustworthiness Observational research or Research on research

(3) Typical stages of a systematic review Defining studies (inclusion and exclusion criteria) Searching exhaustively (search strategy) Describing the key features of studies Assessing their quality/weight of evidence Synthesising findings across studies Setting question and developing protocol Communication and engagement

(4) Types of questions for systematic reviews* Effectiveness Screening and diagnosis Exploring risk or protective factors Observational associations between interventions and outcomes Questions about prevalence Questions about meanings and process Methodological questions Economic questions * from Petticrew and Roberts (2005)

(5) Examples of synthesis methods Statistical meta-analysis Meta-ethnography Grounded theory Thematic analysis Realist synthesis Critical interpretive synthesis Bayesian synthesis Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A (2005) Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy 10:

(6) Why synthesise quantitative research? Meta-analysis refers to the analysis of analyses... the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature. Glass, 1976, p 3

(7) From DiCenso et al. (2002) Interventions to reduce unintended pregnancies amongst adolescents: a systematic review of randomised controlled trials. British Medical Journal 231: Does sex education increase the use of contraception amongst young people?

(8) Why synthesise qualitative research? …the full contribution of qualitative research will not be realised if individual studies merely accumulate and some kind of synthesis is not carried out…there are generalisations to be made across qualitative research studies that do not supplant the detailed findings of individual studies, but add to them Britten et al. (2002)

(9) Deriving descriptive themes

(10) Mixed method systematic reviews (1/2) Policy and practice concerns often precede, or go beyond, questions of effectiveness Different types of questions likely to be answered by different types of findings Different types of findings may require different types of synthesis

(11) Mixed method systematic reviews (1/2) Single reviews with three syntheses 1)Effect sizes from trials pooled using statistical meta-analysis 2)Findings from qualitative studies synthesised using thematic analysis 3)Synthesis 2) used to interrogate synthesis 1)

(12) An example of a mixed method review Children and healthy eating: a systematic review of barriers and facilitators* *Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, Brunton G, Kavanagh J (2003a) Children and Healthy Eating: A systematic review of barriers and facilitators. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London (The full report of this review is available at the EPPI-Centre website :.

(13) REVIEW PROCESS Searching, screening and mapping Synthesis 1: Trials (n=33) 1. Quality assessment 2. Data extraction 3. Statistical meta-analysis Synthesis 2: Qualitative studies (n=8) 1. Quality assessment 2. Data extraction 3. Thematic synthesis Synthesis 3: Trials and qualitative studies Focus narrowed to fruit &veg

(14) Synthesis 1: Statistical meta-analysis

(15) Line-by-line coding (Synthesis 2)

(16) 1) Children dont see it as their role to be interested in health. 2) Children do not see future health consequences as personally relevant or credible. 3) Fruit, vegetables and confectionary have very different meanings for children. 4) Children actively seek ways to exercise their own choices with regard to foods. 5) Children value eating as a social occasion. 6) Children recognise contradiction between what is promoted and what is provided. Children consider taste, not health, to be a key influence on their food choice Food labelled as healthy may lead children to reject them (I dont like it so it must be healthy) Buying healthy foods not seen as a legitimate use of their pocket money Synthesis 2: Thematic analysis

(17) Brand fruit and vegetables as tasty rather than healthy. Reduce health emphasis of messages Do not promote fruit and vegetables in the same way within the same intervention. Create situations for children to have ownership over their food choices. Ensure messages promoting fruit and vegetables are supported by appropriate access to fruit and vegetables 1) Children dont see it as their role to be interested in health. 2) Children do not see future health consequences as personally relevant or credible. 3) Fruit, vegetables and confectionary have very different meanings for children. 4) Children actively seek ways to exercise their own choices with regard to foods. 5) Children value eating as a social occasion. 6) Children recognise contradiction between what is promoted and what is provided. Preparing for synthesis 3

(18) Childrens viewsTrials Recommendation for interventions Good qualityOther Do not promote fruit and vegetables in the same way 00 Brand fruit and vegetables as an exciting or child- relevant product, as well as a tasty one 55 Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health 56 Synthesis 3: Across studies

(19) Increase (standardised portions per day) in vegetable intake across trials Little or no emphasis on health messages Synthesis 3: Across studies

(20) Added value of mixing methods Operationally, the method is simple, but conceptually, it is strong Integrates quantitative estimates of benefit and harm with qualitative understanding from peoples lives Facilitates a critical analysis of intervention studies from the point of view of those targeted by interventions - and vice versa Preserves the integrity of the findings of the different types of studies

(21) James Thomas EPPI-Centre SSRU 18 Woburn Square London, WC1H 0NR The methods described here can be found in: Thomas et al (2004) Integrating qualitative research with trials in systematic reviews. British Medical Journal 328: For full details of the systematic review discussed in this paper and other EPPI-Centre reviews please see our website: