Meta-analysis, systematic reviews and research syntheses © LOUIS COHEN, LAWRENCE MANION AND KEITH MORRISON © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

STRUCTURE OF THE CHAPTER Meta-analysis Systematic reviews Methodologically inclusive research syntheses © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

META-ANALYSIS The analysis of other analyses. Uses data from randomized controlled trials. Aggregates and combines the results of comparable studies into a coherent account to discover main effects. Often uses statistical processes. Looks at effect size, not only statistical significance. Combines the results of small-scale studies. Uses transparent means to draw conclusions. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

META-ANALYSIS AND RESEARCH SYNTHESIS Meta-analysis involves rigorous statistical integration of findings reported across a number of primary research studies. Meta-analysts employ explicit protocols to enhance consistency Meta-analyses facilitate comparisons across quantitative studies by bringing them on a common metric (‘effect size’), but they are unsuitable for synthesizing qualitative research. Research syntheses are often used for qualitative research. ‘Research synthesis’ can draw together studies in a range of styles into a single review, seeking aims to produce new knowledge by making explicit connections and relations that were previously invisible between individual reports. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

WORKING WITH META-ANALYSIS Formulating a problem Searching for relevant literature Extracting relevant information from selected studies Integrating findings across studies Presenting the findings in a report © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

AN EIGHT-STEP PROCESS FOR META-ANALYSIS 1 Identify the variables for focus (dependent and independent) 2 Identify all the studies which feature the variables in which the researcher is interested and which meet selection criteria 3 Code each study for those characteristics that might be predictors of outcomes and effect sizes (e.g. age, gender, ethnicity) 4 Estimate the effect sizes (i.e. a common metric) for each pair of variables (dependent and independent variable), weighting the effect-size by the sample size © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

AN EIGHT-STEP PROCESS FOR META-ANALYSIS 5 Calculate the mean and the standard deviation of effect-sizes across the studies, i.e. the variance across the studies 6 Determine the effects of sampling errors, measurement errors and range of restriction 7 If a large proportion of the variance is attributable to the issues in Step 6, then the average effect-size may be an accurate estimate of relationships between variables 8 If a large proportion of the variance is not attributable to the issues in Step 6, review those characteristics of interest which correlate with the study effects © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

EFFECT SIZE Effect size is calculated as: © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

CONCERNS ABOUT META-ANALYSIS Lacks comprehensiveness, as they make selective use of subsets of studies. Misrepresents research findings. Relies too heavily on significance tests. Understates effect sizes as a way of supporting hypotheses. Fails to recognize that sampling error can play a part in creating variations in findings. Overlooks differing/conflicting research findings. Published research is favoured over unpublished research. Interaction effects overlooked in favour of main effects. Unclear how it differentiated between ‘good’ and ‘bad’ research, e.g. between rigorous and poorly constructed research. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

CONCERNS ABOUT META-ANALYSIS Difficult to draw logical conclusions from studies that use different interventions, measurements, definitions of variables, and participants. Results from poorly designed studies take their place alongside results from higher-quality studies. Multiple results from a single study are used, making the overall meta-analysis appear more reliable than it is, since the results are not independent. Dangerous consequences because its apparent objectivity and precision may disguise procedural invalidity in the studies. Risks trying to synthesize studies which are insufficiently similar to each other to permit this. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

CONCERNS ABOUT META-ANALYSIS Fails to examine critically the evidence, methods and conclusions of previous reviews. Overlooks the extent to which findings from research are mediated by sample characteristics. Overlooks the importance of intervening variables in research. Unreplicable because procedures for integrating the research findings not made explicit. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS Include studies that are not solely randomized controlled trials. Apply explicit protocols and rigorous criteria for searching for and selecting relevant primary, usually empirical, studies, to remove bias; standards for acceptable methodological rigour; relevance to the topic in question; scope of the studies included; adoption of a consistent and clearly stated approach to combining information from across different studies; drawing careful, relevant conclusions and recommendations. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS Combine qualitative and quantitative studies. Follow a standard set of stages. Are accountable, able to be replicated and updated. Must be relevant and useful to users. Must answer specific research questions. Are evidence-based. © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS Clear, transparent criteria for: inclusion and exclusion of studies standards for acceptable methodological rigour relevance to the topic in question scope of the studies included team approaches to reviewing in order to reduce bias adoption of a consistent and clearly stated approach to combining information from across different studies careful, relevant conclusions and recommendations © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

SEQUENCE FOR CONDUCTING SYSTEMATIC REVIEWS 1 Formulate the problem and kinds of relevant research evidence to answer the research question or hypothesis 2 Examine the research designs in the studies used, the treatment of the main effects, the kind of research, the conceptual relevance of the studies, i.e. screen the studies. 3 Search the literature 4 Gather information from studies (including developing and using a coding guide, identifying predictor and outcome variables, research designs used, sampling, context, statistics used, effect sizes) © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

SEQUENCE FOR CONDUCTING SYSTEMATIC REVIEWS 5 Evaluate the quality of studies (decide which studies to include and exclude, identify problems in the research studies used, evaluate the suitability of the research design for the research synthesis, evaluate the quality and rigour of the research) 6 Analyze and integrate the outcomes of studies 7 Interpret the evidence 8 Present the results © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

METHODOLOGICALLY INCLUSIVE RESEARCH SYNTHESES Suri’s (2014) three general guiding principles for a quality research synthesis Informed subjectivity and reflexivity Purposefully informed selective inclusivity Audience-appropriate transparency © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

SURI’S (2014) SIX PHASES FOR CONDUCTING A RESEARCH SYNTHESIS Identify an appropriate epistemological orientation 2 Identify an appropriate purpose 3 Search for relevant literature 4 Evaluate, interpret and distil evidence from selected reports 5 Construct connected understandings 6 Communicate with an audience © 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors