1. META-ANALYSIS, RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON

2. STRUCTURE OF THE CHAPTER Evidence-based research Meta-analysis Research syntheses and systematic reviews

3. EVIDENCE-BASED RESEARCH Comes from randomized controlled trials and research syntheses Based on accumulated evidence of ‘what works’ Evidence-based approaches are necessary in order to: –challenge the imposition of unproven practices –solve problems and avoid harmful procedures –create improvement that leads to more effective learning Looks at effect size, not only statistical significance

4. META-ANALYSIS The analysis of other analyses Uses data from randomized controlled trials Aggregates and combines the results of comparable studies into a coherent account to discover main effects Often uses statistical processes Looks at effect size, not only statistical significance Combines the results of small-scale studies Uses transparent means to draw conclusions

5. CONCERNS ABOUT META-ANALYSIS Lacks comprehensiveness, as it makes selective use of subsets of studies. Misrepresents research findings. Relies too heavily on significance tests. Understates effect sizes as a way of supporting hypotheses. Fails to recognize that sampling error can play a part in creating variations in findings. Overlooks differing/conflicting research findings. Published research is favoured over unpublished research. Interaction effects overlooked in favour of main effects. Unclear how it differentiates between ‘good’ and ‘bad’ research, e.g. between rigorous and poorly constructed research.

6. CONCERNS ABOUT META-ANALYSIS Difficult to draw logical conclusions from studies that use different interventions, measurements, definitions of variables, and participants. Results from poorly designed studies take their place alongside results from higher quality studies. Multiple results from a single study are used, making the overall meta-analysis appear more reliable than it is, since the results are not independent. Dangerous consequences because its apparent objectivity and precision may disguise procedural invalidity in the studies. Risks trying to synthesize studies which are insufficiently similar to each other to permit this.

7. CONCERNS ABOUT META-ANALYSIS Fails to examine critically the evidence, methods and conclusions of previous reviews Overlooks the extent to which findings from research are mediated by sample characteristics Overlooks the importance of intervening variables in research Unreplicable because procedures for integrating the research findings not made explicit

8. AN EIGHT-STEP PROCESS FOR META-ANALYSIS Step 1: Identify the variables for focus (dependent and independent). Step 2: Identify all the studies which feature the variables in which the researcher is interested. Step 3: Code each study for those characteristics that might be predictors of outcomes and effect sizes (e.g. age, gender, ethnicity). Step 4: Estimate the effect sizes for each pair of variables (dependent and independent variable), weighting the effect-size by the sample size.

9. AN EIGHT-STEP PROCESS FOR META-ANALYSIS Step 5: Calculate the mean and the standard deviation of effect-sizes across the studies, i.e. the variance across the studies. Step 6: Determine the effects of sampling errors, measurement errors and range of restriction. Step 7: If a large proportion of the variance is attributable to the issues in Step 6, then the average effect-size may be an accurate estimate of relationships between variables. Step 8: If a large proportion of the variance is not attributable to the issues in Step 6, review those characteristics of interest which correlate with the study effects.

10. EFFECT SIZE The effect size is calculated as:

11. RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS Include studies that are not solely randomized controlled trials. Apply explicit protocols and rigorous criteria for searching for and selecting relevant primary, usually empirical studies, and to remove bias. Combine qualitative and quantitative studies. Follow a standard set of stages. Are accountable, able to be replicated and updated. Must be relevant and useful to users. Must answer specific research questions. Are evidence-based.

12. RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS Clear, transparent criteria for: –inclusion and exclusion of studies –standards for acceptable methodological rigour –relevance to the topic in question –scope of the studies included –team approaches to reviewing in order to reduce bias –adoption of a consistent and clearly stated approach to combining information from across different studies –careful, relevant conclusions and recommendations

13. SEQUENCE FOR CONDUCTING SYSTEMATIC REVIEWS Formulate the problem and kinds of relevant research evidence to answer the research question or hypothesis. Examine the research designs in the studies used, the treatment of the main effects, the kind of research, the conceptual relevance of the studies, i.e. screen the studies. Search the literature. Gather information from studies (including developing and using a coding guide, identifying predictor and outcome variables, research designs used, sampling, context, statistics used, effect sizes).

14. SEQUENCE FOR CONDUCTING SYSTEMATIC REVIEWS Evaluate the quality of studies (decide which studies to include and exclude, identify problems in the research studies used, evaluate the suitability of the research design for the research synthesis, evaluate the quality and rigour of the research). Analyze and integrate the outcomes of studies. Interpret the evidence. Present the results.