What is meta-analysis? ESRC Research Methods Festival Oxford 8 th July, 2010 Professor Steven Higgins Durham University

Key points Understanding ‘effect-size’ Why do we need it? What are its limitations? What is its potential?

What is an “effect size”? Standardised way of looking at difference Different methods for calculation Odds Ratio Correlational (Pearson’s r) Standardised mean difference  Difference between control and intervention group as proportion of the dispersion of scores

Examples of Effect Sizes: ES = 0.2 “Equivalent to the difference in heights between 15 and 16 year old girls” 58% of control group below mean of experimental group Probability you could guess which group a person was in = 0.54 Change in the proportion above a given threshold: from 50% to 58% or from 75% to 81%

“Equivalent to the difference in heights between 14 and 18 year old girls” 69% of control group below mean of experimental group Probability you could guess which group a person was in = 0.60 ES = 0.5 Change in the proportion above a given threshold: from 50% to 69% or from 75% to 88%

“Equivalent to the difference in heights between 13 and 18 year old girls” 79% of control group below mean of experimental group Probability you could guess which group a person was in = 0.66 ES = 0.8 Change in the proportion above a given threshold: from 50% to 79% or from 75% to 93%

Significance versus effect size Traditional test is of statistical ‘significance’ The difference is unlikely to have occurred by chance However it may not be: Large Important, or even Educationally ‘significant’

The rationale for using effect sizes Traditional reviews focus on statistical significance testing Highly dependent on sample size Null finding does not carry the same “weight” as a significant finding Meta-analysis focuses on the direction and magnitude of the effects across studies From “Is there a difference?” to “How big is the difference?” and “How consistent is the difference?” Direction and magnitude represented by “effect size”

Meta-analysis Synthesis of quantitative data Cumulative Comparative Correlational “Surveys” educational research (Lipsey and Wilson, 2001)

Forest plots Effective way of presenting results Studies, effect sizes, confidence intervals Provides an overview of consistency of effects Summarises an overall effect (with confidence interval) Useful visual model of a meta-analysis

Anatomy of a forest plot… Studies N of study Line of no effect C.I Study effect size Pooled effect size Study effect size (with C.I.) Weighting of study in meta- analysis

Issues and challenges in meta-analysis Conceptual Reductionist - the answer is 42 Comparability - apples and oranges Atheoretical - ‘flat-earth’ Technical Heterogeneity Publication bias Methodological quality

Schulze, R. (2007) The state and the art of meta-analysis Zeitschrift fur Psychologie/ Journal of Psychology, 215 pp

RDI in Quantitative Synthesis Collaboration between the Universities of Durham, Birmingham and the Institute of Education, University of London Rob Coe, Mark Newman, James Thomas and Carole Torgerson Levels 1 and 2: Intro and practical training Durham, Edinburgh London, Belfast, York, Cardiff Level 3: expert workshops Prof Mark Lipsey, Prof Larry Hedges, Doctoral support through BERA

References, further readings and information Books and articles Borenstein, M., Hedges, L.V., Higgins, J.P.T. & Rothstein, H.R. (2009) Introduction to Meta Analysis (Statistics in Practice) Oxford: Wiley Blackwell. Chambers, E.A. (2004). An introduction to meta-analysis with articles from the Journal of Educational Research ( ). Journal of Educational Research, 98, pp Cooper, H.M. (1982) Scientific Guidelines for Conducting Integrative Research Reviews Review Of Educational Research 52; 291. *Cooper, H.M. (2009) Research Synthesis and meta-analysis: a step-by-step approach London: SAGE Publications (4th Edition). Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R.O., Hornik, R. C., Phillips, D. C., Walker, D. F., & Weiner, S. S. (1980). Toward reform of program evaluation: Aims, methods, and institutional arrangements. San Francisco, Ca.: Jossey-Bass. Glass, G.V. (2000). Meta-analysis at 25. Available at: (accessed 9/9/08) Lipsey, Mark W., and Wilson, David B. (2001). Practical Meta-Analysis. Applied Social Research Methods Series (Vol. 49). Thousand Oaks, CA: SAGE Publications. Torgerson, C. (2003) Systematic Reviews and Meta-Analysis (Continuum Research Methods) London: Continuum Press. Websites What is an effect size?, by Rob Coe: The meta-analysis of research studies: The Meta-Analysis Unit, University of Murcia: The PsychWiki: Meta-analysis: Meta-Analysis in Educational Research: