1. Wim Van den Noortgate Katholieke Universiteit Leuven, Belgium Belgian Campbell Group Wim.VandenNoortgate@kuleuven-kortrijk.be Workshop systematic reviews Leuven June 4-6, 2012

2. 1. Introducing meta-analysis for group designed studies 2. Effect sizes 3. Meta-analysis of studies with other designs

4. Example: association between gender and math  M = 8 ;  F = 8.5 ;  M =  F = 1.5  M  F

5. Standardized mean difference (Cohen, 1969): Estimated by its sample counterpart: ‘True’ effect size ‘Observed’ effect size 0.20 = small effect 0.50 = moderate effect 0.80 = large effect

6. sMsM sFsF p (2-sided) g 8.109.341.55 0.015 (*)0.80 Example:  M = 8 ;  F = 8.5 ;  M =  F = 1.5 => δ = 0.33 n M =n F = 20

7. sMsM sFsF p (2-sided) g 8.10 7.60 9.34 7.59 1.55 1.23 1.55 1.47 0.015 (*) 0.98 0.80 -0.0069 Example:  M = 8 ;  F = 8.5 ;  M =  F = 1.5 => δ = 0.33

8. sMsM sFsF p (2-sided) g 8.10 7.60 7.96 7.70 8.17 7.86 8.19 8.11 7.86 8.34 9.34 7.59 8.81 8.25 8.81 7.93 8.15 7.94 8.53 1.55 1.23 1.38 1.49 1.76 1.24 1.79 1.76 1.89 1.39 1.55 1.47 1.59 1.65 1.33 1.58 1.78 1.97 1.64 1.79 0.015 (*) 0.98 0.078 0.28 0.87 0.040 (*) 0.65 0.95 0.89 0.71 0.80 -0.0069 0.57 0.35 0.053 0.67 -0.14 0.020 0.042 0.12 Example:  M = 8 ;  F = 8.5 ;  M =  F = 1.5 => δ = 0.33

9. δ g

10.  95 % confidence interval:

12. Suppose simulated data are data from 10 studies, being replications of each other: Vote-counting procedure?

13. Suppose simulated data are data from 10 studies, being replications of each other:

15. 1. Observed effect sizes may be positive, negative, small, moderate and large. 2. CI relatively large 3. 0 often included in confidence intervals 4. Combined effect size close to population effect size (averaging out the noise) 5. CI relatively small (higher accuracy) 6. 0 not included in confidence interval (higher power)

16. Meta-analysis: Gene Glass (Educational Researcher, 1976, p.3): “Meta-analysis refers to the analysis of analyses”

17. δ g n M = n F = 100 δ g n M = n F = 20

19. ( Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97. ) Study Weeks prior contact gjgj 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. Rosenthal et al. (1974) Conn et al. (1968) Jose & Cody (1971) Pellegrini & Hicks (1972) Evans & Rosenthal (1969) Fielder et al. (1971) Claiborn (1969) Kester & Letchworth (1972) Maxwell (1970) Carter (1970) Flowers (1966) Keshock (1970) Henrickson (1970) Fine (1972) Greiger (1970) Rosenthal & Jacobson (1968) Fleming & Anttonen (1971) Ginsburg (1970) 23300333010012331232330033301001233123 0.03 0.12 -0.14 1.18 0.26 -0.06 -0.02 -0.32 0.27 0.80 0.54 0.18 -0.02 0.23 -0.18 -0.06 0.30 0.07 -0.07 0.13 0.15 0.17 0.37 0.10 0.22 0.16 0.25 0.30 0.22 0.29 0.16 0.17 0.14 0.09 0.17

20. Mean effect: 0.060, p=.10

21. ( Keren, R., & Chan, E. (2002). A meta-analysis of randomized, controlled trials comparing short- and long-course antibiotic therapy for urinary tract infections in children. Pediatrics, 109, e70. ) StudyYearSample Size RR (95% CI) Bailey and Abbott1978101.33 (0.17–10.25) Khan et al1981160.20 (0.01–3.61) Stahl et al1984261.20 (0.34–4.28) Fine and Jacobson1985312.34 (0.53–10.30) Gaudreault et al1992401.00 (0.02–48.09) Pitt et al1982422.50 (0.11–58.06) Helin1984432.53 (0.25–25.81) Grimwood et al1988452.80 (0.65–12.02) Avner et al1983494.69 (1.13–19.51) Lohr et al1981501.28 (0.23–7.00) Nolan et al19899010.45 (1.40–78.31) Copenhagen19912641.50 (0.68–3.32)

22. Note:

23. Combined RR = 1.94 (95% CI: 1.19–3.15)

25. Example: testing the difference in the size of tumors in an experimental and a control group What would you conclude if  p =.11?  p <.0001?

26. Misconceptions: ◦ failure to reject the null hypothesis implies no effect ◦ a statistically significant p-value implies a large effect

27. Test of Significance = Size of Effect × Size of Study Rosenthal, R. (1991). Meta-analytic procedures for social research. Newbury Park, CA: Sage

28.  Before being combined in a meta-analysis, findings from primary studies are summarized to a measure of effect size  There are several possible effect size indices: e.g. ◦ Two continuous variables: the correlation coefficient ◦ One continuous, one dichotomous: the standardized mean difference ◦ Two dichotomous: the odds ratio, relative risk, …  To allow comparison over studies, a common measure is used, often a standardized one  In a meta-analysis, effect size measures are compared and combined

29. Final exam Predictive test10 1130 (87 %) 20 (13 %) 150 (100 %) 030 (60 %) 20 (40%) 50 (100 %) 16040200 1. Risk difference:.87-.60 =.27 2. Relative risk:.87/.60 = 1.45 3. Phi: (130 x 20 – 20 x 30)/sqrt (150 x 50 x 160 x 40) = 0.29 4. Odds ratio: (130 x 20 / 20 x 30) = 4.33

30. ◦ direct calculation based on means and standard deviations ◦ algebraically equivalent formulas (t-test) ◦ exact probability value for a t-test ◦ approximations based on continuous data (correlation coefficient) ◦ Results of one way ANOVA with 3 or more groups ◦ Results of ANCOVA ◦ Results of multiple regression analysis ◦ approximations based on dichotomous data Great Good Poor

31. Direction Calculation Method

32. Calculation based on test statistics exact p-values from a t-test or F-ratio can be converted into t-value and the above formula applied 

33. Calculation based on other effect size measures Other conversion formulae: Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage. Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper, & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 231-244). New York: Russell Sage Foundation.

35. 35

36. 36

37. 37

38. 38 11 00

39. 39

40. 40

41. 41

42. o Expressing effects in (quasi-)experimental studies ◦ Comparing experimental & control groups ◦ Comparing one group under several conditions o Expressing association in non-experimental studies ◦ Comparing existing groups (e.g., male vs. female) ◦ Expressing association between continuous variables (e.g., relation between class size and performance) o Describing one single variable (e.g., prevalence rates, means, …) 42

43. Check the internal validity of the design! (are there confounding variables?) Pay attention with the interpretation of your results!