1. Texas A&M University SUMMER STATISTICS INSTITUTE June 2009 Victor L. Willson, Instructor

2. TOPIC AREAS Background Research focus for meta-analysis Finding studies Coding studies Computing effect sizes Effect size distribution Mediators Moderators Report-writing Current issues

3. Background Purposes Historical Meta-analysis as survey research Strengths Weaknesses

4. Purposes for Meta-Analysis Cumulate findings of studies on a particular topic Examine homogeneity of outcomes Estimate effects of independent variables on outcomes in a standardized format – Evaluate moderator and mediator effects on outcomes – Differentiate different types or classes of outcome effects

5. Historical background Criticism of traditional narrative reviews of research Exasperation in social sciences with constructs measured different ways in terms of determining consistencies Need to formulate theoretical relationships based on many studies

6. History part 2 Early 1970s efforts focused on significance testing and “vote counts” of significance Glass (1976) presented a method he called “meta- anlaysis” in Am. Ed. Research Assn. presidential address Others proposed related methods, but Glass and colleagues developed the most widely used approach (Glass, McGaw, & Smith, 1981)

7. Meta-Analysis as Survey Research Research articles as unit of focus Population defined – Conditions for inclusion of articles Data requirements needed for inclusion Completeness of data available in article or estimable Publication sources available, selected Sample vs. population acquisition – Availability of publications and cost – Time to acquisition (how long to wait for retrieval)

8. Strengths of Meta-Analysis Definition of effect and effect size beyond “significant or not” Focus on selection threats in traditional reviews (bias in selection of articles for review) Systematic consideration of potential mediators and moderators of effects Data organization of articles for public review

9. Weaknesses of Meta-Analysis Methodologically sophisticated and expensive Potential ignoring of contextual effects not easily quantified; eg. historical/environmental placement of research Potential improper mixing of studies Averages hiding important subgroupings Improperly weighting studies with different methodological strength/rigor

10. Research focus for meta-analysis Defining and delineating the construct Determining a research outlet Meta-Analysis as an interactive, developing process

11. Recent Criticism Suri & Clarke (2009) : Advancements in Research Synthesis Methods: From a Methodologically Inclusive Perspective (Review of Educational Research, pp. 395-430) They propose 6 overlapping approaches: 1. Statistical research syntheses (eg. meta-analysis) 2. Systematic reviews 3. Qualitative research syntheses 4. Qualitative syntheses of qualitative and quantitative research 5. Critical impetus in reviewing research 6. Exemplary syntheses

12. Some critical comments on Suri & Clarke (2009) Systematic reviews- original Glass criticisms hold: what is the basis for inclusion and exclusion; why are certain articles privileged? Qualitative research syntheses- how can these be done with situated contexts, small samples, environmentally- developed variables, sources, etc.? Will there be a review for every reader, or for every researcher? Same limitation as all qual research Qual syntheses of quant and qual research- potentially doable, with an alternating order: qual first to focus emphases in the quant analysis, or quant first to be validated with the qual studies of particular environments and populations- do they fit/match in reasonable ways? Critical impetus- code words for critical theory/Marxist etc. Answer is already known, why do the research? Exemplary syntheses- what is the purpose?

13. Defining and Delineating the Research Topic Outcome construct definition – Importance to the field to know what has been learned – How big is it? How many potential studies? – Conduct preliminary searches using various databases Refining the construct – How much resource is available? Eg. 1000 studies = 2-3 years work – Are there specific sub-constructs more important than others? Select them or one of them – Are there time-limitations (no studies before 19xx) – Are there too few studies for the given construct, should it be broadened? Too few-> less than 10?

14. Defining and Delineating the Research Topic What is the typical research approach for the topic area? All quantitative All qualitative Mixed quantitative and qualitative Are there sufficient quantitative studies to provide evidence for findings? Can qualitative studies be included as a separate part of the study? How?

15. Determining Research Outlet Does the proposed journal – publish research on the construct? – Publish reviews or meta-analyses? Is there a journal devoted to reviews that your project would fit with? Has a recent similar meta-analysis been published? If so, will yours add anything new? – Ex. Allen, et al (under review) evaluated articles on first grade retention after 1990 focusing on the quality of the research design in each study to determine if the effects were different from a fairly recent meta-analysis by Jimerson (2001)

16. Meta-Analysis as an interactive, developing process View meta-analysis as evolutionary – As studies are reviewed and included, purpose and scope may change Assume initial conceptualizations about both outcomes and potential predictors may change over time – Definitions, instruments, coding may all change as studies are found and included Plan for revisions to all aspects of the meta-analysis

17. FINDING STUDIES Searches Selection criteria

18. Searches Traditional literature review methods: – Current studies are cumulated Branching backward search uses the – Reference Lists of current studies Electronic searches – Google, Google Scholar, PsyInfo, research library catalogs (for major research institution libraries) – Searches of major journal article titles and abstracts (commonly available now through electronic libraries) Abstract vs. full content searches- electronic, pdf, hard copy Author requests: email or hard copy requests for newly published articles or other works not found in typical search outcomes

19. Selection Criteria In or out: – Any quantitative data available? Descriptive data- means and SDs for all groups of interest? Analysis summaries- F- or t-tests, ANOVA tables etc. available that may be utilized? Iterative process: outs may come back in given broader definitions of a construct Duplicated articles/data reports? Decide on which to keep (earliest? Most complete?) why were multiple articles prepared? New groups included that can be used? Keep records of every study considered- excel or hard copy, for example

20. Selection Criteria Useful procedure: Create an index card for each study along with notes of each to refer to Organize studies into categories or clusters Review periodically as new studies are added, revise or regenerate categories and clusters Consider why you organized the studies this way- does it reflect the scope of research, construct organization, or other classes?

21. CODING STUDIES Dependent variable(s) Construct(s) represented Measure name and related characteristics Effect size and associated calculations Independent variables Population Sample Design Potential Mediators and Moderators Bias mechanisms and threats to validity

22. CODING STUDIES- Dependent Variables Construct name(s): eg. Receptive or Expressive Vocabulary Measurement name: Willson EV Test Raw score summary data (mean, SD for each group or summary statistics and standard errors for dep. var): Exp Mean= 22 Exp SD = 5 n=100, Con Mean = 19 Con SD = 4, n=100 Effect size (mean difference or correlation) e = (22-19)/  (20.5) Effect size transformation used (if any) for mean differences: t-test transform ( e = t (1/n1 + 1/n2) ½ ), F-statistic transform (F) ½ = t for df = 1, 198 probability transform to t-statistic: t(198) =  [probt(.02)] point-biserial transform to t-statistic, regression coefficient t-statistic Effect size transformations used (if any) for correlations: t-statistic to correlation: r 2 = t 2 / (t 2 + df) Regression coefficient t-statistic to correlation

23. CODING STUDIES- Independent variables Population(s): what is the intended population, what characterizes it? Gender? Ethnicity? Age? Physical characteristics, Social characteristics, Psychosocial characteristics? Cognitive characteristics? Sample: population characteristics in Exp, Control samples eg. % female, % African-American, % Hispanic, mean IQ, median SES, etc.

24. CODING STUDIES- Independent variables Design (mean difference studies): 1. Random assignment, quasi-experimental, or nonrandom groups 2. Treatment conditions: treatment variables of importance (eg. duration, intensity, massed or distributed etc.); control conditions same 3. Treatment givers: experience and background characteristics: teachers, aides, parents 4. Environmental conditions (eg. classroom, after- school location, library)

25. CODING STUDIES- Independent variables Design (mean difference studies) 5. Time characteristics (when during the year, year of occurrence) 6. Internal validity threats: maturation, testing, instrumentation, regression, history, selection

26. CODING STUDIES- Independent variables Mediators and Moderators Mediators are indirect effects that explain part or all of the relationship between hypothesized treatment and effect: (T)e M In meta analysis we establish that the effect of T on the outcome is nonzero, then if M is significantly related to the effect e. We do not routinely test if T predicts M

27. CODING STUDIES- Independent variables Mediators and Moderators Moderators are variable for which the relationship changes from one moderator value to the next: (T)e for M=1 (T)e for M=2 In meta analysis we establish that the effect of T on the outcome is nonzero, then if M is significantly related to the effect e. We do not routinely test if T predicts M.7.3

28. Coding Studies- Bias Mechanisms Researcher potential bias- membership in publishing cohort/group Researcher orientation- theoretical stance or background Type of publication: Refereed vs. book chapter vs. dissertation vs. project report: do not assume refereed articles are necessarily superior in design or analysis- Mary Lee Smith’s study of gender bias in psychotherapy indicated publication bias against mixed gender research showing no effects by refereed journals with lower quality designs than non- refereed works Year of publication- have changing definitions affected effects? Eg. Science interest vs. attitude- terms used interchangeably in 1940s-1950s; shift to attitude in 1960s Journal of publication- do certain journals only accept particular methods, approaches, theoretical stances?

29. Computing Effect Sizes- Mean Difference Effects Glass: e = (Mean Experimental – Mean Control )/SD o SD = Square Root (average of two variances) for randomized designs o SD = Control standard deviation when treatment might affect variation (causes statistical problems in estimation) Hedges: Correct for sampling bias: g = e[ 1 – 3/(4N – 9) ] where N=total # in experimental and control groups Sg = [ (Ne + Nc)/NgNc + g 2 /(2(Ne + Nc) ] ½

30. Computing Effect Sizes- Mean Difference Effects Example from Spencer ADHD Adult study Glass: e = (Mean Experimental – Mean Control )/SD = (82 – 101)/21.55 =.8817 Hedges: Correct for sampling bias: g = e[ 1 – 3/(4N – 9) ] =.8817 (1 – 3/(4*110 – 9) =.8762 Note: SD computed from t-statistic of 4.2 given in article: e = t*(1/N E + 1/N C ) ½

32. Computing Mean Difference Effect Sizes from Summary Statistics t-statistic: e = t*(1/N E + 1/N C ) ½ F(1,df error ): e = F ½ *(1/N E + 1/N C ) ½ Point-biserial correlation: e = r*(dfe/(1-r 2 )) ½ *(1/N E + 1/N C ) ½ Chi Square (Pearson association):  =  2 /(  2 + N) e =  ½ *(N/(1-  )) ½ *(1/N E + 1/N C ) ½ ANOVA results: Compute R 2 = SS Treatment /Ss total Treat R as a point biserial correlation

33. Excel workbook for Mean difference computation

34. Story Book Reading References 1 Wasik & Bond: Beyond the Pages of a Book: Interactive Book Reading and Language Development in Preschool Classrooms. J. Ed Psych 2001 2 Justice & Ezell. Use of Storybook Reading to Increase Print Awareness in At-Risk Children. Am J Speech-Language Path 2002 3 Coyne, Simmons, Kame’enui, & Stoolmiller. Teaching Vocabulary During Shared Storybook Readings: An Examination of Differential Effects. Exceptionality 2004 4 Fielding-Barnsley & Purdie. Early Intervention in the Home for Children at Risk of Reading Failure. Support for Learning 2003

35. Coding the Outcome 1 open Wasik & Bond pdf 2 open excel file “computing mean effects example” 3 in Wasik find Ne and Nc 4 decide on effect(s) to be used- three outcomes are reported: PPVT, receptive, and expressive vocabulary at classroom and student level: what is the unit to be focused on? Multilevel issue of student in classroom, too few classrooms for reasonable MLM estimation, classroom level is too small for good power- use student data

36. Coding the Outcome 5 Determine which reported data is usable: here the AM and PM data are not usable because we don’t have the breakdowns by teacher-classroom- only summary tests can be used 6 Data for PPVT were analyzed as a pre-post treatment design, approximating a covariance analysis; thus, the interaction is the only usable summary statistic, since it is the differential effect of treatment vs. control adjusting for pretest differences with a regression weight of 1 (ANCOVA with a restricted covariance weight): Interaction ij = Grand Mean – Treat effect –pretest effect = Y… - a i.. – b. j. Graphically, the Difference of Gain inTreat(post-pre) and Gain in Control (post – pre) F for the interaction was F(l,120) = 13.69, p <.001. Convert this to an effect size using excel file Outcomes Computation What do you get? (.6527)

37. Coding the Outcome Y Control Treatment gains Gain not “predicted” from control pre post

38. Coding the Outcome 7 For Expressive and Receptive Vocabulary, only the F- tests for Treatment-Control posttest results are given: Receptive: F(l, 120) = 76.61, p <.001 Expressive: F(l, 120) =128.43, p<.001 What are the effect sizes? Use Outcomes Computation 1.544 1.999

39. Getting a Study Effect Should we average the outcomes to get a single study effect or Keep the effects separate as different constructs to evaluate later (Expressive, Receptive) or Average the PPVT and receptive outcome as a total receptive vocabulary effect? Comment- since each effect is based on the same sample size, the effects here can simply be averaged. If missing data had been involved, then we would need to use the weighted effect size equation, weighting the effects by their respective sample size within the study

40. Getting a Study Effect For this example, let’s average the three effects to put into the Computing mean effects example excel file- note that since we do not have means and SDs, we can put MeanC=0, and MeanE as the effect size we calculated, put in the SDs as 1, and put in the correct sample sizes to get the Hedges g, etc. (.6567 + 1.553 + 2.01)/3 = 1.4036

41. 2 Justice & Ezell Receptive: 0.403 Expressive: 0.8606 Average = 0.6303 3 Coyne et al Taught Vocab: 0.9385 Untaught Vocab: 0.3262 Average = 0.6323 4 Fielding PPVT: -0.0764

42. Computing mean effect size Use e:\\Computing mean effects1.xls Mean

43. Computing Correlation Effect Sizes Reported Pearson correlation- use that Regression b-weight: use t-statistic reported, e = t*(1/N E + 1/N C ) ½ t-statistics: r = [ t 2 / (t 2 + df error ) ] ½ Sums of Squares from ANOVA or ANCOVA: r = (R 2 partial) ½ R 2 partial = SS Treatment /Ss total Note: Partial ANOVA or ANCOVA results should be noted as such and compared with unadjusted effects

44. Computing Correlation Effect Sizes To compute correlation-based effects, you can use the excel program “Outcomes Computation correlations” The next slide gives an example. Emphasis is on disaggregating effects of unreliability and sample-based attenuation, and correcting sample- specific bias in correlation estimation For more information, see Hunter and Schmidt (2004): Methods of Meta-Analysis. Sage. Correlational meta-analyses have focused more on validity issues for particular tests vs. treatment or status effects using means

45. Computing Correlation Effects Example

46. EFFECT SIZE DISTRIBUTION Hypothesis: All effects come from the same distribution What does this look like for studies with different sample sizes? Funnel plot- originally used to detect bias, can show what the confidence interval around a given mean effect size looks like Note: it is NOT smooth, since CI depends on both sample sizes AND the effect size magnitude

47. EFFECT SIZE DISTRIBUTION Each mean effect SE can be computed from SE = 1/  (w) For our 4 effects: 1: 0.200525 2: 0.373633 3: 0.256502 4: 0.286355 These are used to construct a 95% confidence interval around each effect

48. EFFECT SIZE DISTRIBUTION- SE of Overall Mean Overall mean effect SE can be computed from SE = 1/  (  w) For our effect mean of 0.8054, SE = 0.1297 Thus, a 95% CI is approximately (.54, 1.07) The funnel plot can be constructed by constructing a SE for each sample size pair around the overall mean- this is how the figure below was constructed in SPSS, along with each article effect mean and its CI

50. EFFECT SIZE DISTRIBUTION- Statistical test Hypothesis: All effects come from the same distribution: Q-test Q is a chi-square statistic based on the variation of the effects around the mean effect Q =  w i ( g – g mean ) 2 Q  2 (k-1) k

51. Example Computing Q Excel file effectdw Qiprob(Qi)sig? 10.585.43 0.71515980.397736175no 2-0.0510.24 0.73262480.392033721no 30.524.35 0.39579490.52926895no 40.029.69 0.3663190.545017585no 5-0.3040.65 10.6973490.001072891yes 60.1429.94 0.16866160.681304025no 70.6854.85 11.7274520.000615849yes 8-0.024.00 0.21256220.644766516no 0.2154 Q=25.015924 df7 prob(Q)=0.0007539

52. Computational Excel file Open excel file: Computing Q Enter the effects for the 4 studies, w for each study (you can delete the extra lines or add new ones by inserting as needed) from the Computing mean effect excel file What Q do you get? Q = 39.57 df=3 p<.001

53. Interpreting Q Nonsignificant Q means all effects could have come from the same distribution with a common mean Significant Q means one or more effects or a linear combination of effects came from two different (or more) distributions Effect component Q-statistic gives evidence for variation from the mean hypothesized effect

54. Interpreting Q- nonsignificant Some theorists state you should stop- incorrect. Homogeneity of overall distribution does not imply homogeneity with respect to hypotheses regarding mediators or moderators Example- homogeneous means correlate perfectly with year of publication (ie. r= 1.0, p<.001)

55. Interpreting Q- significant Significance means there may be relationships with hypothesized mediators or moderators Funnel plot and effect Q-statistics can give evidence for nonconforming effects that may or may not have characteristics you selected and coded for

56. MEDIATORS Mediation: effect of an intervening variable that changes the relationship between an independent and dependent variable, either removing it or (typically) reducing it. Path model conceptualization: TreatmentOutcome Mediator

57. MEDIATORS Statistical treatment typically requires both paths ‘a’ and ‘b’ to be significant to qualify as a mediator. Meta- analysis seems not to have investigated path ‘a’ but referred to continuous predictors as regressors Lipsey and Wilson(2001) refer to this as “Weighted Regression Analysis” TreatmentOutcome Mediator a b

58. Weighted Regression Analysis Model: e = b X + residual Regression analog: Q = Q regression + Q residual Analyze as “weighted least squares” in programs such as SPSS or SAS In SPSS the weight function w is a variable used as the weighting

59. Weighted Regression Analysis Emphasis on predictor and its standard error: the usual regression standard error is incorrect, needs to be corrected (Hedges & Olkin, 1985): SE’ b = SE b / (MSe) ½ where SE b is the standard error reported in SPSS, and MSe is the reported regression mean square error

60. Weighted Regression Q-statistics Q regression = Sum of Squares regression df = 1 for single predictor Q residual = Sum of Squares residual df = # studies - 2 Significance tests: Each is a chi square test with appropriate degrees of freedom

61. 98.999.0531191.702628.781319 712.810.0324260.267212.369946 89.0911.5630200.7561211.22962 810.8610.5225250.6532111.867084 97.737.8622281.541429.530347 610.1110.7724260.3507112.291338 78.576.9134160.4438110.651743 79.598.5322281.1245210.659409 87.9810.9230200.542111.591384 912.698.1628220.6337111.739213 58.6110.572822-0.5976211.80079 59.347.4324260.3771112.262378 710.3910.126240.7234211.714913 68.669.421290.2413112.094229 79.169.0425250.6637111.847643 88.186.4330200.9038210.928737 910.049.8425250.4603112.177485 712.3311.422280.3948112.087879 78.8310.672327-0.1726212.374215 810.888.8126240.4633112.154409 89.58.0928220.8481211.317137 910.4210.1218320.7114110.885366 911.827.1328220.5407111.891682 611.698.1123270.4926112.05664

62. ANOVA b,c ModelSum of SquaresdfMean SquareFSig. Regression19.166119.16612.096.002 a Residual34.858221.584 Total54.02423 a.Predictors: (Constant), AGE b.Dependent Variable: HEDGE d* c. Weighted Least Squares Regression - Weighted by w Coefficients a,b Model Unstandardized CoefficientsStandardized Coefficients tSig. BStd. ErrorBeta (Constant)-1.037.465-2.230.036 AGE.215.062.5963.478.002 a. Dependent Variable: HEDGE d* b. Weighted Least Squares Regression - Weighted by w SPSS ANALYSIS OUTPUT

63. Example See SPSS “sample meta data set.sav” or the excel version “sample meta data set regression” The d effect is regressed on Age b = 0.215, SE b = 0.062, MSe = 1.584 Thus, SE’ b = 0.062 / (1.584) ½ = 0.0493 A 95% CI around b gives (0.117, 0.313) for the regression weight of age on outcome, p<.001

64. Q-statistic tests Q regression = 19.166 with df=1, p <.001 Q residual = 34.858 with df=22, p =.040 So- are the residuals homogeneous or not? Given a large number of significance tests, one might require the Type I error rate for such tests to be.001 or something small

65. MODERATORS Moderators are typically considered categorical variables for which effects differ across categories or levels In a limited form, this can be considered a treatment- moderator interaction Moderator analysis is more general in the sense that any parameters of a within-category analysis may change across categories (multigroup analysis concept in Structural Equation Modeling)

66. Moderator Analysis- Q Between Analog to ANOVA- split into Q between and Q within Q B =  w i E i 2 – (  w i E i ) 2 /  w i where E i is the mean for category i and w i is the total weight function for E i Remember that you constructed a mean effect for a study; the weight function for that mean effect is the sum of the weights that made up the mean: E i =  w j g j /  w j for J effects in study I w i =  w j

67. Moderator Analysis- Q Within Analog to ANOVA- split into Q between and Q within Q W =  w j(i) (E j(i) - MeanE i ) 2 I j where MeanE i is the mean for each category i, E j(i) is an effect j in category i and w j(i) is the weight function for the j th effect in category i This is analogous to the within-subjects term in ANOVA Lipsey and Wilson do not give a very good equation for this on p. 121- confusing

68. Computational Issues The excel file “Meta means working COMPUTATIONS” provides a workbook to compute such effects An exemplar is shown below, is in your set of materials Computation of Q B and Q W are done from the summary data of Hedge’s g and sample sizes

70. Moderator Example For our Storybook reading example, we can break the effect into two design types: 1 = no baseline equivalence 2 = baseline equivalence Wasik = 2 Coyne = 1 Justice = 2 Fielding = 1

71. Moderator Example Select “Meta means working COMPUTATIONS” excel file Reduce the number of studies to 2 in Design 1 and 2 in design 2 Insert the Hedge’s g effects, Cntrl N, Trmt N into the correct boxes, all other effects will be correctly computed

72. STUDY IDHedges's GSEWCtrl NTrmt NNDesignE*W Sum W(group) Group Means Wgp* Mgp Wgp *Mgp^2 (Wgp*Mgp)^ 2 Wtd Within GP SS 30.62000.2615.2130346419.429927.40390.30858.45442.608371.47631.4757 4-0.08000.2912.192326491-0.976 1.8406 11.40000.2024.896163124234.85232.06111.223439.223847.98681538.50780.7763 20.61000.377.1715 3024.3717 2.6966 SumEW(gp1)= 8.4544 59.4650 47.678250.59511609.9840 SumEW(gp2)=39.224 3.3163 3.4729 All computations per Lipsey & Wilson QB=23.5206 df=1.0000 p(QB)=0.0000 QW=6.7893 df=5.0000 p(QW)=0.2368 Q=30.3098 Storybook Reading Design Moderator effect Q B sig., two design means are different Q W nonsig., homogeneous effects within the two design categories

73. Meta-Analysis Report-Writing Traditional journal approach: -Intro, lit review, methods, results, discussion -References: background, studies in meta analysis* -Tables: effects, SEs, Q’s, mediators, moderators -Figures: Cluster diagrams, funnel plots, graphs of effects by features Literature review approach: -Thematic or theory focus: what lit exists, what does it say Tabular summarizations of works

74. Current Issues Multi-level models: Raudenbush & Bryk analysis in HLM6 Structural equation modeling in meta-analysis Clustering of effects: cluster analysis vs. latent class modeling Multiple studies by same authors- how to treat (beyond ignoring follow-on studies), the study dependence problem Multiple meta-analyses: consecutive, overlapping

75. Multilevel Models Raudenbush & Bryk HLM 6 One effect per study Two level model, mediators and moderators at the second level Known variance for first level (w i ) Mixed model analysis: requires 30+ studies for reasonable estimation, per power analysis Maximum likelihood estimation of effects

76. Multilevel Models Model: Level 1:g i = g i + e i where there is one effect g per study i Level 2: g i =  0 +  1 W + u i where W is a study-level predictor such as design in our earlier example Assumption: the variance of g i is known = w i

77. Structural Equation Modeling in SEM New area- early work in progress: Cheung & Chan (2005, Psych Methods), (2009, Struc Eqn Modeling)- 2-step approach using correlation matrices (variables with different scales) or covariance matrices (variables measured on the same scale/scaling) Stage 1: create pooled correlation (covariance) matrix Stage 2: fit SEM model to Stage 1 result

78. Structural Equation Modeling in SEM Pooling correlation matrices: Get average r: r mean(jk) =  w i r iij /  w ijk I i where j and k are the subscripts for the correlation between variables j and k, where i is the ith data set being pooled Cheung & Chan propose transforming all r’s to Fisher Z- statistics and computing above in Z If using Z, then the SE for Zi is (1-r 2 )/n ½ and

79. Structural Equation Modeling in SEM Pooling correlation matrices: for each study, COV g (r ij, r kl ) = [.5r ij rkl (r 2 ik + r 2 il + r 2 jk + r 2 jl ) + r ik *r jl + r il *r jk – (r ij *r ik *r il + r ji *r jk *r jl + r ki *r kj *r kl + r li *r lj *r lk )]/n Let  i = covariance matrix for study i, G = {0,1} matrix that selects a particular correlation for examination, Then G = [ |G 1 |’ G 2 |’…| G k |’]’ and  = diag [  1,  2, …  k]

80. Structural Equation Modeling in SEM Beretvas & Furlow (2006) recommended transformations of the variances and covariances: SDrtrans = log(s) + 1/(2(n-1) COV(r i,r j )trans = r 2 ij /(2(n-1)) The transformed covariance matrices for each study are then stacked as earlier

81. Clustering of effects: cluster analysis vs. latent class modeling Suppose Q is significant. This implies some subset of effects is not equal to some other subset Cluster analysis uses study-level variables to empirically cluster the effects into either overlapping or nonoverlapping subsets Latent class analysis uses mixture modeling to group into a specified # of classes Neither is fully theoretically developed- existing theory is used, not clear how well they work

82. Multiple studies by same authors- how to treat (beyond ignoring follow-on studies), the study dependence problem Example: in storybook telling literature, Zevenberge, Whitehurst, & Zevenbergen (2003) was a subset of Whitehurst, Zevenbergen, Crone, Schultz, Velging, & Fischel (1999), which was a subset of Whitehurst, Arnold, Epstein, Angell, Smith, & Fischel (1994) Should 1999 and 2003 be excluded, or included with adjustments to 1994? Problem is similar to ANOVA: omnibus vs. contrasts Currently, most people exclude later subset articles

83. Multiple meta-analyses: consecutive, overlapping The problem of consecutive meta-analyses is now arising: Follow-ons typically time-limited (after last m-a) Some m-a’s partially overlap others: how should they be compared/integrated/evaluated? Are there statistical methods, such as the correlational approach detailed above, that might include partial dependence? Can time-relatedness be a predictor? Willson (1985)

84. CONCLUSIONS Meta-analysis continues to evolve Focus in future on complex modeling of outcomes (SEM, for example) More work on integration of qualitative studies with meta-analysis findings