Dependencies Complex Data in Meta-analysis

Common Dependencies Independent subgroups within a study (nested in lab?) Multiple outcomes on the same people Multiple time periods Multiple groups compared to same control Multiple IVs (regression, SEM)

Independent Subgroups 1 Within a study, ES computed by group Male vs. female Memory by age group Can be treated as independent effect sizes for analysis But still nested; could be correlated Effectiveness of nicorette (M v F) Correlation of SAT & GPA (M v F)

Independent Subgroups 2 1. Treat groups within study as independent studies 2. Calculate separate ES, Average ES within study, weight using total N 3. Estimate results of nongrouped whole study; calculate ES on all participants ignoring groups 1 and 2 give same result if all fixed-effects 2 and 3 give same results only if grouping variable has no effect Most of the time, option 1 or is good; option 3 is often the only thing you can do because the original authors did not split into groups.

Independent 3: Separate Studies StudydN1N2w 1 M F M F M F

Independent 4

Note nesting & correlation of effects within a study – not necessarily independent effects, even though separate samples Could also compute if we did not have separate M and F effect sizes; instead we have one ES with percentage of M or F. Note that power of test for MvF is usually much greater with separate ES for MvF per study than with one ES per study and percentage of F as a continuous moderator.

Dependent Effect Sizes 1 Study has math and reading as outcomes Job satisfaction; turnover Self evaluation; observer evaluation of task performance Typically some studies have 1, some another, and some have both

Dependent ES 2 Extreme remedies Ignore and treat all as independent Double counting Simple average within each study One ES per study Error variance based on that ES and N for the study Perhaps overly conservative – admit less information than is available (this is the position in Borenstein et al.) My opinion is that we should use the simple average ES and V when multiple outcomes indicate same conceptual effect size.

Dependent ES 3 Could use HLM Or Adjust information (variance of average effect) for redundancy Redundancy indexed by correlation b/t measures or times Unfortunately, the meta-analysis programs that I know do not do this for you. Borenstein et al recommend computing the summary ES and its variance. Then use the typical inverse variance weights.

Variance of a composite V(composite) = sum(VCV matrix) V(a+b) = V(a)+V(b)+2Cov(a,b) correlation If a and b have variance = 1.0, and they are correlated.5, then the sum of a and b will have variance 3.0 Cov(a,b) = r*SD(a)*SD(b) covariance If a and b have variances = 2.0 and 3.0, and they are correlated.5, then the sum of a and b will have variance 7.45 If there are more elements, just sum all.

Covariance & Correlation Cov(X,Y) = rS X S Y The correlation is a standardized covariance.

Variance of an Average Multiply X by constant Var(kX)= k 2 Var(X) SD(kX) = kSD(X) Variance of the average: Var[1/2(A+B)]=1/4Var(A+B) = 1/4[Var(A)+Var(B)+2* r* SD(A)*SD(B)] Note that as r gets larger, variance of average gets larger and study weight gets smaller

Numerical Example 1 Cov(X,Y) = rS X S Y

Numerical Example 2 Cov(X,Y) = rS X S Y

Numerical Example 3 The average of the two variances (X1, X2) is 1.9, so 1.45 is somewhat smaller, giving more precision and more weight to this effect size.

Study Weights Compute the average ES w/in study Compute the variance of the average ES w/in Take the inverse = w Use the average and w in meta Combine what you have across studies

Example: hospital stay Effect of psycho-educational material on length of stay and cost of drugs in hospital Patients admitted for procedure X are randomly assigned to treatment or control, amount of meds and length of stay are two most common outcomes across studies.

Data Hospital Stay What to do with missing correlation? Sensitivity analysis. See Excel Sheet.

Take average corr? With the adjustment for correlated effects, you take the average within studies (so the total k is studies, not k ES), but give a bit more weight to studies depending on the amount of correlation.

Moderators with Dependencies When each study contains both variables, one could compute a difference between them (e.g., bigger ES for meds or length of stay?), adjust the variance for dependencies, and take the average over studies, similar to the previous slide. When some studies contain both variables, but other studies contain only one, it is not clear to me how to proceed. Perhaps two meta-analyses – one for the dependent, one for the independent, then combine. But problem with computing and using random effects.

Exercise Your friend is interested in the relations between Leader-Member Exchange (LMX) and affective outcomes at work. This friend has found 5 studies that contain a correlation matrix with LMX, affective commitment to the organization (AC) and job satisfaction (SAT). Your friend’s studies are listed in the Excel sheet named DepLMXstudy.xlsx. The friend wants to combine the two LMX correlations for the meta-analysis because the analysis is about LMX and affect. Pretend you want to help. Compute a simple (unit weighted) average of the effects for the two variables within each study. Compute the unit weighted average of the variances. You should have 5 ES, each with its own V. Compute a weighted average (following Borenstein et al.) for each of the studies to create an average effect size and variance. You should have 5 ES, each with its own V.

Exercise (continued) Run four meta-analyses, the first two treating each of the effects as independent (10 total ES; run both fixed and random analysis on these), second based on the unit weighted averages (5 ES), and the third based on the weighted average (5 ES). The last two, just run random-effects. Compare the results and prepare to share.