1. The Campbell Collaborationwww.campbellcollaboration.org Introduction to Robust Standard Errors Emily E. Tanner-Smith Associate Editor, Methods Coordinating Group Research Assistant Professor, Vanderbilt University Campbell Collaboration Colloquium Copenhagen, Denmark May 30 th, 2012

2. The Campbell Collaborationwww.campbellcollaboration.org Outline Types of dependencies Dealing with dependencies Robust variance estimation Practical considerations – Choosing weights – Handling covariates Robust variance estimation in Stata

3. The Campbell Collaborationwww.campbellcollaboration.org Types of Dependencies Most meta-analysis techniques assume effect sizes are statistically independent But there are many instances when you might have dependent effect sizes – Multiple measures of the same underlying outcome construct – Multiple measures across different follow-up periods – Multiple treatment groups with a common control group – Multiple studies from the same research laboratory

4. The Campbell Collaborationwww.campbellcollaboration.org Types of Dependencies Assume T i = θ i + ε i, where – T i is the effect size estimate – θ i is the effect size parameter – ε i is the estimation error Statistical dependence can arise because – ε i are correlated – θ i are correlated – or both

5. The Campbell Collaborationwww.campbellcollaboration.org Correlated Effects Model (ε i correlated) kj correlated estimates of the study specific ES τ 2 = between study variation in ES Meta-regression Dependent effect size meta- regression Study 1, θ1Estimate 1.1 of θ1Estimate 1.2 of θ1Estimate 1.3 of θ1Study 2, θ2Estimate 2.1 of θ2Study 3, θ3Estimate 3.1 of θ3Estimate 3.2 of θ3

6. The Campbell Collaborationwww.campbellcollaboration.org Hierarchical Model (θ i correlated) ω 2 = between-study, within-cluster variation in ES τ 2 = between cluster variation in average ES Meta- regression Dependent effect size meta- regression Cluster 1, θ1 Study 1.1 estimate of θ1.1 Study 1.2 estimate of θ1.2 Study 1.3 estimate of θ1.3 Cluster 2, θ2Study 2.1 estimate of θ2.1Cluster 3, θ3Study 3.1 estimate of θ3.1 Study 3.2 estimate of θ3.2

7. The Campbell Collaborationwww.campbellcollaboration.org Dealing with Dependencies Ignore it and analyze the effect sizes as if they are independent ( not recommended ) Select a set of independent effect sizes – Create a synthetic mean effect size – Randomly select one effect size – Choose the “best” effect size Model the dependence with full multivariate analysis – This requires information on the covariance structure Use robust variance estimation (Hedges, Tipton, & Johnson, 2010)Hedges, Tipton, & Johnson, 2010

8. The Campbell Collaborationwww.campbellcollaboration.org Variance Estimation Assume T = Xβ + ε, where T = (T 1,…,T m )’ T j is a k j x 1 vector of effect sizes for study j X = (X 1,…,X m )’ X j is a k j x p design matrix for study j β = (β 1,…,β p )’ β is a 1 x p vector of unknown regression coefficients ε = (ε 1,…,ε m )’ ε j is a k j x 1 vector of residuals for study j E( ε j ) = 0, V( ε j ) = Σ j

9. The Campbell Collaborationwww.campbellcollaboration.org Variance Estimation We can estimate β by: And the covariance matrix for this estimate is The problem is that although the variances in Σ j are known, the covariances are UNKNOWN

10. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimator (RVE) The RVE of b is where e j = T j – X jb is the k j x 1 estimated residual vector for study j

11. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimator (RVE) A robust test of H 0 : β a = 0 uses the statistic where v R aa is the a th diagonal of the V R matrix Note: the t-distribution with df = m - p should be used for critical values

12. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimator (RVE) Under regularity conditions and as m -> ∞, V m R is a consistent estimator of the true covariance matrix RVE theorem is asymptotic in the number of studies m, not the number of effect sizes Results apply to any type of dependency No distributional assumptions needed for the effect sizes Correlations do not need to be known or specified, though may impact the standard errors RVE theorem applies for any set of weights

13. The Campbell Collaborationwww.campbellcollaboration.org Practical Issues: Choosing Weights Although the RVE works for any weights, the most efficient weights are inverse-variance weights – In the hierarchical model:

14. The Campbell Collaborationwww.campbellcollaboration.org Practical Issues: Choosing Weights In the correlated effects model, we can estimate approximately efficient weights by assuming a simplified correlation structure: – Within each study j, the correlation between all pairs of effect sizes is a constant ρ – ρ is the same in all studies – k j sampling variances within the study are approximately equal with average V  j

15. The Campbell Collaborationwww.campbellcollaboration.org Practical Issues: Choosing Weights In the correlated effects model, we can take a conservative approach when calculating weights by also assuming ρ = 1, and weights become: Conservative approach, because studies do not receive additional weight for contributing multiple effect sizes

16. The Campbell Collaborationwww.campbellcollaboration.org Practical Issues: Choosing Weights In the correlated effects model, ρ also occurs in the estimator of τ 2 : Use external information about ρ if available (test reliabilities, correlations reported in studies, etc.) Take a sensitivity approach when estimating τ 2 by estimating the model with various values of ρ in (0, 1)

17. The Campbell Collaborationwww.campbellcollaboration.org Practical Issues: Choosing Weights For the correlated effects model, RVE software is currently programmed to default to (per Hedges, Tipton, & Johnson recommendation): – Conservative approach to estimate weights (assume ρ = 1) – User must specify ρ for estimation of τ 2 ; sensitivity tests recommended

18. The Campbell Collaborationwww.campbellcollaboration.org Practical Issues: Handling Covariates Some covariates may vary within groups (i.e., studies or clusters) and between groups, e.g., – Length of follow-up after intervention – Time frame of outcome measure – Outcome reporter (self-report vs. parent-report) – Type of outcome construct (frequency vs. quantity of alcohol use) When modeling the effects of a covariate, ask if the effect of interest is between- or within-groups

19. The Campbell Collaborationwww.campbellcollaboration.org Practical Issues: Handling Covariates In a standard meta-regression with independent effect sizes, T j = β 0 + X j β 1 + … where X j is length to follow-up, β 0 and β 1 can be interpreted as: – β 0 = the average effect size when X j = 0 e.g. the average effect size in studies in which the intervention just occurred – β 1 = the effect of a 1-unit increase in X j on T j e.g. the effect size change associated with moving from a study in which the intervention just occurred to a study in which the effect size was measured at a 1 month posttest follow-up

20. The Campbell Collaborationwww.campbellcollaboration.org Practical Issues: Handling Covariates In the correlated effects model, for a fixed study ( j = 1), now assume there are multiple outcomes. This study has its own regression equation: T i1 = β 01 + X i1 β 11 +… The coefficients β 01 and β 11 can be interpreted as: – β 01 = the average effect size when X i1 = 0 e.g. the average effect size for units in the study ( j = 1) when the intervention just occurred – β 11 = the effect of a 1-unit increase in X i1 on T i1 e.g. the effect size change for units in the study ( j = 1) at the time of intervention and at follow-up 1 month later

21. The Campbell Collaborationwww.campbellcollaboration.org Practical Issues: Handling Covariates When using the RVE, these two types of regression occur in one analysis: Within GroupT ij = β 0j + X ij β 2 + … Between Groupβ 0j = β 0 + X  j β 1 + … These two regressions are combined into one analysis and model: T ij = β 0 + X ij β 2 + X  j β 1 + …

22. The Campbell Collaborationwww.campbellcollaboration.org Practical Issues: Handling Covariates To properly separate estimation of within- and between- group effects of covariates, use group mean centering: X cij = X ij – X  j where X  j is the mean value of X ij in group j (and where group is either study or cluster). So now, T ij = β 0 + X cij β 2 + X  j β 1 + … If you don’t center X ij you are actually modeling a weighted combination of the within- and between-study effect, which is difficult to interpret

23. The Campbell Collaborationwww.campbellcollaboration.org Practical Issues: Handling Covariates When using a covariate, ask if the effect of interest is between- or within-groups Make sure to group-center your within-group variables Acknowledge that if only a few groups have variability in X ij – Within-group estimate of β 2 (associated with X cij ) will be imprecise (i.e. have a large standard error) – The types of groups in which X ij varies may be different than (i.e. not representative of) groups in which X ij does not vary

24. The Campbell Collaborationwww.campbellcollaboration.org Calculating Robust Variance Estimates Variables you will need in your dataset – Group identifier (e.g., study/cluster identification number) – Effect size estimate – Variance estimate of the effect size – Any moderator variables or covariates of interest Additional pieces of information you will need to specify – In a correlated effects model: assumed correlation between all pairs of effect sizes ( ρ ) – Fixed, random, hierarchical, or user-specified weights

25. The Campbell Collaborationwww.campbellcollaboration.org Stata ado file available at SSC archive: type ssc install robumeta SPSS macro available at: http://peabody.vanderbilt.edu/peabody_research_institute/methods_resources.xml R functions available at: http://www.northwestern.edu/ipr/qcenter/RVE-meta-analysis.html Calculating Robust Variance Estimates

26. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata Install robumeta.ado file from the Statistical Software Components (SSC) archive

27. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata Access example datasets and syntax/output files here: https://my.vanderbilt.edu/emilytannersmith/training-materials/

28. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata robumeta.ado command structure robumeta depvar [indepvars] [if] [in], study(studyid) variance(variancevar) weighttype(weightingscheme) rho(rhoval) [options]

29. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata Example of a correlated effects model (correlated ε) Fictional meta-analysis on the effectiveness of alcohol abuse treatment for adolescents Effect sizes represent post-treatment differences between treatment and comparison groups on some measure of alcohol use (positive effect sizes represent beneficial treatment effects) – Number of effect sizes k = 172 – Number of studies m = 39 – Average number of effect sizes per study = 4.41

30. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata

31. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata

32. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata

33. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata

34. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata 4 moderators of interest: 2 vary within and between studies, 2 vary between studies only

35. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata To model both the within- (X cij ) and between- effects (X  j ) of the type of alcohol outcome and follow-up time frame, create group mean and group mean centered variables

36. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata

37. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata Let’s say we have a similar meta-analysis, but now need to estimate a hierarchical model (correlated θ ) Effect sizes represent post-treatment differences between treatment and comparison groups on some measure of alcohol use (positive effect sizes represent beneficial treatment effects) – Number of effect sizes k = 68 – Number of research labs m = 15 – Average number of effect sizes per research lab = 4.5

38. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata τ 2 – between lab variance component; ω 2 between-study within-lab variance component

39. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata 5 moderators of interest: all vary within and between clusters (research labs)

40. The Campbell Collaborationwww.campbellcollaboration.org Robust Variance Estimation in Stata To model both the within- (X cij ) and between- effects (X  j ) of the covariates of interest, create group mean and group mean centered variables

41. The Campbell Collaborationwww.campbellcollaboration.org

42. Conclusions & Recommendations Robust variance estimation is one way to handle dependencies in effect size estimates, and allows estimation of within- and between-study effects of covariates – Method performs well when there are 20 or more studies with an average of 2 or more effect size estimates per study Choose the proper model for the type of dependencies in your data (correlated ε or correlated θ )

43. The Campbell Collaborationwww.campbellcollaboration.org Conclusions & Recommendations When using the correlated effects model (correlated ε ), with efficient weights, if you have no information on ρ: – Use a sensitivity approach for estimating τ 2 – Assume ρ = 1 in your weights, i.e., For each covariate X ij in your model, remember that you can estimate: – Between-group effect: group mean (X  j ) – Within-group effect: group mean centered variable (X cij = X ij – X  j )

44. The Campbell Collaborationwww.campbellcollaboration.org Recommended Reading Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39-65.

45. The Campbell Collaborationwww.campbellcollaboration.org P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway E-mail: info@c2admin.org http://www.campbellcollaboration.org