1.   Meta-Analysis of Correlated Data

3. Common Forms of Dependence Multiple effects per study –Or per research group! Multiple effect sizes using same control Phylogenetic non-independence Measurements of multiple responses to a common treatment Unknown correlations…

4. Multiple Sample Points per Study! StudyExperiment in StudyHedges DV Hedges D Ramos & Pinto 201014.327.23 Ramos & Pinto 201022.346.24 Ramos & Pinto 201033.895.54 Ellner & Vadas 20031-0.542.66 Ellner & Vadas 20032-4.548.34 Moria & Melian 200813.449.23

5. Hierarchical Models Study-level random effect Study-level variation in coefficients Covariates at experiment and study level

6. Hierarchical Models Random variation within study (j) and between studies (i) T ij  ij,  ij 2 )  ij  j,  j 2   j ,  2 

7. Study Level Clustering

8. Hierarchical Partitioning of One Study Grand Mean Study Mean Variation due to  Variation due to 

9. Example: Data Set 1 Group Effect Variance 1 A 0.2 0.10 2 A 0.6 0.15 3 A 0.5 0.05 4 A 0.1 0.06 5 B 0.8 0.08 6 B 0.4 0.05 7 B 0.9 0.04 8 C 0.2 0.09...

10. A Two-Step Solution T ij  ij,  ij 2 )  ij  j,  j 2   j ,  2  library(plyr) data1_study <- ddply(data1,.(Group), function(adf){ mod <- rma(Effect, Variance, data=adf) cbind(theta_j = coef(mod), se_theta_j = coef(summary(mod))[1,2], omega2 = mod$tau2) })

11. A Two-Step Solution T ij  ij,  ij 2 )  ij  j,  j 2   j ,  2  > data1_study Group theta_j se_theta_j omega2 1 A 0.3312500 0.1369306 0.00000000 2 B 0.7005364 0.1654476 0.02854676 3 C 0.6788453 0.1987595 0.17151248 4 D 0.7836646 0.2677693 0.26470540 5 E 0.8552760 0.1556476 0.14561528 jj jj

12. A Two-Step Solution T ij  ij,  ij 2 )  ij  j,  j 2   j ,  2  > rma(theta_j, I(se_theta_j^2), data=data1_study) Random-Effects Model (k = 5; tau^2 estimator: REML) tau^2 (estimated amount of total heterogeneity): 0.0272 (SE = 0.0414)... estimate se zval pval ci.lb ci.ub 0.6472 0.1087 5.9545 <.0001 0.4342 0.8603 *** 22 

13. Multiple Effects per Research Group

14. Solutions to Multiple Hierarchies Multiple-Step Meta-analyses Multi-level hierarchical model fits –Better estimator –Accommodates more complex data structures –May need to go Bayesian (don't be scared!) Model correlation…

15. Common Forms of Dependence Multiple effects per study –Or per research group! Multiple effect sizes using same control Phylogenetic non-independence Measurements of multiple responses to a common treatment Unknown correlations…

16. Multiple Effect Sizes with Common Control Effect of each treatment calculated using same control!

17. The Control Keeps Showing Up! n c and sd c are going to be the same for all treatments Effect sizes will covary

18. Calculating Covariance Formulae available or derivable for all effect sizes

19. A Mixed Effect Group Model Group means, random study effect, and then everything else is error T i  im,  i 2 ) where  im  m,  2 

20. A Mixed Effect Group Model Group means, random study effect, and then everything else is error T i  MVN  i,  i ) where  i  MVN  X i ,  2 

21. What are  i and  i ? i =i=i =i= T i  MVN  i,  i )

22. What about the treatment effects? X i =   i =  i  MVN  X i ,  2 

23. What if treatments are correlated?  i = T i  MVN  i,  i )

24. Why does covariance matter?   x-y =   x +   y + 2   x,y In asking if two treatments differ, cov helps tighten confidence intervals High cov  more weight for a study as treatments share information

25. Multiple Treatments study trt m1i m2i sdpi n1i n2i 1 1 1 7.87 -1.36 4.2593 25 25 2 1 2 4.35 -1.36 4.2593 22 25 3 2 1 9.32 0.98 2.8831 38 40 4 3 1 8.08 1.17 3.1764 50 50 5 4 1 7.44 0.45 2.9344 30 30 6 4 2 5.34 0.45 2.9344 30 30 Common Control! http://www.metafor-project.org/doku.php/analyses:gleser2009

26. Calculating the Variance/Covariance Matrix [,1] [,2] [,3] [,4] [,5] [,6] [1,] 0.113 0.060 0.000 0.000 0.000 0.000 [2,] 0.060 0.098 0.000 0.000 0.000 0.000 [3,] 0.000 0.000 0.105 0.000 0.000 0.000 [4,] 0.000 0.000 0.000 0.064 0.000 0.000 [5,] 0.000 0.000 0.000 0.000 0.098 0.055 [6,] 0.000 0.000 0.000 0.000 0.055 0.082 http://www.metafor-project.org/doku.php/analyses:gleser2009

27. Fitting a Model with a VCOV Matrix > rma.mv(yi ~ factor(trt)-1, V, random =~ 1|study, data=dat)

28. Comparison to No Correlation Model With correlation estimate se zval pval ci.lb ci.ub factor(trt)1 2.3796 0.1641 14.4984 <.0001 2.0579 2.7013 factor(trt)2 1.5784 0.2007 7.8662 <.0001 1.1851 1.9716 Without correlation estimate se zval pval ci.lb ci.ub factor(trt)1 2.3759 0.1511 15.7196 <.0001 2.0797 2.6722 factor(trt)2 1.5177 0.2125 7.1405 <.0001 1.1011 1.9343

29. Common Forms of Dependence Multiple effects per study –Or per research group! Multiple effect sizes using same control Phylogenetic non-independence Measurements of multiple responses to a common treatment Unknown correlations…

30. Effect Size on Related Organisms Not Independent Warming on Litterfall Pine Trees Redwoods Fir Trees Oak Trees {

31. Phylogenetic Distances Determines Covariances for Weights

33. What about Multiple Studies of Some Species?

34. Common Forms of Dependence Multiple effects per study –Or per research group! Multiple effect sizes using same control Phylogenetic non-independence Measurements of multiple responses to a common treatment Unknown correlations…

35. Common Treatments Treatment Response 1Response 2Response 3

36. Common Treatments CO 2 CO 2 Assimilation GS Stomatal Conductance PN

37. Correlation Between Responses

38. What does Correlation between effects mean? X i =   i =  i  MVN  X i ,  2 

39. What Do We Do? 1. Create a 'composite' measure –Average –Weighted Average 2. Estimate different coefficients directly 3. Robust Variance Estimation (RVE)

40. The CO 2 Effect Data experiment Paper Measurement Hedges Var 1 1 121 GS -0.4862 0.3432 2 1 121 PN 0.9817 0.3735 3 2 121 GS 0.1535 0.3343 4 2 121 PN 2.0668 0.5113 5 3 121 GS 0.0965 0.3337 6 3 121 PN 2.6101 0.6172 7 4 121 GS 0.0000 0.2857 8 4 121 PN 3.6586 0.7638 9 5 168 GS -1.5271 0.4305 10 5 168 PN 1.8355 0.4737

41. Direct Estimation rma.mv(Hedges ~ Measurement, Var, random =~ Measurement|Paper, data=co2data, struct="HCS")

42.  and Different Correlation Structures Different structures for different data We do not always know which one is correct!

43. Estimates of Variance, Covariance Multivariate Meta-Analysis Model (k = 68; method: REML) Variance Components: outer factor: Paper (nlvls = 18) inner factor: Measurement (nlvls = 2) estim sqrt k.lvl fixed level tau^2.1 4.5098 2.1236 34 no GS tau^2.2 3.5799 1.8921 34 no PN rho 0.4751 no

44. Disadvantages to Multivariate Meta-Analysis 1. Difficult to estimate with few studies 2. Additional assumptions of covariance structure 3. Often little improvement over univariate meta-analysis 4. Publication bias exacerbated if data not missing at random Jackson et al. 2011 Satist. Med.

45. Robust Variance Estimation Essentially, bound weights within a group j to 1/mean var j and assume a value of  –Test sensitivity to choice of  –Correct DF for small sample sizes Methods developed by Hedges, Tipton, and others robumeta package in R

46. robumeta & RVE library(robumeta) robu(Hedges ~ Measurement, data=co2data, studynum=Paper, var.eff.size=Var)

47. RVE: Correlated Effects Model with Small-Sample Corrections Model: Hedges ~ Measurement Number of studies = 18 Number of outcomes = 68 (min = 2, mean = 3.78, median = 4, max = 10 ) Rho = 0.8 I2 = 85.59992 Tau.Sq = 2.561661 Struct="CS" only so far

48. Often, Choice of  Matters Little > sensitivity(co2modRVE) Type Variable rho=0 rho=0.2 rho=0.4 rho=0.6 rho=0.8 rho=1 1 Estimate intercept 0.00454 0.00457 0.00459 0.00462 0.00464 0.00467 2 - MeasurementPN 1.03149 1.03139 1.03128 1.03118 1.03107 1.03097 3 Std. Err. intercept 0.51173 0.51179 0.51185 0.51192 0.51198 0.51204 4 - MeasurementPN 0.61984 0.61990 0.61996 0.62003 0.62009 0.62015 5 Tau.Sq - 2.55334 2.55542 2.55750 2.55958 2.56166 2.56374

49. Results May Differ… Multivariate Meta-Analysis Model Results: estimate se zval pval ci.lb ci.ub intrcpt -0.0503 0.5221 -0.0963 0.9233 -1.0735 0.9730 MeasurementPN 1.0579 0.5359 1.9742 0.0484 0.0076 2.1082 * Robust Variance Estimation Model Results: Estimate StdErr t-value df P(|t|>) 95% CI.L 95% CI.U Sig 1 intercept 0.00464 0.512 0.00907 16.7 0.993 -1.077 1.09 2 MeasurementPN 1.03107 0.620 1.66278 16.7 0.115 -0.279 2.34

50. Other Sources of Unknown Correlations Shared system types Shared environmental events Labs or investigators Re-sampling experiments Experiments repeated in a region More…

51. Why Model Correlation instead of Hierarchy? Depends on question Analytical difficulty Leveraging correlation to aid with missing data