EEF Archive analysis overview Steve Higgins and Adetayo Kasim With support from ZhiMin Xiao, Nasima Akhter and Ewoud De Troyer s.e.higgins@durham.ac.uk a.s.kasim@durham.ac.uk @profstig EEF Conference 1st June, 2017

RESEARCH QUESTIONS How consistent are the results from the different analytic approaches or Methods? What is the impact of non-compliance on effect size calculation? Are low or high performing children more or less likely to benefit from an intervention? What proportion of children benefit from an educational intervention?

STRATEGIES Different effect size calculation Multi-level models Cohen’s d, Hedges’ g and Glass’ Δ Outome type Post-test only, Gain score and ancova Fixed Effect Model Ordinary least squares (OLS) Adjusting for school effects Multi-level models Explore variances within, between, total Frequentist versus Bayesian approach Design specific Individual randomisation, cluster, multi-site Xiao Z., Kasim, A., Higgins, S.E. (2016) Same Difference? Understanding Variation in the Estimation of Effect Sizes from Educational Trials International Journal of Educational Research 77: 1-14 http://dx.doi.org/10.1016/j.ijer.2016.02.001 Xiao Z., Higgins, S.E. & Kasim, A., (under review) Lord’s paradox revisited (post/gain comparisons)

KEY FINDINGS 17 EEF projects Four analytic models Results converge in larger, unproblematic trials Point estimates and estimates of precision vary when trials are problematic Results tend to diverge if ICC ≥ 0.2 and there are few clusters/schools MLM total variance ‘most conservative’ model (wider Confidence Intervals) Bayesian with vague priors give similar results as frequentist methods but more precise (narrower Credible Intervals)

CHALLENGES FACED Identifying the same data used by evaluators The impact of the different randomization approach Matching study design and analysis Impact of the different randomization factors and other covariates

CONCLUSION Overview of other analyses are presented below: Variation Cumulative quantiles Complier average causal effect (CACE) Internal validity Permutation p-values Software R package ‘eefAnalytics’ https://cran.r-project.org/web/packages/eefAnalytics/eefAnalytics.pdf Local influence index SD change Bootstrap confidence intervals