1. EEF Archive analysis overview
Steve Higgins and Adetayo Kasim
With support from ZhiMin Xiao, Nasima Akhter and Ewoud De Troyer
@profstig
EEF Conference
1st June, 2017

2. RESEARCH QUESTIONS
How consistent are the results from the different analytic approaches or Methods?
What is the impact of non-compliance on effect size calculation?
Are low or high performing children more or less likely to benefit from an intervention?
What proportion of children benefit from an educational intervention?

3. STRATEGIES Different effect size calculation Multi-level models
Cohen’s d, Hedges’ g and Glass’ Δ
Outome type
Post-test only, Gain score and ancova
Fixed Effect Model
Ordinary least squares (OLS)
Adjusting for school effects
Multi-level models
Explore variances within, between, total
Frequentist versus Bayesian approach
Design specific
Individual randomisation, cluster, multi-site
Xiao Z., Kasim, A., Higgins, S.E. (2016) Same Difference? Understanding Variation in the Estimation of Effect Sizes from Educational Trials International Journal of Educational Research 77:
Xiao Z., Higgins, S.E. & Kasim, A., (under review) Lord’s paradox revisited (post/gain comparisons)

4. KEY FINDINGS 17 EEF projects Four analytic models
Results converge in larger, unproblematic trials
Point estimates and estimates of precision vary when trials are problematic
Results tend to diverge if ICC ≥ 0.2 and there are few clusters/schools
MLM total variance ‘most conservative’ model (wider Confidence Intervals)
Bayesian with vague priors give similar results as frequentist methods but more precise (narrower Credible Intervals)

5. CHALLENGES FACED Identifying the same data used by evaluators
The impact of the different randomization approach
Matching study design and analysis
Impact of the different randomization factors and other covariates

6. CONCLUSION Overview of other analyses are presented below: Variation
Cumulative quantiles
Complier average causal effect (CACE)
Internal validity
Permutation p-values
Software
R package ‘eefAnalytics’
Local influence index
SD change
Bootstrap confidence intervals