School of Education Archive Analysis: On EEF Trials Adetayo Kasim, ZhiMin Xiao, and Steve Higgins

School of Education EEF Conference 2 Outline Design Methods – Simple Randomised Trial (SRT) – Multi-Site Trial (MST) – Cluster Randomised Trial (CRT) Estimation Framework – Frequentist versus Bayesian Approach Discussion Introduction

School of Education EEF Conference 3 Introduction This presentation is based on the data released by the FFT in December We present here 15 effect sizes from 11 randomised trials, which involve three design specifications: Design# Trials# Interv. SRT710 MST23 CRT22 The goal of this presentation is to facilitate discussion around analysis of educational trials using different analytical models.

School of Education EEF Conference 4 Simple Randomised Trials These trials randomised children using simple randomisation without acknowledging the nested structure of pupils within schools. We analysed the data to investigate if: – there is any difference between Cohen’s d and Hedges’ g. – simple randomisation results in zero correlation within schools, i.e., Intra-Cluster Correlation (ICC) equals zero.

School of Education EEF Conference 5 Simple Randomised Trials

School of Education EEF Conference 6 Multi-Site Trials These trials performed randomisation within schools in order to account for differences in effect between schools and the nested structure of pupils within schools. We analysed the data to investigate: – if randomisation within schools removes ICC. – fixed versus random effects using multilevel modelling (MLM).

School of Education EEF Conference 7 Multi-Site Trials No InteractionWith Interaction Interv.Fixed (95% CI)MLM (95% CI)FixedMLM (95% CI) 1Within0.32 (0.07, 0.70)0.31 (0.10, 0.66)-0.31 (0.11, 0.84) Total-0.28 (0.08, 0.51)-0.28 (0.07, 0.50) ICC Within0.40 (0.18, 0.75)0.40 (0.21, 0.74)-0.41 (0.26, 1.00) Total-0.32 (0.13, 0.51)0.32 ( ) ICC Within0.08 (-0.07, 0.24)0.08 (-0.08, 0.24)-0.08 (-0.10, 0.25) Total-0.08 (-0.08, 0.24)-0.07 (-0.10, 0.23) ICC

School of Education EEF Conference 8 Multi-Site Trials One advantage of the fixed effect model is that it does not require a minimum number of schools per treatment arm. However, it relies on a strong assumption of no treatment-by-school interaction, an assumption we cannot verify because most studies are not powered enough to detect such interactions.

School of Education EEF Conference 9 Multi-Site Trials The multilevel model is more robust than the fixed effect model because treatment-by-school interaction is specified as random effects. It will always result in a single effect size estimation per outcome. However, MLM may be unsuitable for studies with small number clusters. For Gaussian data, a minimum of five clusters per treatment arm has been recommended for MLM.

School of Education EEF Conference 10 Cluster Randomised Trials Randomisation to treatment is implemented at school level. All pupils in the same school receive the same intervention. We used different sources of variability to calculate the probabilities of observing a certain effect size given the data we happened to observe.

School of Education EEF Conference 11 Cluster Randomised Trials

School of Education EEF Conference 12 Cluster Randomised Trials Using total variability is the most conservative approach and least likely to result in false positives compared to the use of within or between variability. Within variance is sometimes preferred to ensure comparability across studies. However, it could also lead to false positives if there is substantial between- school variability. Between variability is very prone to false positives. This should be used with caution!!

School of Education EEF Conference 13 Frequentist versus Bayesian Methods There is a general concern about inference based on non-random samples due to the validity of standard errors. This is perhaps one reason for many to choose Bayesian inference over classical frequentist methods. We compared results from three approaches, namely, Bayesian, frequentist with non-parametric bootstrapping, and classical frequentist with standard errors.

School of Education EEF Conference 14 Interv.Bayesian (95% CRI)Bootstrap (95% QCI)Freq.SE (95% CI) (-0.00, 0.14)0.07 (0.00, 0.14)0.07 (-0.13, 0.28) (0.02, 0.53) 0.28 (0.08, 0.51) 0.28 (-0.01, 0.57) (0.10, 0.54) 0.32 (0.13, 0.51) 0.32 (-0.00, 0.64) (0.16, 1.22)0.69 (0.42, 0.92)0.67 (0.07, 1.28) (-0.08, 0.25) 0.08 (-0.08, 0.24) 0.08 (-0.11, 0.27) Frequentist versus Bayesian Methods Effect sizes from MST and CRT using Bayesian and frequentist methods

School of Education EEF Conference 15 Discussion Should simple randomisation be used in Educational trials? Fixed or random effect model for multisite trials? Should bootstrapped confidence interval or Bayesian credible interval be used to quantify uncertainty in effect size estimation? Total or within variance for effect size calculation?

School of Education Thank You Adetayo Kasim, ZhiMin Xiao, and Steve Higgins