Narrowing the evaluation gap Session 2.B. Beyond Randomised Controlled Trials – How non-experimental methods can contribute to educational research 2018 EEF Evaluators’ Conference Narrowing the evaluation gap #EEFeval18 @EducEndowFound

University of Warwick and Institute for Fiscal Studies Claire Crawford University of Warwick and Institute for Fiscal Studies

What do we mean by non-experimental methods? Methods that take advantage of “natural” experiments which produce close to random assignment of treatment (conditional on covariates) (Or that can account for some types of unobserved heterogeneity) e.g. RDD (sharp/fuzzy, i.e. partnered with IV) DiD (Panel data methods?) (OLS/matching not typically included in this group)

How might nonexperimental methods be used? Initial piloting of policies (DiD) e.g. EMA, free school meals Geographic rollout of policies (DiD) e.g. expansion of free nursery places Administrative rules used for implementation of policies (RDD) e.g. date of birth cut-offs determining entitlement to start nursery/school e.g. rules to regulate class sizes in Israel Key assumption required (vs. RCTs) is that these “natural” experiments mimic random assignment; credibility relies on this assumption

University of Westminster Richard Dorsett University of Westminster

A role for non-experimental methods in EEF research Where an RCT is impractical/impossible perhaps logistical challenges programmes already underway/scaling up Where some early/quick/rough evidence is needed How much of a change is this? Will involve a different approach to impact estimation May require different data collection May leave process study unchanged (if prospective, not if retrospective) not clear what happens to costs

RCT vs non-experimental estimators RCTs gold standard in the hierarchy of evidence Face validity – easy to understand/communicate/persuade Non-experimental estimators Cover a range of approaches… … which vary in their underlying assumptions (some credible, others less so). Often lower face validity (are data rich enough? is instrument valid?)… but no issue with randomisation bias (possible external validity advantage). In practice Practical issues can reduce purity of RCT design (drop-out, attrition, fidelity) Non-experimental estimators may form part of RCT analysis (e.g. CACE, imputation) spectrum

Going forward - some considerations Non-experimental estimators need careful consideration of underlying assumptions How to assess non-experimental estimators’ security alongside RCT security When is a careful non-experimental estimator better than a flawed RCT? Take best practice elements from RCTs Protocol/SAP – need justification for choice of approach Power calculations – not traditionally a focus of nonexperimental analysis Blinding – where feasible Greater role for sensitivity analysis? Understanding more about non-experimental estimators in this space: Revisit completed RCTs using nonexperimental estimators Understanding general suitability of commonly-used data (NPD), e.g.: Stability of school trends over time Situations where selection on observed characteristics is plausible

Table Discussions 1.- How can we assess the quality and credibility of non-experimental impact estimates? How credible are non-experimental estimates? Is there are hierarchy of evidence? How can we assess the validity and credibility of non-experimental estimates? How can we assess the stability of results in one study? What is the role of using multiple specifications and methods? How can we combine those results? 2.- How can we ensure that non-experimental estimates are comparable – to each other and to experimental estimates? Is it possible to specify a prospective protocol/SAP for non-experimental methods? To what extent will these overlap or differ by method? What is the level of detail that should be expected in prospective documents? Should they include robustness checks and falsification tests? Should prospective documents and reporting follow any specific standards? If yes, which ones? Under what circumstances should analyses that deviate from those set out in prospective documents be considered? Are these circumstances the same as for RCTs or different?