Statistical Analysis Plans EEF Evaluators’ Conference 2016 Dr Ben Styles Head of NFER’s Education Trials Unit

The replicability crisis Many research findings are false or exaggerated (Ioannidis, 2014) Suggestions to make more research findings true: replication culture, reproducibility practices, better statistical methods and standardisation of definitions and analyses Ioannidis JPA (2014) How to Make More Published Research True. PLoS Med 11(10): e doi: /journal.pmed e doi: /journal.pmed

Education trials Potentially even more vulnerable: median effect size of 0.1 (Sanders and Chonaire, 2015); very little replication; diversity of approaches to design and analysis Distribution of effect sizes in education research – Sanders and Chonaire (2015)

EEF approach Strategy to maximise replicability EEF approach Replication cultureIndependent evaluation…wider availability of trial data and analysis code…occasional re-trial Better statistical methodsAnalysis guidance Standardisation of analysesR-package and forthcoming peer review of SAPs

What happens without a SAP? All levels of randomisation Which outcome variable? Which covariates? How much missing data prompts MI/sensitivity analyses? What type of MI/sensitivity analysis? Comparisons when >2 arms? Interim analyses >1 follow-up?

What happens without a SAP? Randomised block/multi- site Cluster randomised Fixed or random effect for school? School by treatment interaction? School means/MLM/GEE/Huber- White standard errors

What happens without a SAP? = possible cherry picking!

Why SAPs To limit ‘cherry picking’ To distinguish between planned and exploratory analyses To provide the opportunity for developer/external challenge before the analysis is carried out Analysis represents a small percentage of the overall trial budget yet the decisions a statistician makes are critical Clinical trials have SAPs

Opposition ConcernRepost SAPs kill innovationSAPs do not preclude exploratory analyses Unnecessary bureaucratic hurdleAnalysis has to be written up anyway; EEF template Damaging inflexibility e.g. what if a less biased/more efficient analysis method is proposed in the interim Both analyses are performed and it is explained why one method is superior Green DP (2015) Pre-Analysis Plans: Pros and Cons of Limiting Discretion in the Analysis of RCTs. RCTs in the Social Sciences Conference 2015: Keynote Address

Suggested headings for EEF SAP template Introduction Trial design Randomisation Sample size calculations Extent of follow-up Primary outcome Secondary outcomes Protocol changes Primary outcome analysis (ITT) Effect size calculation Missing data analysis strategy On-treatment/per-protocol analysis Secondary outcome analyses Subgroup analyses Additional analyses Interim analyses Analysis of multiple follow-up Sample representation School/student characteristics Test reliability MDES calculation on the basis of parameters seen Baseline effect size for analysed groups (re attrition) Student characteristics of analysed groups (re attrition) List of tables and figures Format of tables and figures

SAP sign-off EEF (through peer review) Developer

Next steps Write your SAP at least three months before analysis EEF will publish as an addendum to the protocol Volunteer with EEF to review SAPs (small payment, Durham workshop in autumn) In the future: analysis syntax pre-written

Workshop on analysis

NFER provides evidence for excellence through its independence and insights, the breadth of its work, its connections, and a focus on outcomes. National Foundation for Educational Research The Mere, Upton Park Slough, Berks SL1 2DQ T: F: E: