Statistical Analysis Plan review Dr Ben Styles CStat Head of NFER’s Education Trials Unit

Summary Outcomes from SAP reviewer day The process of review Improvements to analysis The future

SAP reviewer day United in our goal Avoided being prescriptive Tension between template and guidance SAP template finalised

Process Timing agreed: within three months of randomisation Reviewed by EEF, an independent SAP reviewer and the delivery partner Initial review anonymous but possibility of teleconference if an issue needs further discussion

Resolved issues Drop-out post randomisation but before knowledge of allocation The use of raw scores rather than age-standardised scores A suitable approach to Bayesian analysis Only unique report tables need be specified in the SAP Gain scores should only be used if ANCOVA is also conducted

Unresolved issues Effect size – total variance with or without covariates Models for multi-site (randomised block) trials e.g. random/fixed effects Re-randomisation and inclusion of covariates in analysis models

Examples of improvements – open science Exact pre-specification of outcomes NPD outcomes identified that might no longer be available Exact pre-specification of the model to be used Clarity over approach to missing data Greater scrutiny of discrepancies between protocol and SAP To enable REPLICATION

Examples of improvements - quality of analysis Complier Average Causal Effect analysis Reporting of effect sizes in multi-armed trials Better analysis models

Specific example Multiple measures of the same outcome

Pupils accumulate; same outcome

SAP solutions Wait until you have all the data; or Use a stopping rule, noting this also affects the calculation of confidence intervals for the final analysis (e.g. Pocock, 1977) Pocock SJ (1977). Group sequential methods in the design and analysis of clinical trials. Biometrika 64 (2): 191–9

Same pupils; same outcome; repeated measures

SAP solutions Pre-specify a single primary time-point and analyse this in the usual way; or Carry out a repeated measures analysis and pre-specify the model coefficient that will determine intervention effectiveness

Conclusions and future HARKing has been banished SAP template could be more detailed Analysis syntax could be pre-written We need more reviewers!