1. Statistical Analysis Plan review
Dr Ben Styles CStat Head of NFER’s Education Trials Unit

2. Summary Outcomes from SAP reviewer day The process of review
Improvements to analysis
The future

3. SAP reviewer day United in our goal Avoided being prescriptive
Tension between template and guidance
SAP template finalised

4. Process Timing agreed: within three months of randomisation
Reviewed by EEF, an independent SAP reviewer and the delivery partner
Initial review anonymous but possibility of teleconference if an issue needs further discussion

5. Resolved issues
Drop-out post randomisation but before knowledge of allocation
The use of raw scores rather than age-standardised scores
A suitable approach to Bayesian analysis
Only unique report tables need be specified in the SAP
Gain scores should only be used if ANCOVA is also conducted

6. Unresolved issues
Effect size – total variance with or without covariates
Models for multi-site (randomised block) trials e.g. random/fixed effects
Re-randomisation and inclusion of covariates in analysis models

7. Examples of improvements – open science
Exact pre-specification of outcomes
NPD outcomes identified that might no longer be available
Exact pre-specification of the model to be used
Clarity over approach to missing data
Greater scrutiny of discrepancies between protocol and SAP
To enable REPLICATION

8. Examples of improvements - quality of analysis
Complier Average Causal Effect analysis
Reporting of effect sizes in multi-armed trials
Better analysis models

9. Specific example
Multiple measures of the same outcome

10. Pupils accumulate; same outcome

11. SAP solutions Wait until you have all the data; or
Use a stopping rule, noting this also affects the calculation of confidence intervals for the final analysis (e.g. Pocock, 1977)
Pocock SJ (1977). Group sequential methods in the design and analysis of clinical trials. Biometrika 64 (2): 191–9

12. Same pupils; same outcome; repeated measures

13. SAP solutions
Pre-specify a single primary time-point and analyse this in the usual way; or
Carry out a repeated measures analysis and pre-specify the model coefficient that will determine intervention effectiveness

14. Conclusions and future
HARKing has been banished
SAP template could be more detailed
Analysis syntax could be pre-written
We need more reviewers!