The English RCT of ‘Families and Schools Together’ Pippa Lord and Dr Ben Styles NFER’s Education Trials Unit RCTs in the Social Sciences 11th Annual Conference 7th-9th September 2016 Public

About this presentation Outline the FAST programme FAST trial main design features Integrating FAST programme and trial design Analytical considerations (sample size, power, dilution) Quasi experimental design (QED) Conclusions for trials with this kind of ToC

Rationale FAST is ready for an effectiveness trial in England: the existing evidence base on softer outcomes from the US, the results of meta-analyses of the effect of parental engagement on attainment and the whole-school nature of FAST as applied in England.

The FAST programme in England The programme Founded in US by Prof. Lynn MacDonald in 1988 UK license holders – Middlesex University Delivered by Save the Children (SCUK) in UK primary schools Phase 1 – partner training, family recruitment, Hub model Phase 2 – 8-week programme (2.5 hrs a week), graduation Phase 3 – FASTworks (22 months)

FAST trial main features Effectiveness trial Cluster randomised (schools) – between 60+60 and 80+80 intervention/control Primary outcome – attainment in English and maths at Key Stage 1 (EYFS as baseline) Secondary outcomes – measures from the ‘Strengths and Difficulties Questionnaire’ (pre-, mid- and longer-term) QED – primary outcome, FAST sub-group and matched control sub-group Process evaluation

Integrating FAST and design How to recruit schools? ‘Normal’ SCUK recruitment (School Agreements adapted for trial) Sign up consent from schools and all Y1 families (opt out) When to randomise? Before family recruitment (families’ participation part of the programme) Before training (considerable effort by schools and local partners) Who to measure? FAST runs across EYFS and KS1; trial measures Y1 Invitations to families as normal, aim for 1/3 of Y1 in each school SDQs anonymous (opt out required for this trial) How to deliver programme and trial to 80 schools? Blocked, termly design for delivery, randomisations and data collections

FAST programme delivery flow School agreement School recruitment Local partner training Family recruitment Phase 1 Week 0 (pre-questionnaires) Graduation Week 9 (post-questionnaires) 8-week cycle 22 months Phase 3 feedback FASTworks

FAST trial delivery flow (per Block) KS1 and NFER end-point SDQs Randomise School agreement School consent School recruitment Local partner training Family recruitment Phase 1 Week 0 (pre-questionnaires) Graduation Week 9 (post-questionnaires) 8-week cycle 22 months Phase 3 feedback FASTworks Y1 parent consent, NFER pre-SDQs, pupil data, school pro-forma NFER mid-point-SDQs

Blocking Three overall blocks of between 40 and 60 schools (with sub-blocks 1a, 1b, 2a, 2b, 3a, 3b for randomisation and delivery) 1st block in September 2015 2nd block in January 2016 3rd block in April 2016 Summer 2017 for attainment measures across all three blocks

SCUK recruit between 120 and 160 schools (158 achieved, in 3 blocks) Parental opt-out consent for data sharing (KS1 data) Upload of pupil UPNs and school baseline pro-forma, to NFER secure portal SDQ baseline administration by teachers RANDOMISATION FAST programme Control (business as usual) Observations of training and of programme Mid-point SDQ Mid-point SDQ, £500 Evaluation visits post-programme FASTworks data collection, and follow-up KS1 testing, SDQ, pro-forma £1000, feedback KS1 testing, SDQ, pro-forma, feedback

SCUK recruit between 120 and 160 schools (158 achieved, in Blocks) Parental opt-out consent for data sharing (KS1 data) Upload of pupil UPNs and school baseline pro-forma, to NFER secure portal SDQ baseline administration by teachers RANDOMISATION FAST programme Control (business as usual) Observations of training and of programme Mid-point SDQ Mid-point SDQ, £500 Evaluation visits post-programme FASTworks data collection, and follow-up KS1 testing, SDQ, pro-forma, feedback KS1 testing, SDQ, pro-forma £1000, feedback

Process evaluation

Sample size FAST is considered a whole-school intervention Recruitment of families to attend the sessions is part of the intervention Spill-over to other families is intended Sample size therefore takes into account an estimated ‘dilution’ of the effect

Power curves for overall effect

But… We require FAST families to experience ES= 0.45-0.50 (assuming no spill-over) if a third volunteer This is unlikely for the primary outcome (although EEF toolkit cites one study of parental involvement with ES=0.6) A slightly better quasi-experiment than normal

QED Families in control group that would have received FAST if randomised to intervention Using PSM, select control Year 1 pupils on the basis of FSM, EYFS etc. (to ensure ‘common support’) Multi-level modelling using same background factors including pre-test (EYFS) Post-test (KS1; independently marked)

Sensitivity analysis for QED Targeting of families for FAST is not done using eligibility criteria We therefore miss factor(s) that lead to a family being selected and agreeing An unmeasured variable changing its correlation with the outcome by 0.1 can ‘remove’ an effect of 0.2 (Coe, 2009) Given the equivalence of the schools, will the QED results be more robust here?

Conclusions We are at the more difficult end of the RCT ‘continuum of feasibility’ Concessions in design and delivery Blocking Data security concerns The RCT will still allow robust conclusions about FAST effectiveness at the school level The ‘enhanced’ QED will allow conclusions about FAST effectiveness at the pupil level, with caveats

Acknowledgements Save the Children UK Education Endowment Foundation NFER Research Operations team: Dave Hereward and Asma Ullah Centre for Children and Families Research, Loughborough University

Contact us: Dr. Ben Styles (Head of NFER’s Education Trials Unit) b.styles@nfer.ac.uk Pippa Lord (FAST Trial Manager) p.lord@nfer.ac.uk