1. The English RCT of ‘Families and Schools Together’
Pippa Lord and Dr Ben Styles NFER’s Education Trials Unit
RCTs in the Social Sciences
11th Annual Conference
7th-9th September 2016
Public

2. About this presentation
Outline the FAST programme
FAST trial main design features
Integrating FAST programme and trial design
Analytical considerations (sample size, power, dilution)
Quasi experimental design (QED)
Conclusions for trials with this kind of ToC

3. Rationale
FAST is ready for an effectiveness trial in England: the existing evidence base on softer outcomes from the US, the results of meta-analyses of the effect of parental engagement on attainment and the whole-school nature of FAST as applied in England.

4. The FAST programme in England
The programme
Founded in US by Prof. Lynn MacDonald in 1988
UK license holders – Middlesex University
Delivered by Save the Children (SCUK) in UK primary schools
Phase 1 – partner training, family recruitment, Hub model
Phase 2 – 8-week programme (2.5 hrs a week), graduation
Phase 3 – FASTworks (22 months)

5. FAST trial main features
Effectiveness trial
Cluster randomised (schools) – between and intervention/control
Primary outcome – attainment in English and maths at Key Stage 1 (EYFS as baseline)
Secondary outcomes – measures from the ‘Strengths and Difficulties Questionnaire’ (pre-, mid- and longer-term)
QED – primary outcome, FAST sub-group and matched control sub-group
Process evaluation

6. Integrating FAST and design
How to recruit schools?
‘Normal’ SCUK recruitment (School Agreements adapted for trial)
Sign up consent from schools and all Y1 families (opt out)
When to randomise?
Before family recruitment (families’ participation part of the programme)
Before training (considerable effort by schools and local partners)
Who to measure?
FAST runs across EYFS and KS1; trial measures Y1
Invitations to families as normal, aim for 1/3 of Y1 in each school
SDQs anonymous (opt out required for this trial)
How to deliver programme and trial to 80 schools?
Blocked, termly design for delivery, randomisations and data collections

7. FAST programme delivery flow
School agreement
School recruitment
Local partner training
Family recruitment
Phase 1
Week 0 (pre-questionnaires)
Graduation
Week 9 (post-questionnaires)
8-week cycle
22 months
Phase 3 feedback
FASTworks

8. FAST trial delivery flow (per Block)
KS1 and NFER end-point SDQs
Randomise
School agreement
School consent
School recruitment
Local partner training
Family recruitment
Phase 1
Week 0 (pre-questionnaires)
Graduation
Week 9 (post-questionnaires)
8-week cycle
22 months
Phase 3 feedback
FASTworks
Y1 parent consent, NFER pre-SDQs, pupil data, school pro-forma
NFER mid-point-SDQs

9. Blocking
Three overall blocks of between 40 and 60 schools (with sub-blocks 1a, 1b, 2a, 2b, 3a, 3b for randomisation and delivery)
1st block in September 2015
2nd block in January 2016
3rd block in April 2016
Summer 2017 for attainment measures across all three blocks

10. SCUK recruit between 120 and 160 schools (158 achieved, in 3 blocks)
Parental opt-out consent for data sharing (KS1 data)
Upload of pupil UPNs and school baseline pro-forma, to NFER secure portal
SDQ baseline administration by teachers
RANDOMISATION
FAST programme
Control (business as usual)
Observations of training and of programme
Mid-point SDQ
Mid-point SDQ, £500
Evaluation visits post-programme
FASTworks data collection, and follow-up
KS1 testing, SDQ, pro-forma £1000, feedback
KS1 testing, SDQ, pro-forma, feedback

11. SCUK recruit between 120 and 160 schools (158 achieved, in Blocks)
Parental opt-out consent for data sharing (KS1 data)
Upload of pupil UPNs and school baseline pro-forma, to NFER secure portal
SDQ baseline administration by teachers
RANDOMISATION
FAST programme
Control (business as usual)
Observations of training and of programme
Mid-point SDQ
Mid-point SDQ, £500
Evaluation visits post-programme
FASTworks data collection, and follow-up
KS1 testing, SDQ, pro-forma, feedback
KS1 testing, SDQ, pro-forma £1000, feedback

12. Process evaluation

13. Sample size FAST is considered a whole-school intervention
Recruitment of families to attend the sessions is part of the intervention
Spill-over to other families is intended
Sample size therefore takes into account an estimated ‘dilution’ of the effect

14. Power curves for overall effect

15. But…
We require FAST families to experience ES= (assuming no spill-over) if a third volunteer
This is unlikely for the primary outcome (although EEF toolkit cites one study of parental involvement with ES=0.6)
A slightly better quasi-experiment than normal

16. QED
Families in control group that would have received FAST if randomised to intervention
Using PSM, select control Year 1 pupils on the basis of FSM, EYFS etc. (to ensure ‘common support’)
Multi-level modelling using same background factors including pre-test (EYFS)
Post-test (KS1; independently marked)

17. Sensitivity analysis for QED
Targeting of families for FAST is not done using eligibility criteria
We therefore miss factor(s) that lead to a family being selected and agreeing
An unmeasured variable changing its correlation with the outcome by 0.1 can ‘remove’ an effect of 0.2 (Coe, 2009)
Given the equivalence of the schools, will the QED results be more robust here?

18. Conclusions
We are at the more difficult end of the RCT ‘continuum of feasibility’
Concessions in design and delivery
Blocking
Data security concerns
The RCT will still allow robust conclusions about FAST effectiveness at the school level
The ‘enhanced’ QED will allow conclusions about FAST effectiveness at the pupil level, with caveats

19. Acknowledgements
Save the Children UK Education Endowment Foundation NFER Research Operations team: Dave Hereward and Asma Ullah Centre for Children and Families Research, Loughborough University

20. Contact us:
Dr. Ben Styles (Head of NFER’s Education Trials Unit) Pippa Lord (FAST Trial Manager)