1. A randomised controlled trial to improve writing quality during the transition between primary and secondary school
Natasha Mitchell, Research Fellow
Hannah Buckley, Statistician
York Trials Unit, University of York

2. Co-authors: Ainsworth, H. , Heaps, C. , Hewitt, C. , Jefferson, L
Co-authors: Ainsworth, H., Heaps, C., Hewitt, C., Jefferson, L., Torgerson, C., & Torgerson, D. Funded by: Education Endowment Foundation Intervention delivery partner: Calderdale Excellence Partnership
Obviously with thanks to the EEF who have funded these evaluations

3. Protocol available at:
Full report available at:
Obviously with thanks to the EEF who have funded these evaluations

4. Overview EEF Transitions Round: Individually Randomised (Discover)
Evaluate 3 different writing interventions with aim to improve children’s writing skills during transition from primary school to secondary school
Individually Randomised (Discover)
Split Plot Design (Exeter)
Cluster Randomised (Calderdale)
Part of the EEF transitions round.
All randomised designs evaluating different writing interventions, the specific nature of each situation has lead to 3 different trial designs.
Begin with a simple individually randomised trial, then a slightly more novel split plot design and a cluster trial.

5. Background
Many children leave primary school without achieving a Level 4 or above in writing (Dept of Education, 2013)
The intervention was aimed particularly at children:
from disadvantaged backgrounds
not achieving a ‘secure Level 4’ in year 6
Not reaching Level 4 at the end of KS2 linked to progress in secondary school (Dept of Education, 2014)

6. Background
Using memorable experiences and the Self-Regulated Strategy Development (SRSD) programme.
SRSD strategy developed in America
Writing process model involves:
Plan
Draft
Edit
Revise
Key aspects include self monitoring & goal setting to provide pupils with ownership over improving their writing.
SRSD uses ‘heuristics’ which provide scaffolding of structures and devices that aid the composition of argumentative writing – in particular planning – which can include examining a question, brainstorming, organising and sequencing ideas and evaluating.
The effect sizes of individual studies, largely undertaken in North America, were very large with estimates in some instances exceeding 1 standard deviation between the intervention and control groups.
The approach was created in the United States in the 1990s. It has been designed, and is suitable, for children who are aged between 8 and 14 (Andrews et al., 2006) so it is appropriate for children passing through the transition from primary to secondary school.
Nevertheless, despite these promising results in a North American context it was important to establish whether or not such an intervention would be effective in a British setting. For this reason the intervention was appropriate for an efficacy trial in English schools.

7. Research Question
What is the effectiveness of the Improving Writing Quality programme compared with “business as usual” on the writing skills of participating children?
The second trial is a pragmatic cluster randomised controlled trial based in Calderdale in West Yorkshire using an intervention to improve writing quality.
We started this trial in the summer term of 2013
The intervention team - Calderdale Excellence Partnership, based in Calderdale West Yorkshire
The research question for this trial is…..

8. Design Pragmatic cluster randomised controlled trial.
Approximately 24 schools (feeding into 3 secondary schools) randomised on a 1:1 ratio to intervention and control groups
Primary School unit of randomisation
Class based intervention, therefore individual randomisation not appropriate
We have utilised opt-out consent  Year 6 pupils received a letter about the project & parents needed to return a form to the school if they did not want their child’s data to be shared with the evaluation team
 Year 7 pupils will also receive a letter about the project & about opting out – we have had to send this letter as there will be pupils in Year 7 at each of the secondary schools who were not at participating primary schools & all Year 7 pupils would be completing the Progress in English Test (except those below level 3)

9. Design Intervention Group
The intervention was offered to all pupils in Year 6 in 2013.
Pupils predicted to achieve Level 3 or an insecure Level 4 in English (based on teacher assessment) continued to receive intervention in Year 7 (Autumn term).
Memorable experiences for pupils in Year 6.
Pupils used SRSD programme to embed these experiences into their writing.
Professional development for primary and secondary school teachers in key elements of the writing intervention SRSD programme (including discussion, modelling and planning).
Control Group
Business as Usual in 2013.
Primary schools trained in SRSD in 2014.
Intervention: Improving Writing Quality
Year 6
Level 3/4 based on teacher assessment in their Year 6 autumn term
Having memorable experiences to help with writing
Teachers at intervention schools have been trained to use Self-Regulated Strategy Development
Control Group: Business as usual – however the pupils would be usually taught
to increase recruitment / reduce attrition of primary schools, design uses a wait-list, where schools allocated to control will be offered the intervention after the trial (next academic year 2014 – summer)

10. Design
Hosted a information meeting to explain intervention and evaluation, hosted by delivery partner and evaluators in February 2013.
Recruited: 23 primary schools; 3 secondary schools.
Randomised primary schools in March 2013.
Post test conducted in December 2013 after transition to secondary school.
Information Meeting  Opportunity to explain, randomisation and the importance of a control group, trial procedures, answer any questions

11. Outcomes
Outcome measure
Progress in English (PiE) 11: Second Edition Long Form (LF) Test, GL Assessment
Primary outcome
Combined score on extended writing tasks
Secondary outcome
Combined score on reading tasks and combined score on the spelling and grammar tasks

12. Analysis Methods Cluster trial
Pupils from same primary school transitioning to different secondary schools
Cross-classified multilevel model used to account for this
Adjustment was made for baseline predicted KS2 writing level alongside other predictors of attainment (gender, FSM status, EAL status, month of birth)
Using principles of ITT

13. CONSORT diagram

14. Eligible for inclusion in the primary analysi at randomisation
Eligible pupils
At randomisation
209 control
223 intervention
Eligible for inclusion in the primary analysi at randomisation

15. Not attending participating secondary school
Eligible pupils
Not attending participating secondary school
209 control
223 intervention
- 79 control
- 69 intervention

16. Attending participating secondary school
Eligible pupils
Attending participating secondary school
209 control
223 intervention
- 79 control
- 69 intervention
130 control
154 intervention

17. Missing primary outcome
Eligible pupils
Missing primary outcome
209 control
223 intervention
- 79 control
- 69 intervention
130 control
154 intervention
- 11 control
- 12 intervention

18. Included in primary analysis
Eligible pupils
Included in primary analysis
209 control
223 intervention
- 79 control
- 69 intervention
130 control
154 intervention
- 11 control
- 12 intervention
119 control
142 intervention
261 total

19. Results
Significant increase of 2.53 marks in writing score (n=261, p=0.002) for struggling writers in the intervention group compared with the control group.
Effect size = 0.74 (95% CI: 0.26 to 1.22)
Subgroup analysis on pupils eligible for FSM (n=86) showed a larger effect size in these pupils
Effect size = 1.60 (95% CI: 0.21 to 2.98)
BUT no significant differential effect between pupils eligible for free school meals (FSM) and those not

20. Results
Significant increase of 2.53 marks in writing score (n=261, p=0.002) for struggling writers in the intervention group compared with the control group.
Effect size = 0.74 (95% CI: 0.26 to 1.22)
Subgroup analysis on pupils eligible for FSM (n=86) showed a larger effect on these pupils
Effect size = 1.60 (95% CI: 0.21 to 2.98)
No significant differential effect between pupils eligible for FSM and those not (p=0.69)

21. Results
There was no evidence of a difference between struggling writers in the randomised groups in terms of secondary outcomes:
Reading n=196, decrease of 0.31 marks (p=0.72) ES=-0.09, 95% CI: to 0.41
Spelling and grammar n=254, decrease of 0.44 marks (p=0.50) ES= -0.13, 95% CI: to 0.25

22. Results
There was no evidence of a difference between struggling writers in the randomised groups in terms of secondary outcomes:
Reading n=196, decrease of 0.31 marks (p=0.72) ES = -0.09, 95% CI: to 0.41
Spelling and grammar n=254, decrease of 0.44 marks (p=0.50) ES = -0.13, 95% CI: to 0.25

23. Results
No significant effect on non-struggling writers with respect to any outcome:
Writing n=181, increase < 0.01 marks (p=1.00) ES≈ 0.00, 95% CI: to 0.45
Reading n=148, decrease of 0.39 marks (p=0.59) ES= -0.12, 95% CI: to 0.33
Spelling and grammar n=182, increase of 0.10 marks (p=0.89) ES= 0.04, 95% CI: to 0.56

24. Results
No significant effect on non-struggling writers with respect to any outcome:
Writing n=181, increase < 0.01 marks (p=1.00) ES ≈ 0.00, 95% CI: to 0.45
Reading n=148, decrease of 0.39 marks (p=0.59) ES= -0.12, 95% CI: to 0.33
Spelling and grammar n=182, increase of 0.10 marks (p=0.89) ES= 0.04, 95% CI: to 0.56

25. Strengths & Limitations
Study conducted to CONSORT standards.
Used blind marking of test papers.
Good relationship between CEP & schools therefore had high completion.
Analysis plan produced a priori.
Analysis accounted for clustering.
Followed principles of intention to treat.

26. Strengths & Limitations
During the trial there was ‘intervention development’.
Anglicising the intervention.
Refresher training to secondary schools incorporated ‘lessons learnt’ and adaptations.
Imperfect implementation in secondary schools.
Based in single geographical area.
From ongoing discussions with CEP throughout the trial period, it was clear that a process of ‘intervention development’ occurred over the trial period and is ongoing. CEP gave the primary schools flexibility to develop and implement some components of the intervention as each saw fit following the training received in March CEP also noted the ‘American-ness’ of the intervention and felt it needed to be to be Anglicised for the English classroom context. This was especially true of the ’self-talk’ element which CEP reported many of the primary schools had reframed into a more British approach.
CEP delivered their own refresher training to secondary schools in September, and incorporated ‘lessons learnt’ and adaptations following the intervention period in primary schools. CEP have set up a working group to discuss how the intervention should be modified and adapted for implementation next year.
despite imperfect implementation (one teacher at one secondary school received training but taught a control class; at another secondary school, two teachers taught intervention classes without having had the initial SRSD training)

27. Conclusions
This intervention appears to be highly effective at improving the writing skills of struggling children despite being used for a relatively short period of time.
No evidence of an impact on reading or spelling and grammar scores for struggling writers
No evidence of any impact for non-struggling writers

28. What next?
Larger-scale effectiveness RCT of the intervention to confirm findings of this trial.
Multiple geographical areas to be included.
Developing an ‘Anglicised’ version for use in primary schools.
Given the large effect size and positive feedback from teachers, we would recommend a much larger-scale effectiveness RCT of the intervention in undertaken to confirm the findings of this trial. The rationale behind this is as follows. First, the trial was geographically confined and focused on ‘transfer’ or ‘transition’ pupils. Second, Calderdale teachers adapted the intervention to suit their context and a formal evaluation of training teachers in an ‘Anglicised’ version would be appropriate. We consider that a trial of an Anglicised version of SRSD aimed at all primary school pupils with KS2 English assessments as the main outcome would be useful.

29. Thank you
Questions?