A randomised controlled trial to improve writing quality during the transition between primary and secondary school Natasha Mitchell, Research Fellow Hannah Buckley, Statistician York Trials Unit, University of York

Co-authors: Ainsworth, H. , Heaps, C. , Hewitt, C. , Jefferson, L Co-authors: Ainsworth, H., Heaps, C., Hewitt, C., Jefferson, L., Torgerson, C., & Torgerson, D. Funded by: Education Endowment Foundation Intervention delivery partner: Calderdale Excellence Partnership Obviously with thanks to the EEF who have funded these evaluations

Protocol available at: http://educationendowmentfoundation.org.uk/uploads/pdf/Transitions_-_Calderdale.pdf Full report available at: http://educationendowmentfoundation.org.uk/uploads/pdf/EEF_Evaluation_Report_-_Improving_Writing_Quality_-_May_2014_v2.pdf Obviously with thanks to the EEF who have funded these evaluations

Overview EEF Transitions Round: Individually Randomised (Discover) Evaluate 3 different writing interventions with aim to improve children’s writing skills during transition from primary school to secondary school Individually Randomised (Discover) Split Plot Design (Exeter) Cluster Randomised (Calderdale) Part of the EEF transitions round. All randomised designs evaluating different writing interventions, the specific nature of each situation has lead to 3 different trial designs. Begin with a simple individually randomised trial, then a slightly more novel split plot design and a cluster trial.

Background Many children leave primary school without achieving a Level 4 or above in writing (Dept of Education, 2013) The intervention was aimed particularly at children: from disadvantaged backgrounds not achieving a ‘secure Level 4’ in year 6 Not reaching Level 4 at the end of KS2 linked to progress in secondary school (Dept of Education, 2014)

Background Using memorable experiences and the Self-Regulated Strategy Development (SRSD) programme. SRSD strategy developed in America Writing process model involves: Plan Draft Edit Revise Key aspects include self monitoring & goal setting to provide pupils with ownership over improving their writing. SRSD uses ‘heuristics’ which provide scaffolding of structures and devices that aid the composition of argumentative writing – in particular planning – which can include examining a question, brainstorming, organising and sequencing ideas and evaluating. The effect sizes of individual studies, largely undertaken in North America, were very large with estimates in some instances exceeding 1 standard deviation between the intervention and control groups. The approach was created in the United States in the 1990s. It has been designed, and is suitable, for children who are aged between 8 and 14 (Andrews et al., 2006) so it is appropriate for children passing through the transition from primary to secondary school. Nevertheless, despite these promising results in a North American context it was important to establish whether or not such an intervention would be effective in a British setting. For this reason the intervention was appropriate for an efficacy trial in English schools.

Research Question What is the effectiveness of the Improving Writing Quality programme compared with “business as usual” on the writing skills of participating children? The second trial is a pragmatic cluster randomised controlled trial based in Calderdale in West Yorkshire using an intervention to improve writing quality. We started this trial in the summer term of 2013 The intervention team - Calderdale Excellence Partnership, based in Calderdale West Yorkshire The research question for this trial is…..

Design Pragmatic cluster randomised controlled trial. Approximately 24 schools (feeding into 3 secondary schools) randomised on a 1:1 ratio to intervention and control groups Primary School unit of randomisation Class based intervention, therefore individual randomisation not appropriate We have utilised opt-out consent  Year 6 pupils received a letter about the project & parents needed to return a form to the school if they did not want their child’s data to be shared with the evaluation team  Year 7 pupils will also receive a letter about the project & about opting out – we have had to send this letter as there will be pupils in Year 7 at each of the secondary schools who were not at participating primary schools & all Year 7 pupils would be completing the Progress in English Test (except those below level 3)

Design Intervention Group The intervention was offered to all pupils in Year 6 in 2013. Pupils predicted to achieve Level 3 or an insecure Level 4 in English (based on teacher assessment) continued to receive intervention in Year 7 (Autumn term). Memorable experiences for pupils in Year 6. Pupils used SRSD programme to embed these experiences into their writing. Professional development for primary and secondary school teachers in key elements of the writing intervention SRSD programme (including discussion, modelling and planning). Control Group Business as Usual in 2013. Primary schools trained in SRSD in 2014. Intervention: Improving Writing Quality Year 6 Level 3/4 based on teacher assessment in their Year 6 autumn term Having memorable experiences to help with writing Teachers at intervention schools have been trained to use Self-Regulated Strategy Development Control Group: Business as usual – however the pupils would be usually taught to increase recruitment / reduce attrition of primary schools, design uses a wait-list, where schools allocated to control will be offered the intervention after the trial (next academic year 2014 – summer)

Design Hosted a information meeting to explain intervention and evaluation, hosted by delivery partner and evaluators in February 2013. Recruited: 23 primary schools; 3 secondary schools. Randomised primary schools in March 2013. Post test conducted in December 2013 after transition to secondary school. Information Meeting  Opportunity to explain, randomisation and the importance of a control group, trial procedures, answer any questions

Outcomes Outcome measure Progress in English (PiE) 11: Second Edition Long Form (LF) Test, GL Assessment Primary outcome Combined score on extended writing tasks Secondary outcome Combined score on reading tasks and combined score on the spelling and grammar tasks

Analysis Methods Cluster trial Pupils from same primary school transitioning to different secondary schools Cross-classified multilevel model used to account for this Adjustment was made for baseline predicted KS2 writing level alongside other predictors of attainment (gender, FSM status, EAL status, month of birth) Using principles of ITT

CONSORT diagram

Eligible for inclusion in the primary analysi at randomisation Eligible pupils At randomisation 209 control 223 intervention Eligible for inclusion in the primary analysi at randomisation

Not attending participating secondary school Eligible pupils Not attending participating secondary school 209 control 223 intervention - 79 control - 69 intervention

Attending participating secondary school Eligible pupils Attending participating secondary school 209 control 223 intervention - 79 control - 69 intervention 130 control 154 intervention

Missing primary outcome Eligible pupils Missing primary outcome 209 control 223 intervention - 79 control - 69 intervention 130 control 154 intervention - 11 control - 12 intervention

Included in primary analysis Eligible pupils Included in primary analysis 209 control 223 intervention - 79 control - 69 intervention 130 control 154 intervention - 11 control - 12 intervention 119 control 142 intervention 261 total

Results Significant increase of 2.53 marks in writing score (n=261, p=0.002) for struggling writers in the intervention group compared with the control group. Effect size = 0.74 (95% CI: 0.26 to 1.22) Subgroup analysis on pupils eligible for FSM (n=86) showed a larger effect size in these pupils Effect size = 1.60 (95% CI: 0.21 to 2.98) BUT no significant differential effect between pupils eligible for free school meals (FSM) and those not

Results Significant increase of 2.53 marks in writing score (n=261, p=0.002) for struggling writers in the intervention group compared with the control group. Effect size = 0.74 (95% CI: 0.26 to 1.22) Subgroup analysis on pupils eligible for FSM (n=86) showed a larger effect on these pupils Effect size = 1.60 (95% CI: 0.21 to 2.98) No significant differential effect between pupils eligible for FSM and those not (p=0.69)

Results There was no evidence of a difference between struggling writers in the randomised groups in terms of secondary outcomes: Reading n=196, decrease of 0.31 marks (p=0.72) ES=-0.09, 95% CI: -0.59 to 0.41 Spelling and grammar n=254, decrease of 0.44 marks (p=0.50) ES= -0.13, 95% CI: -0.51 to 0.25

Results There was no evidence of a difference between struggling writers in the randomised groups in terms of secondary outcomes: Reading n=196, decrease of 0.31 marks (p=0.72) ES = -0.09, 95% CI: -0.59 to 0.41 Spelling and grammar n=254, decrease of 0.44 marks (p=0.50) ES = -0.13, 95% CI: -0.51 to 0.25

Results No significant effect on non-struggling writers with respect to any outcome: Writing n=181, increase < 0.01 marks (p=1.00) ES≈ 0.00, 95% CI: -0.45 to 0.45 Reading n=148, decrease of 0.39 marks (p=0.59) ES= -0.12, 95% CI: -0.58 to 0.33 Spelling and grammar n=182, increase of 0.10 marks (p=0.89) ES= 0.04, 95% CI: -0.49 to 0.56

Results No significant effect on non-struggling writers with respect to any outcome: Writing n=181, increase < 0.01 marks (p=1.00) ES ≈ 0.00, 95% CI: -0.45 to 0.45 Reading n=148, decrease of 0.39 marks (p=0.59) ES= -0.12, 95% CI: -0.58 to 0.33 Spelling and grammar n=182, increase of 0.10 marks (p=0.89) ES= 0.04, 95% CI: -0.49 to 0.56

Strengths & Limitations Study conducted to CONSORT standards. Used blind marking of test papers. Good relationship between CEP & schools therefore had high completion. Analysis plan produced a priori. Analysis accounted for clustering. Followed principles of intention to treat.

Strengths & Limitations During the trial there was ‘intervention development’. Anglicising the intervention. Refresher training to secondary schools incorporated ‘lessons learnt’ and adaptations. Imperfect implementation in secondary schools. Based in single geographical area. From ongoing discussions with CEP throughout the trial period, it was clear that a process of ‘intervention development’ occurred over the trial period and is ongoing. CEP gave the primary schools flexibility to develop and implement some components of the intervention as each saw fit following the training received in March 2013. CEP also noted the ‘American-ness’ of the intervention and felt it needed to be to be Anglicised for the English classroom context. This was especially true of the ’self-talk’ element which CEP reported many of the primary schools had reframed into a more British approach. CEP delivered their own refresher training to secondary schools in September, and incorporated ‘lessons learnt’ and adaptations following the intervention period in primary schools. CEP have set up a working group to discuss how the intervention should be modified and adapted for implementation next year. despite imperfect implementation (one teacher at one secondary school received training but taught a control class; at another secondary school, two teachers taught intervention classes without having had the initial SRSD training)

Conclusions This intervention appears to be highly effective at improving the writing skills of struggling children despite being used for a relatively short period of time. No evidence of an impact on reading or spelling and grammar scores for struggling writers No evidence of any impact for non-struggling writers

What next? Larger-scale effectiveness RCT of the intervention to confirm findings of this trial. Multiple geographical areas to be included. Developing an ‘Anglicised’ version for use in primary schools. Given the large effect size and positive feedback from teachers, we would recommend a much larger-scale effectiveness RCT of the intervention in undertaken to confirm the findings of this trial. The rationale behind this is as follows. First, the trial was geographically confined and focused on ‘transfer’ or ‘transition’ pupils. Second, Calderdale teachers adapted the intervention to suit their context and a formal evaluation of training teachers in an ‘Anglicised’ version would be appropriate. We consider that a trial of an Anglicised version of SRSD aimed at all primary school pupils with KS2 English assessments as the main outcome would be useful.

Thank you Questions?