1. Experimental evaluation in education Professor Carole Torgerson School of Education, Durham University, United Kingdom carole.torgerson@durham.ac.uk International workshop Social Experiments and Innovation: a new paradigm for public intervention? Barcelona, September 26 th 2013

2. Some landmark experimental evaluations in education in the US and the developing world Cambridge Somerville youth experiment (US) – Early intervention to reduce ‘juvenile delinquency’ Tennessee class-size experiment (US) – Class-size reduction intervention in early years education PROGRESA experiment (Mexico) – Rural anti-poverty intervention Balsakhi experiment (India) – Teaching assistants intervention in literacy and numeracy

3. Why social experiments? Powerful tool for evaluating impacts in education: – Simple to understand – Simple to analyse – Ethical to use random allocation in the presence of resource scarcity – Scientifically the most rigorous evaluative approach – Deal with known and unknown confounding

4. Experimental evaluation in education in an age of austerity Educational interventions that do not work waste money or worse Educational budgets across Europe under strain due to poor economic conditions Imperative that the most effective and cost- effective interventions are adopted by policy makers Randomised experiments offer the best evidence for efficacy, effectiveness and cost-effectiveness

5. Some examples of expensive educational interventions that don’t work (UK) Financial incentives (Brooks et al, 2008) ICT and spelling (Brooks et al, 2006) Nurse numeracy intervention (Ainsworth et al, 2011)

6. Example of educational intervention that does work: Every Child Counts Every Child Counts (ECC) was previous UK government’s flagship policy to help children (age 7) at risk in numeracy Expensive one-to-one tutoring intervention (Numbers Count) delivered each day over one school term (12 weeks) Randomised experiment commissioned to establish effectiveness and cost-effectiveness (Torgerson et al, 2013a; Torgerson et al, 2013b) Pre-post test evaluation (undertaken by developer) demonstrated large effect size (>1 SD) and cost- effectiveness (using weak design)

7. The ECC evaluation design Three linked randomised experiments: – ECC (Numbers Count) vs. ‘business as usual’ (Trial 1): 44 schools – ECC (Numbers Count) pairs vs. ECC one-to-one (Trial 2): 15 schools – ECC triplets (Numbers Count) vs. ECC one-to-one (Trial 3): 7 schools Process evaluation: – Random sample of schools – Implementation and delivery

8. Design of Trial 1 12 children in 44 schools eligible for ‘Numbers Count’ intervention Numeracy test (Sandwell test) (pre-test) at beginning of autumn term (administered by teachers) Random allocation of 12 children to term of delivery: autumn, spring or summer: ‘waiting list’ design Intervention group: autumn children Control group: spring and summer children Numeracy test (Progress in Maths test) after 12 weeks (administered by independent testers) (post-test) Simple analysis: compare the mean numeracy post-test score of intervention children with mean numeracy score of control children and conclude whether ‘Numbers Count’ is more effective than ‘business as usual’ Rigorous design: excludes some alternative explanations for results

9. Design features that minimise alternative explanations for results Large sample size: excludes chance finding Randomisation: intervention and control groups are equivalent at start so design controls for history, maturation, regression to the mean, selection bias Intervention and control conditions are both numeracy interventions and both last for 30 minutes each day for 12 weeks: the comparison is a ‘fair’ one Independent ‘blinded’ testing: eliminates possibility of tester bias

10. Results Intervention Group Control Group PIM 6 (0-30)15.8 N = 144 14.0 N = 440

11. Results

12. Intervention Group Control Group Effect Size 95% Confidence Interval PIM 6 (0-30)15.8 (4.9) N = 144 14.0 (4.5) N = 440 0.33 (0.12 to 0.53)

13. Results ECC better than business as usual (0.33 SD) but expensive No evidence that one-to-one was better than ECC in pairs or triplets One-to-one not cost effective – more cost- effective to deliver in pairs or triplets

14. Design limitations: Generalisability ECC schools were identified: by policy-makers/funders of programme - education policy ‘roll out’ in England, i.e., schools in disadvantaged areas Ideally, a random sample of all secondary schools in England should have been approached and asked to take part

15. Design limitations: Intervention One-to-one teaching with intervention children being withdrawn from classroom Problem of attribution: was effect due to Numbers Count intervention? Or to one-to-one teaching? Or to withdrawal from classroom? Design could have included additional one-to-one arm

16. Design limitations: ‘Contamination’/’spill over’ effects Children withdrawn from usual classroom teaching – may have benefited remaining children Teachers using intervention have applied it to some control children. Instead of randomising individual children, design could have randomised by school (cluster randomisation, where school is the cluster) to avoid these problems.

17. Design limitations: Long term effect s Wait list design prevented long term follow-up; effects may have ‘washed out’ soon after intervention was finished. Could have used cluster randomisation; Could have recruited children above threshold and randomised these to intervention or long term follow-up; All options (above) rejected by funder.

18. Conclusions Design and conduct warranted conclusion Numbers Count (as delivered) more effective than usual classroom teaching BUT because of design limitations couldn’t answer some really important questions These questions could have been answered if a different experimental design had been used: cluster randomisation (randomisation of schools), long-term follow-up (control group that didn’t receive intervention); one to one control group (literacy or other numeracy)

19. 3 EEF ‘transitions’ projects Background: – Interventions aimed at children from disadvantaged backgrounds and those struggling to reach national key stage writing standards. Primary outcome measure: – combined score on the 2 writing tasks within the Progress in English test (GL assessment) Secondary outcomes: – scores on the reading, spelling and grammar components of the Progress in English test (GL assessment)

20. DISCOVER Research question: – What is the effectiveness of the Discover summer writing workshop intervention compared with a ‘business as usual’ control group on the writing abilities of participating children? Individually randomised experiment

21. Discover: Trial Design Diagram

22. Improving Writing Quality Intervention: Calderdale Research question: – What is the effectiveness of the Improving Writing Quality programme compared with ‘business as usual’ on the writing skills of participating children? Pragmatic cluster randomised design

23. Calderdale: Trial Design Diagram

24. Exeter Grammar for Writing Intervention Research questions: 1. What is the effectiveness of the whole class Grammar for writing intervention compared with a ‘business as usual’ control group on writing skills of participating children? 2. What is the effectiveness of the whole class Grammar for writing intervention plus additional small group intervention compared with a ‘business as usual’ control group on writing skills of participating children? 3. What is the effectiveness of the whole class Grammar for writing intervention plus additional small group intervention compared with the whole class Grammar for writing intervention only on writing skills of participating children? Partial split plot design

25. Exeter: Trial Design Diagram

27. Answers research question 1: effectiveness of whole class intervention compared with “business as usual” on writing skills?

28. Exeter: Trial Design Diagram

29. Answers research question 2: Effectiveness of whole class intervention plus additional small group intervention compared with “business as usual” control group on writing skills?

30. Exeter: Trial Design Diagram

31. Answers research question 3: effectiveness of whole class intervention plus additional small group intervention compared with whole class intervention only on writing skills?

32. Some challenges in promoting and undertaking experimental evaluations in education Resistance from within research community Lack of political will Lack of funding opportunities for individual experiments and for capacity building Lack of capacity (experience and expertise) to undertake rigorous experiments Potential for conflict of interest (Developer of intervention) Recruitment and retention

33. References Ainsworth, H., Torgerson, D., Torgerson, C. et al (2011) Computer-based instruction for improving student nurses’ general numeracy: Is it effective? Two RCTs, Educational Studies Brooks, G., Burton, M., Coles, P., Miles, J., Torgerson, C., Torgerson, D. (2008) Randomised controlled trial of incentives to improve attendance at adult literacy classes, Oxford Review of Education, 34(4) Brooks, G., Miles, J.N.V., Torgerson, C.J. and Torgerson, D.J. (2006) Is an intervention using computer software effective in literacy learning? A randomised controlled trial, Educational Studies, 32(1) Torgerson, C.J., Wiggins, A., Torgerson, D.J., Ainsworth, H., Hewitt, C. Every Child Counts: Testing policy effectiveness using a randomized controlled trial, designed, conducted and reported to CONSORT standards, Journal of Research in Mathematics Education, July 2013 Torgerson, C.J., Wiggins, A., Torgerson, D.J., Ainsworth, H., Hewitt, C. The effectiveness of an intensive individual tutoring programme (Numbers Count) delivered individually or to small groups of children: A randomised controlled trial, Effective Education, Apr., 2013