Trials and Tribulations Peter Tymms York 2008

Outline Introduction Background issues –Impact of educational policies –Level of intervention –Monitoring and the possibilities for RCTs –Studying cause and studying possibilities Three interventions: –Small scale teaching intervention –Large scale simplicity –Large scale complexity Conclusions

The Impact (Failure) of Educational Policies One example –In England The National Literacy Strategy cost £500 million. –Its impact on reading levels has not been detected.

Level of Interventions Most systematic reviews (of RCTs) deal with pupil level interventions. But policy level strategies deal with school level interventions To change a teacher's classroom approach takes 30 hours of INSET

Monitoring

Establishing cause and establishing what is possible Monitoring alone does not establish cause –(MLMs can delude “explain”, “effect” etc) Regression discontinuity & time series can. –But they may not tell us what is possible RCTs can, and –To know how to improve education we must try to improve education.

First Intervention Cross age peer tutoring in Chemistry

Design etc 14 2 nd year students paired with 1 st years Random assignment Two half hour sessions Post test

Results and lessons ES = 0.3 (n.s.) Lessons –Much harder than usual teaching –A learning experience –Qualitative – Mandy’s experience –Telling others and its impact Colleagues Policy makers

Second Intervention ADHD: Naming and Informing (include in NICE)

Attention Deficit Hyperactivity Disorder Very common special need Children with ADHD fall behind academically Often unpopular Social difficulties Behavioural problems

Scores at age 5: DSM(IV)

A Dilemma Feeding names back to schools One psychologist: “You must not do this” Another: “You must take action”

Interventions Randomly assign to schools IdentificationNo Identification Booklet No Booklet

Interventions Randomly assigned to Districts NothingInformation packs Conference & packs

Numbers 24 Districts 2040 schools 60,000 pupils Over 2 years

Results LEA Interventions –No impact on any of the outcome measures

Results All Pupils For booklet –Higher Y2 reading score Identification –No impact

Results Pupils with ADHD characteristics For booklet –More positive attitudes –Better behaviour in Year 2 For Identification –No impact BUT book + identification –Negative impact on maths and reading

Teachers’ Quality of Life Your perception of work: Relaxing Stressful Enjoyable Unpleasant On top of things Overwhelmed The behaviour of the pupils in your class: No problems Challenging

Teachers’ Quality of Life Your perception of work: Relaxing Stressful Enjoyable Unpleasant On top of things Overwhelmed The behaviour of the pupils in your class: No problems Challenging Teachers more positive if given information books

Major Issues Implementation –The intervention was very clear –But schools using the booklet ? –Taking notice of feedback? Cheap with low impact Very cost effective For the future: –Implementation: Aim for wider, deeper impact

PostScript A follow up study by Sayal with CEM showed: –Naming appears to have long term problems Lesson –Once you carried out and RCT long term follow up because a possibility.

Third Intervention Peer Tutoring

Peer Learning Good research pedigree from individual RCTs –“Low hanging fruit” Boruch Effect Sizes of 0.4 to 0.8 in small trials For attainment & other outcomes For tutors and tutees

The Overall Plan Run Peer Learning in schools for two years in reading and mathematics. Use monitoring data to check impact

We know that.. Peer Learning is effective But Can a whole Authority change together? And Which is best – Cross-age or Same-age? – Mix or separate subjects? – Intensive or light?

Allocation to forms of peer learning 120 schools initially agreed to be randomly assigned Factorial design Separation of allocation & analysis from intervention

Issues Immediately after we had sent out allocation letters one of the allocations was changes! Some schools and pupils changed No effect after one year! Maths tutoring did not involve maths!

Preliminary Results

MLMs

For younger children (8-9) Cross age (tutee only) Intensive Reading & Mathematics ES = 0.2

For older (10-11) Cross age (tutors only) Light & intensive Reading & mathematics ES = 0.2

Issues Practicalities and logistics –Setting the whole thing up Implementation –Arrangements for Peer Learning –What was being taught –Fidelity The effects are small but the potential is great.

Finally Establishing cause is not sufficient; –We need to know what we can change. Large scale monitoring with ready access to the data is fairly new We can carry out RCTs within a monitoring framework. Professionals can see the value in RCTs We need: –Qualitative data, monitoring data and RCTS Meaningful change to educational systems may not come from making the system more efficient but from changing its nature.

Thank you Peter Tymms York 2008

References Fitz-Gibbon, C. T. (1996). The Design And Use Of Monitoring Education: Indicators, Quality and Effectiveness. London: Cassell. Fitz-Gibbon, C. T. (1992). Peer and Cross-Age Tutoring. In M. C. Alkin (Ed.), Encyclopedia of Educational Research (Sixth ed., pp ). New York: Macmillan Publishing Company. Tymms, P. B. (1989). Peer Tutoring with 'A' level Chemistry Students. Paired Learning, 5. Tymms, P., & Coe, R. (2003). Celebration of the Success of Distributed Research with Schools: the CEM Centre, Durham. British Educational Research Journal, 29(5), Tymms, P., & Merrell, C. (2006). The impact of screening and advice on inattentive, hyperactive and impulsive children. European Journal of Special Needs Education, 21(3), Tymms, P. Merrell, C. Heron, T., Jones, P., Albone, S., and Henderson, B. (2008 in press) The Importance of Districts School Effectiveness and School Improvement Tymms, P., Merrell, C. and Coe, R. (2008) Educational Policies and Randomized Cotrolled Trials The Psychology of Education Review, Vol. 32, No. 2, September 2008