1. TRANSFORMING EDUCATION THROUGH EVIDENCE. The Centre for Effective Education SCHOOL OF Education Conducting Educational Randomised Control Trials in Disadvantaged Community Settings Andy Biggart & Liam OHare

2. SCHOOL OF Medicine, Dentistry and Biomedical Sciences Presentation Overview 1.Introduction to research context and programmes 2.Overview of Key Results 3.Conextualising Results in Wider Literature 4.Some methodological and ethical reflections of conducting RCTs in community settings The Centre for Effective Education | SCHOOL OF Education

3. SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education Research Context Educational Research & Randomised Control Trials The Atlantic Philanthropies Some of first contemporary trials in UK and Ireland The two progammes grew out of a consultation with the local community in Tallaght (Dublin)

4. SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education Two After-schools Programmes Doodle Den Newly developed after-schools programme Aimed at 5 and 6 year olds identified as struggling readers Adopted a balanced literacy framework Delivered by teacher and youth worker/child care assistant Ran throughout the school year 3 days a week for 1.5 hours afterschool Highly structured programme with strong emphasis on professional development

5. SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education Two After-schools Programmes Mate Tricks Very similar in delivery to Doodle Den Hybrid Pro-social Behaviour Programme: Coping Power and Strengthening Families Program Aimed at 9 and 10 year olds Universally applied in schools in Tallaght Parental Components Designed to enhance childrens pro-social development, reduce anti-social behaviour, improve parenting skills and parent/child interactions

6. SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education Common Methodology Individually randomised trials Qualitative Process Evaluation Children recruited on basis of teacher referrals 3 year rolling cohort design Over 600 children allocated to either intervention or to control: business as usual Child, parent and teacher measures Outcomes measured according to pre- validated measures where possible.

7. SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education Summary Outcome Results – Doodle Den Primary Outcomes: –Statistically significant effects on both direct child and teacher assessed measures of literacy (d=0.17-0.3) Secondary Outcomes: –Statistically significant effect on childrens behaviour in regular school class (d=0.18) –No impact on attendance rates at school Exploratory Analysis –No significant differences in terms of literacy –Boys behaviour improved relative to control group –Greater attendance at the programme led to more positive outcomes

8. SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education Summary Outcome Results – Mate Tricks Primary Outcomes: –No statistically significant effects on pro-social behaviour –Two significant negative effects on anti-social behaviour (d=0.18; 0.20) Secondary Outcomes: –No statistically significant positive effects on wide range of secondary measures –Some significant negative effects in relation to child reported parenting practices Exploratory Analysis –Child programme attendance not related to outcomes –Parental engagement with the programme significant predictor of a range of positive outcomes

9. SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education Conextualising Results – Doodle Den First randomised trial of an after-school program in UK or Ireland focused on academic outcomes Mixed results of previous trials of the effectiveness on academic outcomes –e.g. 21 st Century schools U.S. national evaluation Reviews of evidence highlight complexity of the field Recent promising practice guides have concluded a need for much more research on what works in after-school settings Potential features driving positive outcomes: –Age of children; targeted at those most in need, well structured clearly defined curriculum, qualified staff with focus on professional development and fun engaging activities

10. SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education Conextualising Results – Mate Tricks Hybrid programme with some evidence of effectiveness behind the two programmes No evidence of effectiveness of Mate Tricks in improving behavioural outcomes and some negative effects apparent Some evidence of in-school programmes focusing on social and emotional learning having promising effects e.g. Incredible years However, previous reviews have also highlighted null or adverse effects in after- school contexts The current study adds to existing evidence that behavioural programmes have the potential to cause adverse effects as well as benefits r

11. SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education Some Lessons Learned: Methodological Issues Despite the critics, it is possible to implement RCTs in real world community settings. Making key stakeholders aware of key principles of RCT evaluation designs A willingness to be flexible, without undermining the core of the study and randomisation process Developing and maintaining good relationships Keeping attrition to a minimum without overburdening participants Importance of measurement and agreeing analysis plan in advance RCTs good at assessing intended outcomes but good process data should not be neglected

12. SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education Ethical Issues and RCTs Equipose Randomisation perceived as a fair way of allocation Ethical Issues may arise at any stage Potential for burden Dissemination and not spoiling the field, especially when faced with null or negative findings

13. SCHOOL OF Medicine, Dentistry and Biomedical SciencesThe Centre for Effective Education | SCHOOL OF Education EVIDENCE MATTERS!!! We intervene in childrens lives on a regular basis! We may also do damage when we intend to do good! Interventions often driven by fashion rather than sound evidence! Do we owe it to children to try and ensure that the policies they are subjected to are effective in improving their life chances? Both reports available on CEE website: www.qub.ac.uk/cee