Evaluation research findings Annamari Ylonen and Brahm Norwich Graduate School of Education, University of Exeter, UK

Outline Context: project phases 1 and 2 Research strands and main findings: Teaching strategies for MLD/SEN Lesson Study process evaluation – contexts, mechanisms and outcomes. Learning outcomes for pupils (with identified MLD). Goal Monitoring and Evaluation Conclusions

Context Phase 1 ( ) 14 schools (29 teachers): more training /support 2 terms 2-3 LS per school; 38 Lesson Studies Phase 2 ( ) 15 schools (30 teachers) : less training and support 1 term 2 LS per school; 28 Lesson Studies Main evaluation strategies in phases 1 and 2 Phase 1: the concept of MLD; teaching strategies; process evaluation Phase 2: process evaluation; ethnographic research; learning outcomes for pupils (using Goal Monitoring and Evaluation)

Pedagogic strategies in LS What pedagogic / teaching strategies do teachers report as having developed for pupils with MLD from using Lesson Study? How specialised are these pedagogic strategies: is there an MLD specialist pedagogy? Methods Questionnaire after completion of 6 months use of LS about the teaching strategies used/developed during the project for pupils with MLD, July 2011 (n=22) Analysis of LS case reports for reported strategies used

MLD pedagogy: findings Broad concept of pedagogy relevant to pupils identified as having MLD Not just about cognitive demand; nor simple idea of differentiation There were no distinct pedagogic approaches that were not relevant to others without MLD (e.g. low attainment or other SEN such as SpLD) Consistent with idea of intensified general pedagogic strategies : continuum of pedagogic strategies (Lewis and Norwich, 2000; Fletcher-Campbell, 2004)

Pedagogic approaches: is there a specialist pedagogy for MLD? Specialist/SEN Generic adapted/ Low attaining to above average Generic intensified / SEN Generic adapted/ Low attaining to above average

Process evaluation: aims and methods Informed by Realistic Evaluation (Pawson and Tilley, 1997): links between contexts, mechanisms/ processes and outcomes of LS  Why and how the LS works in schools/impact on teaching and teachers  A programme theory of LS designed of C-M-Os at teacher and school levels Survey for participating teachers at the end of Phase 1 (July 2011) (n=16) and Phase 2 (June 2012) (n=15) Semi-structured interviews about LS process and outcomes (Phase 1, n=9; Phase 2, n=6)

Lesson Study outcomes for teachers (rating scale 1-4) TEACHER LEVELMeans and SD Phase 1 (n=16) Means and SD Phase 2 (n=15) More confidence to try out novel teaching approaches in lessons 3.81 (.40)3.60 (.63) More willingness to make changes to usual teaching approaches 3.67 (.62)3.60 (.74) More open to learning from others and exposing your teaching to others in safe settings 3.47 (.91)3.60 (.74) The Lesson Study process has improved the quality of planning of your teaching 3.44 (.63)3.40 (.74) More understanding about the nature and complexity of the learning needs of pupils with MLD 3.25 (.86)3.13 (.83)

Lesson Study outcomes for schools (rating scale 1-4) SCHOOL LEVELMeans and SD Phase 1 (n=15) Means and SD Phase 2 (n=14) Attendance at meetings is regular and prioritised 2.88 (1.15)3.07 (.99) LS teachers feel supported by senior leaders in their LS work 2.80 (1.08)3.21 (.98) Senior teachers and those with SEN and CPD responsibilities encourage LS teachers 2.67 (1.23)3.29 (.91) Some teachers who are not in the LS team want join in or get involved 2.40 (1.12)3.00 (1.2)

Context: Time to undertake LS Despite funds: cover hard to timetable; teachers do not want to disturb regular teaching Some lack of support from senior leaders Process: Observation, risk taking, team-work, no blame environment Outcomes: Confidence to adjust teaching, more innovative strategies Enhance lesson planning (tailor teaching to individual needs) Benefits beyond focus pupils with MLD; other pupils benefit too C-M-Os: key points from interviews

Summary of theory of LS arising from project Context: School interest in professional learning communities Advance timetabling/ cover available, senior teacher support Processes: Team work, no blame climate, consult case pupils as part research lesson review, team observation focus on learning, risk taking about lesson planning Outcomes: Enhanced lesson planning; tailored pedagogic strategies, with wider benefits, broaden focus onto wider learning, not just external criteria; challenge conceptions about what pupils can do; wide range of pupils gains (cognitive, affective and learning approaches)

Pupil outcomes Phase 1: pupil outcomes assessed by teachers in broad terms (case reports) What is the best way to assess outcomes on pupils as a result of LS? Goal Monitoring and Evaluation (GME) introduced in phase 2 design of LS Before LS starts: teams decide on research questions and goals for each case pupil; they set ‘baseline’ and ‘expected’ levels After LS: teams decide on an ‘achieved’ level based on pupils’ progress

Example of a GME goal and levels

GME: findings 15 schools (21 Lesson Studies; 1-2 case pupils per LS). 69 Lesson Study goals set: 54% of goals were met or exceeded progress as expected = 24%; progress more than expected = 31% 46% of goals progress NOT met expected level 0% ‘No progress’ (i.e. stays at baseline level or declines)

GME: findings Three types of goals given Subject related, e.g. ‘developing written ideas independently’ Learning process related, e.g. ‘having more confidence in group activities’ Subject related and learning process, e.g. ‘being more frequently engaged in activities when emphasis is on using key vocabulary’

Level of goal attainment by type of goal None (score = 0) < expected (score=1) As expected (score =2) > Expected (score =3) TOTALMEAN Subject related Learning process SR + LP Total 032 (46%)17 (25%)20 (29%)691.83

Conclusions Research findings of this project show that: The Lesson Study process had beneficial outcomes for the teachers (e.g. Improved planning; innovative teaching strategies; team work) The LS process improved learning outcomes of pupils who had MLD (GME: all pupils made some progress from the baseline level) Outcomes for the participating schools were more complex (contextual factors critical for successful LS implementation)