ResearchED Time for a reality check? Robert Coe ResearchED Research Leads Network Day, 13 December 2014

∂ ResearchED  Engagement with educational research –“ …the start of something wonderful” –Or just another fashion?  And what does it mean anyway?  What is a reasonable hope/expectation for research and evidence to make a difference to practice? 2

∂ Dimensions of great teaching 1.Quality of instruction 2.(Pedagogical) content knowledge 3.Classroom management / behaviour / control 4.Classroom climate / relationships / expectations 5.Beliefs (theory) about subject, learning & teaching 6.Wider professional elements: collegiality, PD, stakeholder relationships 3

∂ The problem with ‘What works?’ What you want to know: What will work for me What research/evidence tells you: Something seems to have ‘worked’ for researchers 4 Research is always equivocal Is it clear what it is? Past tense Does it transfer to ‘real life’ in my context? What outcomes are used / valued?

∂  A choice between two or more approaches that are –Well-defined (clear exactly what you must do – even if one approach is ‘continue normal practice’) –Feasible (cost, time, acceptability, ethics) –Replicable (anyone who wants to can do them)  A context that is generalisable –The choice and context have to be similar enough to those faced by others  Agreement about criteria for success –What outcome matters, and how measured? 5 Requirements for evidence-based practice

∂ Requirements for evidence-based practice Are the same as for prediction or giving advice Teacher: I want my pupils to learn to read. Should I use synthetic phonics or real books? Researcher: Each context is unique, research can’t tell you that Teacher: OK, should I give them books at all, or just do dancing? Researcher: Each context is unique, research can’t tell you that 6

∂ Why might research help (1) Knowledge of research  Some pedagogical practices are better supported by evidence than others –Aim for what is actually effective, not just fashionable  Some interventions are better supported by evidence than others –Go with the best bets, based on evidence  Evidence can help in challenging & developing teachers’ existing theories and beliefs about learning –Effective professional learning requires this (Timperley, et al, 2007) 7

∂ Should we challenge teacher beliefs?  Beliefs about –Why some students do not learn Ability, social background, effort, motivation Their fault vs my fault –What does it mean to learn (mathematics) Connectionist / Transmission / Discovery orientations (Askew et al, 1997) –How does feedback work Correction vs accommodation 8

∂ Impact vs cost Cost per pupil Effect Size (months gain) £0 0 8 £1000 Meta-cognitive Peer tutoring Early Years 1-1 tuition Homework (Secondary) Mentoring Summer schools After school Aspirations Performance pay Teaching assistants Smaller classes Ability grouping Most promising for raising attainment May be worth it Small effects / high cost Feedback Phonics Homework (Primary) Collaborative Small gp tuition Parental involvement Individualised learning ICT Behaviour Social

∂ Why might research help (2) R esearch Mindset  Monitoring ongoing effectiveness –“Know thy impact” (Hattie)  Evaluating impact of changes –What is working for you  Critical perspective –Show me the evidence –Devil’s Advocate –“Tolerance of ambiguity” (Sherrington) 10

∂ ‘Distributed Research’ “Central to the philosophy is the concept of ‘distributed research’, the idea that the recipients of the feedback (i.e. teachers in schools and colleges) are themselves active researchers in the process, analysing and interpreting the data, rather than simply passive recipients. The research is seen as a collaborative process. The CEM Centre has skills in test design, the creation of monitoring systems and takes advantage of economies of scale in its management of the process. It also provides the basic structure of the analysis, using the collective dataset. After that, however, interpretation is a local matter and cannot be done from the centre, though considerable support is offered to help schools and colleges to interpret the data for themselves. The important thing is that the data are fed back to the smallest unit of management that can take responsibility for acting on the feedback (Fitz-Gibbon, 1996, p. 161).” 11 Tymms & Coe, 2003

∂  Clear, well defined, replicable intervention  Good assessment of appropriate outcomes  Well-matched comparison group EEF DIY Evaluation Guide Key elements of good evaluation What could you evaluate?

∂ Do we know a good lesson when we see one? 13

∂ Lesson Observation 1.Two teachers observe the same lesson, one rates it ‘Inadequate’. What is the probability the other will agree? a) 10% b) 40% c) 60% d) 80% 2.An observer judges a lesson ‘Outstanding’. What is the probability that pupils are really making sustained, outstanding progress? a) 5% b) 30% c) 50% d) 70% 14

∂ ‘Improvement’ often isn’t  School/college would have improved anyway –Volunteers/enthusiasts improve: misattributed to intervention –Chance variation (esp. if start low)  Poor outcome measures –Perceptions of those who worked hard at it –No robust assessment of pupil learning  Poor evaluation designs –Weak evaluations more likely to show positive results –Improved intake mistaken for impact of intervention  Selective reporting –Dredging for anything positive (within a study) –Only success is publicised (Coe, 2009, 2013)