1. Evaluating Teacher Performance Daniel Muijs, University of Southampton

2. Evaluating teacher performance Long a part of performance management Increased interest in light of PRP Leaves key questions: –How can we do it? –Can we do it reliably and validly?

3. Aims of this presentation Look at different ways of evaluating teacher performance from an international perspective Look at the reliability and validity of these methods Look at whether developing a reliable system of evaluating is even possible, and, if so, what it would take to develop

4. A bit of methodology… Validity: are we measuring what we want to measure Reliability: –Do we get the same (order of) results if we measure at different times? –Do our items measure on thing?

5. Evaluating teacher performance Most common methods worldwide: –Using pupil outcomes –Observing classroom practise Other methods used: –Pupil feedback –Balanced approach

6. Using pupil outcomes Key arguments: –This is what really matters –It is what you achieve, not how you do it that is important –Provides right incentives to teachers –Benefits pupils

7. Using pupil outcomes – some key issues What outcomes to measure? For what can teachers be held accountable? How to measure these? In practise, many desired outcomes are not actually measured (e.g. British Values) Not all possible outcomes equally influenced by teachers and schools, school and classroom effects mainly exist for cognitive outcomes (Vignoles & Meschi, 2010)

8. Using pupil test scores to measure teacher performance Therefore, usually only cognitive outcomes studied, in particular test scores. These are –Reliably measurable –Reflect strongly desired outcomes of education –Are directly influenced by teaching In England, these are the basis of accountability at the school level (‘league tables’).

9. Using teacher test scores to measure performance As measures of teacher performance very problematic, as teacher level explains no more than 30% of variance in such measures (Chapman et al, forthcoming). Other determinants: –Social background –Gender –Ability –Prior attainment Therefore, many advocate use of ‘value-added’ measures

10. Value-added assessment Value-added assessment measures progress over time (so current attainment compares to prior attainment) Contextual value-added also controls for factors such as social background Value added models are the statistical methods used to calculate them. In England: used as accountability measure at school level under previous government

11. Value –added assessment at teacher level In other countries: used to evaluate teachers, e.g. Tennessee (TVAAS), New York City Gates Foundation’s MET project also studied this Teachers receive a score based on how much value they have added to their class, in some cases this linked to Performance-Related Pay (Muijs & Chahine, 2014)

12. Value-Added Assessment at Teacher Level Advantages: –Takes prior attainment and context into account –Teachers have strongest influence on gains in attainment (Sammons et al, forthcoming) –Established statistical models

13. Value-Added Assessment at Teacher Level Disadvantages –Requires very extensive testing regime –Models depend on what variables you put in –Perverse incentives –Stability over time? –Uncertainty in measurement: the issue of confidence intervals

15. Using observation to measure teacher performance Value-added, though it has received a lot of interest, is still only used in minority of countries. More common: classroom observation Common in England through Ofsted and its influence in schools Also done in many other systems across the world Many different systems and frameworks (e.g. Dutch ICALT model, Ofsted framework, US Danielson model)

16. Using observation to measure teacher performance Advantages: –Under actual control of teachers –Can be done reliably –Immediate results –Formative as well as summative –Can be done at school level

17. Using observation to measure teacher performance Some significant issues, however: –What do we observe (e.g. Ofsted’s ever-changing framework)? –What CAN we observe (Learning? Progress? Time-on- task?) –Can we observe reliably?

18. What should we observe? Should be based on evidence and research, e.g. –Time on task –Direct Instruction –Self-regulated learning (see Muijs et al, 2014) Should show some stability Needs to take subject and age-specificity into account (differential effectiveness)

19. What can we observe and can we do so reliably? Not everything is observable! Issues of reliability –Observer bias and halo effects, interobserver reliability –Changes in behaviour Reliability only achieved through: –Proper training of observers –Valid and reliable (and tested) observation schedules –A sufficient number of observations! –Not overestimating reliability of classifications Again, beware unintended consequences

20. Student feedback Questionnaires to students Used in some systems, e.g. parts of US Main method in UKHE Also studied in MET Again, many different survey instruments exist

21. Student feedback Advantages: –Students most direct observers of teaching –Cheap and convenient method –Strong correlation with external observations Disadvantages: –Bias –Age dependent –Possible perverse incentives

22. Balanced approach MET project recommendations: combine VA measures, observation and student surveys (Bill & Melinda Gates Foundation, 2013) Seems a balanced and sensible approach, but –In MET studies correlation between observations and value-added is modest –Expensive –Combined measure still leaves unexplained variance

23. Conclusion Evaluating teacher performance is not straightforward, if the system is to be reliable and fair No one method will work Balanced approach needs to include broader framework to encompass variety of teacher roles Observation and student feedback useful components

24. Thanks for listening! D.Muijs@soton.ac.uk @ProfDanielMuijs