Assessment (What every teacher needs to know) @ProfCoe Assessment (What every teacher needs to know) Robert Coe, Durham University researchED, 10 Sept 2016

htpps://respond.cc WARNING: This talk contains interactivity (or, as we call it, assessment) To play along, go to: htpps://respond.cc Enter session key for each section

#AssessBetter

What every teacher needs to know Reliability Validity Formative use Monitoring Retrieval effects Evaluating impact Accountability

I Reliability

Which of these are part of reliability? respond.cc 418941 Which of these are part of reliability? Yes No Replicability: would you get the same score again? Precision: how accurate is the score? Independence of item selection: would a different choice of test questions alter the score? Independence of markers: would a different marker change the score? Standardisation of contexts: would changing the assessment context (eg different room, time of day, invigilator, instructions, etc) make a difference?          

Which of these would make an assessment more reliable? respond.cc 892461 yes, provided ... possibly, it depends on ... no, unless ... Make it less valid Give candidates more time Get two markers to mark and take the average Add more items Increase the number of marks available but keep the number of items/tasks the same Replace poorly discriminating items with better ones Narrow the range (content) of what is assessed Limit the range of assessment methods used (eg don't mix MC & essay, practical & written ) Use a 'comparative judgement' process

Why does reliability matter? We may over- (or under-) interpret small differences between scores small changes over time responses to single questions We may fail to account for regression to the mean We may design assessments that are less reliable than they could be

How accurately can we measure progress? respond.cc 623985 Suppose you use annual assessments to estimate the progress a learner has made. How reliable does the assessment have to be before the typically expected annual progress exceeds the measurement error of a ‘change’ score? (Enter a number between 0 and 1) What proportion of GCSE units reach this level of reliability? Required reliability is 0.97. No GCSE units achieve this: highest found by Bramley and Dhawan was 0.934 (Table 1.1, p12) Bramley, T. and Dhawan, V. (2010) Estimates of Reliability of Qualifications. Ofqual Reliability Programme / Cambridge Assessment https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/377717/2011-03-16-estimates-of-reliability-of-qualifications.pdf

Bramley, T. and Dhawan, V. (2010) Estimates of Reliability of Qualifications. Ofqual Reliability Programme / Cambridge Assessment https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/377717/2011-03-16-estimates-of-reliability-of-qualifications.pdf (Bramley & Dhawan, 2010)

Hinge questions How much does a correct/incorrect answer affect your estimate of whether a student has ‘mastered’ the concept? Case 1: You know nothing about the student before asking

Hinge questions How much does a correct/incorrect answer affect your estimate of whether a student has ‘mastered’ the concept? Case 2: You have good prior attainment data

Hinge questions How much does a correct/incorrect answer affect your estimate of whether a student has ‘mastered’ the concept? Case 3: You have good feedback about the student’s learning of this concept

Is social class more important than early ability? “Rich, thick kids do better than poor, clever children” (Michael Gove 2010) Feinstein, L (2003) ‘Inequality in the early cognitive development of British children in the 1970 cohort’. Economica, 70, 277, 73-97. Michael Gove, Evidence to the Commons Education Select Committee, 27 July 2010. Feinstein (2003)

Or is it just regression to the mean? Jerim J and Vignoles A (2011) ‘The use (and misuse) of statistics in understanding social mobility: regression to the mean and the cognitive development of high ability children from disadvantaged homes’, DoQSS working paper 11–01, London: Institute of Education (Jerrim & Vignoles 2011)

II Validity

If you are a member of each of these groups, which kind of assessment should you prefer? Teacher assessment Objective tests respond.cc 536115 Pupils with EAL Pupils with professional parents Chinese-British pupils African Caribbean girls High-attaining pupils Very low-attaining pupils Pupils with ‘challenging behaviours’ Black pupils with a white teacher    Harlen W (2004) A systematic review of the evidence of reliability and validity of assessment by teachers used for summative purposes. In: Research Evidence in Education Library. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. [http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=116&language=en-US] Bennett et al., 1993 Peter Tymms: ‘Teachers show bias to pupils who share their personality’. The Conversation 25 Feb 2015 https://theconversation.com/teachers-show-bias-to-pupils-who-share-their-personality-38018 Burgess, S. and Greaves, E (2009) Test Scores, Subjective Assessment and Stereotyping. Centre for Market and Public Organisation, Bristol University. Working Paper No. 09/221. http://www.bristol.ac.uk/media-library/sites/cmpo/migrated/documents/wp221.pdf of Ethnic Minorities Lee ??? (cited in Metzler Ouazad, A. (2014). Assessed by a teacher like me: Race and teacher assessments. Education, 9(3), 334-372.

respond.cc 363005 Which of these do you calculate for assessments used in your school/classroom? Always, for every assessment Mostly, for key assessments At least once a year Occasionally Never Item-total correlations Correlations between assessed scores/levels and baseline (eg EYFSP, KS2, MidYIS, CAT) Correlations between assessed scores/levels in different subjects at the same time Correlations between assessed scores/levels and subsequent achievement

How well do the following assessments credit or incentivise desirable learning and teaching? respond.cc 700207 EYFSP KS2 Reading GCSE science practicals Art GCSE Maths A level BTEC First Diploma in Business GCSE MFL speaking & listening Very well; excellent alignment Quite well; broadly aligned Partially; some alignment but not close Poorly; limited or no alignment Negatively; reverse alignment

Why does validity matter? Bias: Different assessment approaches can (dis)advantage different groups Convergent/discriminant validity Construct under-representation: assessing only part of what matters Construct-irrelevant variance: accidentally measuring things that should not be there Alignment between what is assessed and what is worth teaching