1. Assessment (What every teacher needs to know)
@ProfCoe
Assessment (What every teacher needs to know)
Robert Coe, Durham University
researchED, 10 Sept 2016

2. htpps://respond.cc WARNING: This talk contains interactivity
(or, as we call it, assessment)
To play along, go to:
htpps://respond.cc
Enter session key for each section

4. #AssessBetter

5. What every teacher needs to know
Reliability
Validity
Formative use
Monitoring
Retrieval effects
Evaluating impact
Accountability

6. I
Reliability

7. Which of these are part of reliability?
respond.cc
418941
Which of these are part of reliability?
Yes No
Replicability: would you get the same score again?
Precision: how accurate is the score?
Independence of item selection: would a different choice of test questions alter the score?
Independence of markers: would a different marker change the score?
Standardisation of contexts: would changing the assessment context (eg different room, time of day, invigilator, instructions, etc) make a difference?
 
 
 
 
 

8. Which of these would make an assessment more reliable?
respond.cc
892461
yes, provided ...
possibly, it depends on ...
no, unless ...
Make it less valid
Give candidates more time
Get two markers to mark and take the average
Add more items
Increase the number of marks available but keep the number of items/tasks the same
Replace poorly discriminating items with better ones
Narrow the range (content) of what is assessed
Limit the range of assessment methods used (eg don't mix MC & essay, practical & written )
Use a 'comparative judgement' process

9. Why does reliability matter?
We may over- (or under-) interpret
small differences between scores
small changes over time
responses to single questions
We may fail to account for regression to the mean
We may design assessments that are less reliable than they could be

10. How accurately can we measure progress?
respond.cc
623985
Suppose you use annual assessments to estimate the progress a learner has made. How reliable does the assessment have to be before the typically expected annual progress exceeds the measurement error of a ‘change’ score?
(Enter a number between 0 and 1)
What proportion of GCSE units reach this level of reliability?
Required reliability is No GCSE units achieve this: highest found by Bramley and Dhawan was (Table 1.1, p12)
Bramley, T. and Dhawan, V. (2010) Estimates of Reliability of Qualifications. Ofqual Reliability Programme / Cambridge Assessment

11. Bramley, T. and Dhawan, V. (2010) Estimates of Reliability of Qualifications. Ofqual Reliability Programme / Cambridge Assessment
(Bramley & Dhawan, 2010)

12. Hinge questions How much does a correct/incorrect answer affect your estimate of whether a student has ‘mastered’ the concept? Case 1: You know nothing about the student before asking

13. Hinge questions How much does a correct/incorrect answer affect your estimate of whether a student has ‘mastered’ the concept? Case 2: You have good prior attainment data

14. Hinge questions How much does a correct/incorrect answer affect your estimate of whether a student has ‘mastered’ the concept? Case 3: You have good feedback about the student’s learning of this concept

15. Is social class more important than early ability?
“Rich, thick kids do better than poor, clever children” (Michael Gove 2010)
Feinstein, L (2003) ‘Inequality in the early cognitive development of British children in the 1970 cohort’. Economica, 70, 277,
Michael Gove, Evidence to the Commons Education Select Committee, 27 July 2010.
Feinstein (2003)

16. Or is it just regression to the mean?
Jerim J and Vignoles A (2011) ‘The use (and misuse) of statistics in understanding social
mobility: regression to the mean and the cognitive development of high ability children
from disadvantaged homes’, DoQSS working paper 11–01, London: Institute of
Education
(Jerrim & Vignoles 2011)

17. II
Validity

18. If you are a member of each of these groups, which kind of assessment should you prefer?
Teacher assessment
Objective tests
respond.cc
536115
Pupils with EAL
Pupils with professional parents
Chinese-British pupils
African Caribbean girls
High-attaining pupils
Very low-attaining pupils
Pupils with ‘challenging behaviours’
Black pupils with a white teacher 
 
Harlen W (2004) A systematic review of the evidence of reliability and validity of assessment by teachers used for summative purposes. In: Research Evidence in Education Library. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. [
Bennett et al., 1993
Peter Tymms: ‘Teachers show bias to pupils who share their personality’. The Conversation 25 Feb
Burgess, S. and Greaves, E (2009) Test Scores, Subjective Assessment and Stereotyping. Centre for Market and Public Organisation, Bristol University. Working Paper No. 09/221.
of Ethnic Minorities
Lee ??? (cited in Metzler
Ouazad, A. (2014). Assessed by a teacher like me: Race and teacher assessments. Education, 9(3),

19. respond.cc
363005
Which of these do you calculate for assessments used in your school/classroom?
Always, for every assessment
Mostly, for key assessments
At least once a year
Occasionally
Never
Item-total correlations
Correlations between assessed scores/levels and baseline (eg EYFSP, KS2, MidYIS, CAT)
Correlations between assessed scores/levels in different subjects at the same time
Correlations between assessed scores/levels and subsequent achievement

20. How well do the following assessments credit or incentivise desirable learning and teaching?
respond.cc
700207
EYFSP
KS2 Reading
GCSE science practicals
Art GCSE
Maths A level
BTEC First Diploma in Business
GCSE MFL speaking & listening
Very well; excellent alignment
Quite well; broadly aligned
Partially; some alignment but not close
Poorly; limited or no alignment
Negatively; reverse alignment

21. Why does validity matter?
Bias: Different assessment approaches can (dis)advantage different groups
Convergent/discriminant validity
Construct under-representation: assessing only part of what matters
Construct-irrelevant variance: accidentally measuring things that should not be there
Alignment between what is assessed and what is worth teaching