1. Contextualised teacher judgments in a high-stakes assessment system John A Pitman Queensland, Australia

2. Current notions in the international context Assessment for learning and for reporting Authentic assessment Constructed-response and extended writing as well as multiple-choice format Teacher judgment in high-stakes assessment programs Validity as important as reliability

3. Standardised tests Emphasis on discrimination (Wiliam, 2007) Items that all students score correctly and items that all students get wrong are discarded When we are interested in what students have learnt and not learnt, this (discarded) information could be important We need broader notions of reliability in the current international context

4. The psychometric approach Suits notions of validity and reliability for multiple-choice testing Does not readily suit notions of validity and reliability for school-based assessment or assessment in open-ended response modes Requires hermeneutic approach (i.e. use of expert judges)

5. Moss (1994): Reliability Moss provides a reliability framework in which teachers contextualised judgments are in the foreground Has broader notions of reliability 1. Privileging contextualised judgments 2. Generalising across tasks 3. Generalising across readers

6. Instances of broader notions Criteria and standards for marking Accountability in public education The hermeneutic approach (using expert judges) Students (and parents) become part of the dialogue by evaluating and challenging conclusions

7. Authentic assessment Reflects experience in non-school environment Has broad range of response modes Enhances skills developed in other subjects Involves students in the use of relevant and useful knowledge, thinking and practical skills

8. School-based assessment Schools develop their own sets of tests and assessment tasks in reference to a common content framework (or syllabus) Marking of student work depends on professional judgments of teachers and a system of checks and verification

9. Teacher judgment in high- stakes assessment What about validity? Checks for worth, fairness, content etc. What about reliability, the challenge in school-based assessment? Social moderation is the tool to ensure assessment quality and comparability

10. Social moderation Also called consensus moderation, auditing, and verification Performances on distinct assessments are graded using a common framework and interpreted in terms of a common standard (Linn, 1989) An example of social moderation in action: Teacher judgments are reviewed by a panel of their peers.

11. Defining features of moderation by peer review (Pitman, 1999) 1. Standards descriptors 2. Evidence 3. Consensus

12. Conclusion The reliability challenge inherent in school-based assessment can be met through the adoption of broader, hermeneutic approachesthat is, through contextualised teacher judgments.