Deakin University CRICOS Provider Code: 00113B deakin.edu.au/learning/enhancing-your-practice/cradle The ‘unknown unknowns’ of assessment rubrics in practice, policy and research Associate Professor Phillip Dawson Associate Director Centre for Research in Assessment & Digital Learning (CRADLE) Deakin University, Melbourne,

Deakin University CRICOS Provider Code: 00113B

A couple of decades ago, rubric began to take on a new meaning among educators. Measurement specialists who scored students’ written compositions began to use the term to describe the rules that guided their scoring. They could have easily employed a more readily comprehensible descriptor, such as scoring guide, but scoring guide lacked adequate opacity. Rubric was a decisively more opaque, hence technically attractive, descriptor. (Popham 1997, p. 72)

Deakin University CRICOS Provider Code: 00113B Hands up if rubrics make you feel: Confident Concerned Relieved Confused Excited Bored …another feeling you’d like to share?

Deakin University CRICOS Provider Code: 00113B Cumulative frequency, 28 April, 2015.

Deakin University CRICOS Provider Code: 00113B What is a rubric? A tool used in the process of evaluating student work, which usually includes: –evaluative criteria –quality definitions for those criteria at particular levels –a scoring strategy. Per Popham (1997)

Deakin University CRICOS Provider Code: 00113B This workshop Overview of framework Practice sharing Scenarios

Deakin University CRICOS Provider Code: 00113B Framework overview 14 ‘design elements’ representing choices made in rubric design Brief overview of each – stop me if you want more detail

Deakin University CRICOS Provider Code: 00113B Specificity: the particular object of assessment Generic –Including faculty-wide or course level Task-type Task-specific

Deakin University CRICOS Provider Code: 00113B Secrecy: who the rubric is shared with, and when it is shared Top-secret rubrics Rubrics returned with marks/feedback Rubrics shared with task instructions

Deakin University CRICOS Provider Code: 00113B Exemplars: work samples provided to illustrate quality What does a good one of these look like? Why is this one good? What does a good one of these look like? Why is this one good?

Deakin University CRICOS Provider Code: 00113B Scoring strategy: procedures used to arrive at marks and grades Analytic Adding up Logic, conditions, thresholds Combining scores from multiple markers Holistic Assessor uses all information to make a single overall quality judgement

Deakin University CRICOS Provider Code: 00113B Evaluative criteria: overall attributes required of the student (usually the leftmost column) Mandatory (or not?) Connector between indicators of quality and task/subject outcomes

Deakin University CRICOS Provider Code: 00113B Quality levels: the number and type of levels of quality Grade levels (P/C/D/HD) Learning outcomes taxonomies Levels of study Fail? Every criterion at every level?

Deakin University CRICOS Provider Code: 00113B Quality definitions: explanations of attributes of different levels of quality “Mixture of qualitative and quantitative feedback sources”

Deakin University CRICOS Provider Code: 00113B Judgement complexity: the evaluative expertise required of users of the rubric

Deakin University CRICOS Provider Code: 00113B Judgement complexity: the evaluative expertise required of users of the rubric

Deakin University CRICOS Provider Code: 00113B Users and uses: who makes use of the rubric, and to what end Teachers Marking Providing feedback Communicating standards Understanding standards Students Planning work Understanding standards Self assessment Peer assessment Grade grubbing Planning improvement to future work

Deakin University CRICOS Provider Code: 00113B Creators: the designers of the rubric Teachers Textbook publishers Internet communities Students Rubric specialists Researchers Professional bodies

Deakin University CRICOS Provider Code: 00113B Quality processes: approaches to ensure the reliability and validity of the rubric Formal reliability and validity processes Informal review, piloting and revision Haven’t got time for that, just see how it goes

Deakin University CRICOS Provider Code: 00113B Accompanying feedback information: comments, annotation or other notes on student performance Stimulus for feedback discussion Focal point for written feedback A supplement to written feedback Replacement for written feedback

Deakin University CRICOS Provider Code: 00113B Presentation: how the information in the rubric is displayed

Deakin University CRICOS Provider Code: 00113B Explanation: instructions or other additional information provided to users Written instructions Discussion around rubric in class Marking workshop Video explanation

Deakin University CRICOS Provider Code: 00113B How do you use rubrics? Do you feel strongly about any of these elements? Discuss how you use rubrics in groups of three or four Structure your conversation around the framework

Deakin University CRICOS Provider Code: 00113B Scenarios – choose one, in groups Rubric advice 1 st year educational theory course 1000 students Lecturer wants your advice on rubric for 2000 word essay Faculty rubric policy/procedures Education faculty at a large university have contracted you to write rubric policy/procedures What would you mandate?

Deakin University CRICOS Provider Code: