Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice Sally Jordan and Janet Haresnape Faculty of Science The Open University, UK Assessment in Higher Education Conference, June 2013

This is a practice exchange… So please interrupt Our plan Previous practice Drivers for change What do we mean by formative thresholded assessment? Evaluation and stumbling blocks Early findings

The UK Open University Founded in 1969; Supported distance learning; 150,000 students, mostly studying part-time; Undergraduate modules are completely open entry, so students have a wide range of previous qualifications; Normal age range from 18 to ?? 10,000 of our students have declared a disability of some sort; 25,000 of our students live outside the UK.

Historic OU Science Faculty practice Tutor-marked assignments (TMAs) and sometimes interactive computer-marked assignments (iCMAs) combine together into overall continuous assessment score (OCAS); Examination and/or end-of-module assessment (EMA) gives overall examination score (OES).

OCAS is integrated and interactive

Continuous assessment Has a useful pacing function; Students are concerned about the minutiae of the grading; Even though for most students their final grade is determined by their performance in the examination or end-of module assessment; A considerable amount of time and effort goes into producing new TMAs for each presentation of each module; And we don’t always get it right.

Drivers for change “when assessments serve both formative and summative purposes… formative work will always be threatened due to the dominance of summative requirements” (Brearley & Cullen 2012 with echoes of Black & Wiliam 1998, Gibbs 2006, Snyder 1971) Lack of alignment of tutor and student understanding of our assessment strategies. Lack of alignment of tutor and student understanding of the purpose of continuous assessment. Saving time and money.

Formative thresholded assessment – two models OCAS is formative, but students have to demonstrate engagement by scoring more than 30% in x out of y TMAs and iCMAs; final score is determined by OES alone. Students have to reach threshold (usually 40%) for OCAS (overall), but their module result is then determined on the strength of their OES alone.

Risks Students will feel insufficiently prepared for the examination or end-of-module assessment; Students will not engage sufficiently in the formative thresholded OCAS and so will lose valuable formative opportunity; Assignments will be reused, so there may be more plagiarism. For formative assessment, does this matter? DISCUSSION POINT – what do you think about our move to formative thresholded assessment?

Evaluation A series of practitioner-led mini projects; Strengths: we are practitioners and so close to the student experience; we will help each other; Dangers: we are practitioners so we have considerable pressures of other work; we won’t deliver: we will have insufficient expertise. DISCUSSION POINT: Does anyone have experience of research or evaluation carried out in this way?

Evaluation Quantitative and qualitative; Janet – looking at behaviour of students who are taking two modules, one with summative and one with formative thresholded assessment; Lynda – looking at two new modules with subtly different assessment strategies; iCMA usage can give a ‘signature’ of student engagement; TMA submission rates (before and after the change of assessment strategy); What do students think? What do tutors think?

Evaluation - complications We need to compare current with historic data; we need to compare current with historic student perceptions; Confounding variables: other changes to assessment strategy; changing student population; over-committed students.

TMA submission rates

How many TMAs did students omit

Correlations between exam score and omitted TMAs 2011J – for students who submitted all TMAs, mean exam score = 57.5%; for those who did not submit all TMAs, mean exam score = 41.3%; 2012J – for students who submitted all TMAs, mean exam score = 53.4%; for those who did not submit all TMAs, mean exam score = 44.4%; Should this worry us? (DISCUSSION POINT) Not a causal relationship.

Engagement with iCMAs

Engagement with iCMA feedback – same question, different mode of use

Similar assignment, different students

Outcomes from the evaluation Overall findings (and thus our plans for the future) – still too early to say; A beneficial side effect is that we are thinking about our assessment strategy for current and future modules in a more coherent manner.

Sally Jordan Senior Lecturer and Staff Tutor Deputy Associate Dean, Assessment Faculty of Science The Open University blog: