Efficiency of assessment: initial thinking from the ITEAM project Mark Russell, Liz Gormley-Fleming and Karen Robins The ITEAM project. University of Hertfordshire

The ITEAM project Strand A project A (reasonably) tightly defined focus – EVS – QMP – On line assessment and feedback system – Dashboard

A matrix Educationally effective Resource efficiency or (1/t)

A matrix Educationally effective Resource efficiency or (1/t)

What about you? Think about one of your assessments … 1. Where would you locate it on the grid? 2. How might you nudge it to be more educationally effective and more resource efficient ? …Now, next year, the year after … Thoughts in the chat box - please

The ESCAPE assessment for learning themes … Good assessment for learning: …Engages students with assessment criteria …Supports personalised learning …Focuses on student development …Ensures feedback leads to improvement …Stimulates dialogue

The ESCAPE assessment for learning themes … Good assessment for learning: …Engages students with assessment criteria …Supports personalised learning …Focuses on student development …Ensures feedback leads to improvement …Stimulates dialogue …Considers staff and student effort

What about some sample assessments?

Essay - Summative (2000 words) Students Activity Number ActivitiesTime (minutes) Module aims and learning outcomes10 2Writing assignment brief & marking criteria 50 3Essay title.30 4Team agreement20 5External examiner comments20 6Re drafting assignment.10 7Student preparation (specific)60 8Tutorial support-drafts & feedback*20 per student Submission-setting up site, guidance20 10Marking*60 per 3 students Feedback*30 per 3 students Module report30 13Moderation*60 per 3 students External examiner30 15Assessment office*-inputting results2 per student Exam Board10 17Student tutorial post assessment process* 15 per student Total time in minutes Time in minutes per student

EVS test - 1 hour MCQs, 25 questions Students Activity Number ActivityTime (minutes) Module aims and LO’s10 2Create and type up 25MCQ questions15 per question375 3Internal Moderation20 4External moderation20 5Update questions20 6Create practice test375 7Student preparation120 8Preparing environment for assessment*15 9Run MCQ test60 10Marking - creating EVS report10 11Assessment office and inputting of grades*1 per student Providing feedback*15 13Module report30 14External examiner moderation30 15Module board20 16Post assessment process tutorial support*15 per student Student tutorial post assessment process*120 Total in minutes Time in minutes per student

OSCE-summative. Students Activity Number ActivitiesTime (minutes) Module aims and LO’s5555 2Case scenario’s120 / scenario120 3Team meeting120 4External examiner20 5Uploading assessment information30 6Student preparation120 7Environment*450 8OSCE* (role of Academic) - 6hrs*10 academics Marking & feedback*10 per student Moderation*5 per student Module report30 12External examiner20 13Assessment office*5 per student Module board10 15Post assessment process feedback*15 per student Total time in minutes Time in minutes per student

And you …? What sort of things are you doing / (need to do) to help your staff make informed decisions about choosing an assessment type for effectiveness and efficiency ? Thoughts in the chat box - please

Developing a tool

Plotting the data Numbers of students Time (minutes)

Diverging time scales Numbers of students Time (minutes) Assessment type A Assessment type B

Overlapping time scales

Not prescriptive!

And so … In what ways might these tools (and approaches) help or hinder your work? Thoughts in the chat box - please

Does (say) 20 hours of our effort yield the same learning payoff?

An example with EVS

The ESACPE assessment for learning themes … Good assessment for learning: …Engages students with assessment criteria …Supports personalised learning …Focuses on student development …Ensures feedback leads to improvement …Stimulates dialogue …Considers staff and student effort

So let’s always use the least time consuming?

Alas, it’s a little (a lot) more complex and messy than that.

The purposes of assessment are important too! 31 Summative High Stakes Measure learning Formative No / Low Stakes Motivate learning Educative Auditing

The purposes of assessment are important too! 32 Summative High Stakes Measure learning Formative No / Low Stakes Motivate learning Feedback rich? Linked (in-process) tasks? Regular tasks? Error does not lead to punishment Need for feedback is questionable (MBR) One-off tasks End of process tasks? Educative Auditing

Assessment patterns Beyond the single assessment task Beyond the single assessment task

Some alternatives…

And many more …

Programme view of assessment

Far reaching implications Our work (as teachers) extends beyond our classroom. How might we measure this? Performance on singular assessment? Success in Module? Success in Year? Success in Programme? Institutional reputation?

Conclusion – to date Numbers ‘might’ be countable Numbers ‘might’ be meaningful (process improvement, stimulating dialogue, moving in the ‘right direction’)

A possible conclusion – to date Numbers ‘might’ be countable Numbers ‘might’ be meaningful (process improvement, stimulating dialogue, moving in the ‘right direction’) But…

A possible conclusion – to date Numbers ‘might’ be countable Numbers ‘might’ be meaningful (process improvement, stimulating dialogue, moving in the ‘right direction’) But…

Some questions for you to ponder… What is the difference in learning payoff between a 4000 word and a 2000 word essay ? Is a single 4000 word essay more beneficial than (say) two 1500 word essays? What are the resource implications of the above variants? What is the difference in learning payoff between a 4000 word and a 2000 word essay ? Is a single 4000 word essay more beneficial than (say) two 1500 word essays? What are the resource implications of the above variants? What are the various processes of assessment doing? Where might you best place your efforts to reduce the resource burden while keeping the effectiveness of the assessment? What are the various processes of assessment doing? Where might you best place your efforts to reduce the resource burden while keeping the effectiveness of the assessment? In what ways are you facilitating meaningful discussions about both resource efficiency and educational effectiveness?