Opening up multiple choice: Assessing with confidence Self-assessment: strategies and software to stimulate learning HEA workshop, The Open University, 11 th June 2012 Jon Rosewell

MCQ Open question Pros –objective marking –reliable marking –easy to implement Cons –distractors trivial –distractors engender misconceptions –working backwards Pros –numeric easy to mark –tests deeper learning –can find misconceptions Cons –Free text very difficult to mark reliably

Open question

Multiple-choice question

How (not) to do MCQ… Give the name of a supervolcano: Yellowstone Redstone Bluestone Which is a plate boundary: Conservative Destructive Liberal Labour What policy did the Chinese introduce to limit population: One child policy Two child policy The child catcher Reported from BBC Bite Size

Confidence-Based Marking (CBM)Certainty-Based

CBM scoring Confidence Score CorrectWrong Low10 Medium2-2 High3-6 Tentative & correct Confidently correct Cocksure – and wrong! Gardner-Medwin & Curtin (2007) Certainty-Based Marking (CBM) for Reflective Learning and Proper Knowledge Assessment [

CBM motivations Rewards care and effort Greater engagement Encourages reflective learning Encourages self-assessment

Open CBM

Open CBM advantages Mechanically, question remains simple MCQ –answer matching is trivial –easy to implement –possible to reuse existing question banks But transformed closed to open question –student must formulate answer to decide confidence –must decide answer without +ve or –ve clues. –cannot work backward –will not be led into misconceptions.  must answer the question as set.

MCQ Open CBM Pros –objective marking –reliable marking –easy to implement Cons –distractors trivial –distractors engender misconceptions –working backwards Pros –open question –reflection –...as MCQ Cons –not always applicable –intimidating? –personality dependent?

Possible benefits Students –better engagement –improved learning –more reflective learners Staff –easier question setting

eSTEeM project plan Implement open CBM question type –in OpenMark Trial in course assessment –controlled experimental design –pilot (formative) on T216 CISCO networking Log measures –Assignment scores –Time spent on task Structured interviews to probe attitudinal aspects User-lab trials

T216 Cisco networking Questions from existing bank Offered as revision quizzes One set for each of four blocks 25 questions drawn randomly from bank of ~100 Randomly offered as either open CBM or MCQ format Issues: –Some questions bad! –OpenMark report doesn’t show confidence –Too much randomness

Uptake QuizFinishedUnfinished Demo CCNA CCNA CCNA CCNA 46467

Demo questions – scores QuestionTaken byMean score What is 2 + 2? What is derivative of x³? Who painted the 'Mona Lisa'? Who is the 'Mona Lisa'? Uncertainty principle -- whose? Uncertainty principle -- formula? Easy Difficult Tricky!

Demo questions – confidence QuestionHighMediumLow What is 2 + 2? What is derivative of x³? Who painted the 'Mona Lisa'? Who is the 'Mona Lisa'? Uncertainty principle -- whose? Uncertainty principle -- formula?

MCQ Open CBM Pros –objective marking –reliable marking –easy to implement Cons –distractors trivial –distractors engender misconceptions –working backwards Pros –open question –reflection –...as MCQ Cons –not always applicable –intimidating? –personality dependent?

Jon Rosewell