1. Opening up multiple choice: Assessing with confidence Self-assessment: strategies and software to stimulate learning HEA workshop, The Open University, 11 th June 2012 Jon Rosewell J.P.Rosewell@open.ac.uk J.P.Rosewell@open.ac.uk

2. MCQ Open question Pros –objective marking –reliable marking –easy to implement Cons –distractors trivial –distractors engender misconceptions –working backwards Pros –numeric easy to mark –tests deeper learning –can find misconceptions Cons –Free text very difficult to mark reliably

3. Open question

4. Multiple-choice question

5. How (not) to do MCQ… Give the name of a supervolcano: Yellowstone Redstone Bluestone Which is a plate boundary: Conservative Destructive Liberal Labour What policy did the Chinese introduce to limit population: One child policy Two child policy The child catcher Reported from BBC Bite Size

6. Confidence-Based Marking (CBM)Certainty-Based

7. CBM scoring Confidence Score CorrectWrong Low10 Medium2-2 High3-6 Tentative & correct Confidently correct Cocksure – and wrong! Gardner-Medwin & Curtin (2007) Certainty-Based Marking (CBM) for Reflective Learning and Proper Knowledge Assessment [http://www.ucl.ac.uk/lapt/REAP_cbm.pdf]

8. CBM motivations Rewards care and effort Greater engagement Encourages reflective learning Encourages self-assessment

9. Open CBM

10. Open CBM advantages Mechanically, question remains simple MCQ –answer matching is trivial –easy to implement –possible to reuse existing question banks But transformed closed to open question –student must formulate answer to decide confidence –must decide answer without +ve or –ve clues. –cannot work backward –will not be led into misconceptions.  must answer the question as set.

11. MCQ Open CBM Pros –objective marking –reliable marking –easy to implement Cons –distractors trivial –distractors engender misconceptions –working backwards Pros –open question –reflection –...as MCQ Cons –not always applicable –intimidating? –personality dependent?

12. Possible benefits Students –better engagement –improved learning –more reflective learners Staff –easier question setting

13. eSTEeM project plan Implement open CBM question type –in OpenMark Trial in course assessment –controlled experimental design –pilot (formative) on T216 CISCO networking Log measures –Assignment scores –Time spent on task Structured interviews to probe attitudinal aspects User-lab trials

14. T216 Cisco networking Questions from existing bank Offered as revision quizzes One set for each of four blocks 25 questions drawn randomly from bank of ~100 Randomly offered as either open CBM or MCQ format Issues: –Some questions bad! –OpenMark report doesn’t show confidence –Too much randomness

15. Uptake QuizFinishedUnfinished Demo162178 CCNA 1213262 CCNA 286130 CCNA 36951 CCNA 46467

16. Demo questions – scores QuestionTaken byMean score What is 2 + 2?238 2.51 What is derivative of x³?223-0.47 Who painted the 'Mona Lisa'?212 2.14 Who is the 'Mona Lisa'?208 0.21 Uncertainty principle -- whose?207 0.03 Uncertainty principle -- formula?218 0.01 Easy Difficult Tricky!

17. Demo questions – confidence QuestionHighMediumLow What is 2 + 2?210523 What is derivative of x³?964681 Who painted the 'Mona Lisa'?1652126 Who is the 'Mona Lisa'?5641111 Uncertainty principle -- whose?6231114 Uncertainty principle -- formula?3921158

18. MCQ Open CBM Pros –objective marking –reliable marking –easy to implement Cons –distractors trivial –distractors engender misconceptions –working backwards Pros –open question –reflection –...as MCQ Cons –not always applicable –intimidating? –personality dependent?

19. Jon Rosewell J.P.Rosewell@open.ac.uk J.P.Rosewell@open.ac.uk www.open.ac.uk