“There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know.” Donald Rumsfeld, February 2002

The use of regular multiple choice quizzes to enhance the development of students self-assessment and self- feedback skills Pecha Kucha Presentation Alan Brown CHEP Annual Conference January 2013

4 types of student? A thirteenth-century Persian poet, Ibn Yamin said there are four types of men:Persian 1.One who knows and knows that he knows... His horse of wisdom will reach the skies. 2.One who knows, but doesn't know that he knows... He is fast asleep, so you should wake him up! Source: Wikipedia, 2013

4 types of student? 3.One who doesn't know, but knows that he doesn't know... His limping mule will eventually get him home. 4.One who doesn't know and doesn't know that he doesn't know... He will be eternally lost in his hopeless oblivion! Our first job is to get students away from 4 Source: Wikipedia, 2013

Fluid Mechanics in Year 1 Brand new subject Not on ‘A’ level physics or maths syllabus Several new concepts – Pressure – Viscosity – Density

Fluid Mechanics in Year 1 Several ‘Old’ concepts – Forces – Velocity and Acceleration – Momentum But these don’t look the same as in physics Assessments historically based on applied mathematical problems

The problem Enable students to find out what they know and do not know This is the role of Feedback Several new concepts requires lots of rapid feedback I taught this module to 88 students

The solution Multiple choice quiz questions each week on Blackboard Based on the previous week’s material The adaptive release system is used to enforce the quiz questions

Any attempt at the questions allows access to the next set of resources Answers and feedback are given instantly The questions are designed to test conceptual knowledge, not learned definitions Class test uses similar types of multiple choice questions

Result 1: Class test marks Scores in the class test were similar to the previous year (applied maths problems) Quiz designed to test the understanding of the new and old concepts – misunderstandings much easier to spot and correct than using applied mathematics type questions

Result 1: Class test marks Feedback to correct conceptual misunderstandings given 1 week after the test

Result 2: Predicting performance Are students more aware of their own knowledge? Students were asked to predict their grade: – Before they sat the test – After they sat the test – After they had seen all the answers

Result 2: Predicting performance With MCQs the students were much better at predicting their marks With MCQsWithout MCQs

Result 3: Engagement Out of 88 students – 4 attempted none – 7 attempted 1 or 2 – 15 attempted 3 or 4 – 62 attempted 5 or more Test attempts Mean overall mark

Result 3: Engagement Student feedback on the MCQ system : – 87 % of respondents found the MCQs helpful or very helpful, – 5 % said they were unhelpful Many students answered multiple times

What have I learned New concepts are difficult to learn Students need to start from a position of self- knowledge Applied mathematics is not an effective way of assessing conceptual knowledge Most students will engage with feedback opportunities

What I will do next Check MCQ answers to try and determine misunderstandings earlier – Don’t wait until the class test Refine teaching based on misunderstandings from last year Enhance the automated feedback in the MCQs Use MCQs to monitor engagement

Conclusions Rapid feedback should enhance students self- knowledge MCQs used to give rapid automated feedback to 88 students on a weekly basis Adaptive release gives an incentive without marking and assessment

Conclusions Students are better able to predict their marks Students were overwhelmingly positive about the quiz questions Engagement easy to assess Hopefully fewer “Unknown Unknowns”