1. Dr. Yuyuan Zhao Department of Engineering University of Liverpool A Scheme for Converting MCQ Test Scores to Percentage Marks

2. Outline Multiple Choice Questions (MCQ): Benefits and problems Conversion algorithm Conversion table Spreadsheet implementation Merits and applicability Points for attention Summary

3. Why MCQ? Easy to set up (?) Easy to Mark Quick to obtain results Suitable for selection processes Suitable for large classes Suitable for assessing knowledge, aptitude, competence and many skills Especially suitable for fact based subjects

4. Why is MCQ not widely used? Involves guesswork Not always suitable for assessment of certain skills, e.g.: describing phenomena solving problems writing essays analysing data performing calculations Not suitable for big problems Lack of databank of questions Raw scores not compatible with conventional marking scheme

5. A Good MCQ Test Assesses most learning outcomes Has 4 choices of answer to each question Has equally feasible choices to a laymen for each question Has 20 or more questions the probability of obtaining a mark >40 by pure guesswork in a test consisting of 18 four-choice questions is <1%. Raw scores are converted to marks that are a true measure of the students performance.

6. The Conversion Scheme The scheme is developed based on probabilistic analysis For each multiple-choice question, A full score is awarded if the choice of answer is correct A full mark is only awarded if the choice is NOT made by guesswork A partial mark is awarded if a student knows any of the wrong answers (firm answers) No mark is awarded for pure guesswork The scheme removes the guesswork element (YY Zhao, Int. J. Eng. Edu., 21(2005), 1189-1194)

7. Types of 4-Choice Questions

8. Score vs Mark Each score S corresponds to a mark M. f – fraction of firm answers N – number of choices

9. Conversion Table For a full version of the conversion table see: Y Zhao, Int. J. Eng. Edu., 21(2005), 1189-1194

10. Implementation Using EXCEL Enter conversion table in columns A and B Enter student names and their raw scores in columns C and D Enter a formula using VLOOKUP function in column E

11. Merits of the Scheme Compatible with standard percentage marking scheme Independent of class size and historical data MCQ can be used in conjunction with other assessment methods Questions with different weightings acceptable Guesswork allowed but not rewarded

12. Reliability of Scheme MCQ Module Mark Semester Average 20% within 2 marks 50% within 5 marks 70% within 10 marks The marks of a module assessed entirely by MCQ tests correlated very well with the average marks of all the modules in that semester for a class of ~170 students

13. Recommendations For summative assessment each question should have 4 or 5 choices. For formative assessment, 2 or 3- choice questions are acceptable. The number of questions in a MCQ test should be at least 10. Scores of individual assessment units should be converted and then averaged to give the overall mark.

14. Summary A scheme has been developed for converting raw MCQ scores to percentage marks by removing the guesswork element. The converted marks are a true measure of the students performance. The converted marks are compatible with the conventional marking scheme. The conversion scheme is independent of class size and historical data. The conversion scheme is easy to implement in a spreadsheet programme such as MS Excel. The conversion scheme has been proved to be reliable.