The Continual Assessment of Confidence or Knowledge with Hidden MCQ? Short Paper W.I.P. Phil Davies School of Computing University of Glamorgan

Conference 1999 Learning Through Assessment Phil Davies School of Computing University of Glamorgan

Implementation of OLAL  Tests Weeks 3,6,8,12 (60% Subject Mark)  Percentage Marks 5%,15%,15%,25%  “Beep Test”  +2 Correct -1 Incorrect(Really -3)

Student Concentration  Guessing –Why should the student care if incorrect?  Allow TWO PASSES of the test  By concentrating on first pass –will get reward on second pass –formative

Results of The Tests Improvement of Test Results Pass One to Two

Who benefited from the test?

Student Feedback  Excellent way of gaining knowledge  2nd Test Allowed us to learn  Good assistant to learning  Encourages Revision  Very good, promotes learning of a subject early on  Better than sex!!!!

“There’s no confidence in multiple-choice testing …” CAA Conference 2002 Phil Davies

What decides the QUALITY of a question?  In the 2001 World cup qualifier between England and Greece, who scored the all important equalizing goal?  David Beckham  Victoria Beckham  Brooklyn Beckham  Rebecca Loos

Rewards?  Know it  On the tip of my tongue  If I see the answers I’ll know it  Deduction  Guess  Know it … oh no I’ve got it wrong.. misinformation

Proportion of Questions Correct  High Confidence => 84% correct  Fairly Confident => 58% correct  Not confident => 47% correct  72% of all questions answered with high confidence

Student thoughts on Confidence  “Eliminates guesswork”  “I was waiting for a leggy blond to bring on my prize at the end”  “separated the lucky students from me”  “at least I didn’t lose marks for selecting no confidence”  “are you testing my confidence or knowledge”

Quality of a multiple choice question?  In the 2001 World cup qualifier between England and Greece, who scored the all important equalizing goal?  Teddy Sheringham  Michael Owen  Robbie Fowler  QUALITY OF QUESTION NOT DISTRACTERS

“Bite The Bullet”  Use Confidence Testing for all four tests (10%, 15%, 20% & 25%)  Weighting of Passes (50/50, 60/40, 70/30 & 80/20)  How to judge if the students had ‘really’ learned?  Identify improvement >  Select five questions that were answered ‘poorest’ in test one & pass through to test two  Select five questions that were answered ‘poorest’ in test two & pass through to test three ( + original five) …. Etc.

How to identify ‘Poorest’  Using confidence, degree of correctness of answering  Using confidence scores to provide a degree of ‘poorness’ i.e. +4, +2, +1, -6, -5, -4  High degree of ‘poorness’ could just be a ‘BAD’ question  Swing between 1 st and 2 nd pass of degree of correctness should be ‘Reasonable’

80> Test #Average Pass OneAverage Pass TwoRatio First/SecondAverage Total Test One 10% 42.56%72.74%50 / % Test Two 15% 26.80%66.62%60 / % Test Three 20% 41.14%76.40%70 / % Test Four 25% 51.00%81.50%80 / %

Test 2 - 1Test 3 - 2Test 4 - 3Test > Average

Identifying Poor Questions Test One Question Number # Correct Pass 1 # Wrong Pass 1 Weighting Factor # Correct Pass 2 # Wrong Pass 2 Weighting Factor Weighting Difference Pass 2-1 Q Q Q Q Q Q

Identifying the Poorest Questions Test two to three Question Number # Correct Pass 1 # Wrong Pass 1 Weighting Factor # Correct Pass 2 # Wrong Pass 2 Weighting Factor Weighting Difference Pass 2-1 Q Q Q Q Q Q Q Q Q Q Q

Identifying Poorest Questions from Three to Four Question Number # Correct Pass 1 # Wrong Pass 1 Weighting Factor # Correct Pass 2 # Wrong Pass 2 Weighting Factor Weighting Difference Pass 2-1 Q Q Q Q Q

Passed through to Test Four  13 of the poorest question  6 new questions based upon additional work between tests 3-4  21 questions (cross section of questions and topics from previous tests)

Improvement of Results Through Tests

Improvement Pass One of Questions

Improvement for Question 5  Stronger Students improved quickly through the tests  Became more confident in answering the questions  Weaker Students improved less quickly than stronger through the test progress

Is it just Confidence or Knowledge i.e. questions right or wrong? CategoryAverage # of Questions Correct Pass 1 Average # of Questions Correct Pass 2 Average Total # of Questions Correct 80> (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12)

Some Concerns CategoryAverage # of Questions Correct Pass 1 Average # of Questions Correct Pass 2 Average Total # of Questions Correct (194) (266) (236) (225) (209)

Overall Comments  Work in Progress  Results look very positive  Correctness Factor is dependant upon weighting of scores  Guessing not so much of a problem  Changed students mode of revising / study  Fine granularity in the QUALITY OF A QUESTION Possibility of Computerized Adaptive Testing However note the quality of the question change throughout the course of the cumulative tests  Does gender come into it??

Contact Information   Phone: School of Computing University of Glamorgan Pontypridd Mid – Glamorgan South Wales