1. The Continual Assessment of Confidence or Knowledge with Hidden MCQ? Short Paper W.I.P. Phil Davies School of Computing University of Glamorgan

2. Conference 1999 Learning Through Assessment Phil Davies School of Computing University of Glamorgan

3. Implementation of OLAL  Tests Weeks 3,6,8,12 (60% Subject Mark)  Percentage Marks 5%,15%,15%,25%  “Beep Test”  +2 Correct -1 Incorrect(Really -3)

5. Student Concentration  Guessing –Why should the student care if incorrect?  Allow TWO PASSES of the test  By concentrating on first pass –will get reward on second pass –formative

6. Results of The Tests Improvement of Test Results Pass One to Two

7. Who benefited from the test?

8. Student Feedback  Excellent way of gaining knowledge  2nd Test Allowed us to learn  Good assistant to learning  Encourages Revision  Very good, promotes learning of a subject early on  Better than sex!!!!

9. “There’s no confidence in multiple-choice testing …” CAA Conference 2002 Phil Davies

10. What decides the QUALITY of a question?  In the 2001 World cup qualifier between England and Greece, who scored the all important equalizing goal?  David Beckham  Victoria Beckham  Brooklyn Beckham  Rebecca Loos

11. Rewards?  Know it  On the tip of my tongue  If I see the answers I’ll know it  Deduction  Guess  Know it … oh no I’ve got it wrong.. misinformation

13. Proportion of Questions Correct  High Confidence => 84% correct  Fairly Confident => 58% correct  Not confident => 47% correct  72% of all questions answered with high confidence

14. Student thoughts on Confidence  “Eliminates guesswork”  “I was waiting for a leggy blond to bring on my prize at the end”  “separated the lucky students from me”  “at least I didn’t lose marks for selecting no confidence”  “are you testing my confidence or knowledge”

15. Quality of a multiple choice question?  In the 2001 World cup qualifier between England and Greece, who scored the all important equalizing goal?  Teddy Sheringham  Michael Owen  Robbie Fowler  QUALITY OF QUESTION NOT DISTRACTERS

16. “Bite The Bullet”  Use Confidence Testing for all four tests (10%, 15%, 20% & 25%)  Weighting of Passes (50/50, 60/40, 70/30 & 80/20)  How to judge if the students had ‘really’ learned?  Identify improvement >  Select five questions that were answered ‘poorest’ in test one & pass through to test two  Select five questions that were answered ‘poorest’ in test two & pass through to test three ( + original five) …. Etc.

17. How to identify ‘Poorest’  Using confidence, degree of correctness of answering  Using confidence scores to provide a degree of ‘poorness’ i.e. +4, +2, +1, -6, -5, -4  High degree of ‘poorness’ could just be a ‘BAD’ question  Swing between 1 st and 2 nd pass of degree of correctness should be ‘Reasonable’

18. 80> 75-7970-7465-6960-6455-5950-5445-4940-4435-3930-3425-29 1256111315127763 Test #Average Pass OneAverage Pass TwoRatio First/SecondAverage Total Test One 10% 42.56%72.74%50 / 5057.65% Test Two 15% 26.80%66.62%60 / 4042.73% Test Three 20% 41.14%76.40%70 / 3051.72% Test Four 25% 51.00%81.50%80 / 2057.10%

19. Test 2 - 1Test 3 - 2Test 4 - 3Test 4 - 1 80>-242325.00 75-7910.5945.565.00 70-74-31.8439.620.80 65-69-34.8343.8329.6738.67 60-64-45.2753.2716.5524.55 55-59-43.1536.0820.2313.15 50-54-36.248.821.5334.13 45-49-7.027.9238.8359.75 40-44-45.4330.0028.4313.00 35-39-32.8627.0025.7119.86 30-35-47.3344.531.3328.50 25-29-47.6737.00-13.00-23.67 Average-30.25+33.70+23.12+26.56

20. Identifying Poor Questions Test One Question Number # Correct Pass 1 # Wrong Pass 1 Weighting Factor # Correct Pass 2 # Wrong Pass 2 Weighting Factor Weighting Difference Pass 2-1 Q52761-2515632+9260 Q143058-2186028+67285 Q153454-1815533+4185 Q19880-3567612+223579 Q203355-1875830+16203 Q271177-3735830+21394

21. Identifying the Poorest Questions Test two to three Question Number # Correct Pass 1 # Wrong Pass 1 Weighting Factor # Correct Pass 2 # Wrong Pass 2 Weighting Factor Weighting Difference Pass 2-1 Q21474-348799250598 Q62860-2224543-109331 Q82266-2557018145400 Q102860-2154840-66281 Q112068-281573120301 Q182860-200799227427 Q193157-2027414166368 Q212761-235573132267 Q273256-2106622106316 Q282761-2567711238494 Q322365-254826265519

22. Identifying Poorest Questions from Three to Four Question Number # Correct Pass 1 # Wrong Pass 1 Weighting Factor # Correct Pass 2 # Wrong Pass 2 Weighting Factor Weighting Difference Pass 2-1 Q63256-212632573285 Q112761-2197513217436 Q161969-313602858371 Q263058-225817253480 Q403058-2187216183401

23. Passed through to Test Four  13 of the poorest question  6 new questions based upon additional work between tests 3-4  21 questions (cross section of questions and topics from previous tests)

24. Improvement of Results Through Tests

25. Improvement Pass One of Questions

26. Improvement for Question 5  Stronger Students improved quickly through the tests  Became more confident in answering the questions  Weaker Students improved less quickly than stronger through the test progress

27. Is it just Confidence or Knowledge i.e. questions right or wrong? CategoryAverage # of Questions Correct Pass 1 Average # of Questions Correct Pass 2 Average Total # of Questions Correct 80> (1)131.0158.0289.0 75-79 (2)123.5150.5274.0 70-74 (3)117.6148.6266.2 65-69 (4)113.8150.7264.5 60-64 (5)110.3145.0255.3 55-59 (6)107.6143.4251.0 50-54 (7)100.4136.2236.6 45-49 (8)92.5132.6225.1 40-44 (9)94.0126.4220.4 35-39 (10)87.7121.6209.3 30-34 (11)80.5117.3197.8 25-29 (12)76.7117.3194.0

28. Some Concerns CategoryAverage # of Questions Correct Pass 1 Average # of Questions Correct Pass 2 Average Total # of Questions Correct 1281127208 (194) 3102136238 (266) 778117195 (236) 8112141253 (225) 1097137234 (209)

29. Overall Comments  Work in Progress  Results look very positive  Correctness Factor is dependant upon weighting of scores  Guessing not so much of a problem  Changed students mode of revising / study  Fine granularity in the QUALITY OF A QUESTION Possibility of Computerized Adaptive Testing However note the quality of the question change throughout the course of the cumulative tests  Does gender come into it??

30. Contact Information  Email: pdavies@glam.ac.uk pdavies@glam.ac.uk  Phone: 01443 482247 School of Computing University of Glamorgan Pontypridd Mid – Glamorgan South Wales