1. 1 Bacon – T. A. Webinar – 7 March 2012 Transforming Assessment with Adaptive Questions Dick Bacon Department of Physics University of Surrey r.bacon@surrey.ac.uk Using assessments to encourage learning

2. 2 Bacon – T. A. Webinar – 7 March 2012 Transforming Assessment with Adaptive Questions Effective feedback Adaptive questions Question design ‘Help’ design Results and discussion Conclusions Student opinions Context

3. 3 Bacon – T. A. Webinar – 7 March 2012 Context Four level 1 electronics tests containing 27 questions, that are set as coursework The questions have been used in SToMP assessment systems (this is the fourth) since 1995. The latest system SToMP-II is QTI v2.1 compliant, and this paper discusses the deployment of 15 of these questions that have been modified to improve the effectiveness of their feedback. Transforming Assessment with Adaptive Questions www.stomp.ac.uk

4. 4 Bacon – T. A. Webinar – 7 March 2012 To be effective, feedback must be This can all be achieved within one QTI v2.1 adaptive question ► targeted ( use mal rules ) ► relevant to student’s probable understanding ( use history ) ► attended to ( give it immediately but randomise the data ) ► applied ( offer another try ) freely adapted from Gibbs and Simpson (2004) Transforming Assessment with Adaptive Questions

5. 5 Bacon – T. A. Webinar – 7 March 2012 v2 includes support for adaptive questions. A specification for the transfer of assessment questions between different testing systems. The most widely implemented version is v1.2.1, which was published in March 2003. v2.0 was published in May 2005. The structure is similar, but the detail is completely different. Support for numerical questions is much improved in v2, including randomisation and the testing of both accuracy and precision. QTI : Question Test Interoperability

6. 6 Bacon – T. A. Webinar – 7 March 2012 Adaptive questions allow a dialogue to be supported ► question text can be changed ► interactions can be changed ► feedback can be changed When the submit button is pressed: ► further tries can be offered !! There is no limit to the complexity !! Transforming Assessment with Adaptive Questions

7. 7 Bacon – T. A. Webinar – 7 March 2012 Show question Show feedback for right, approximate or precision error. Adjust mark. Set score = mark. Show feedback addressing mal -rule mistake. Reduce mark. right? mal- rule? Show feedback for unrecognised error. Reduce mark. mark > 3 yes no Show final ‘wrong’ feedback. Set score = 0 no yes Combine these for a multiple numeric response question.. Single numeric response question.. blanking the correct responses. Set mark = 10 Transforming Assessment with Adaptive Questions

8. 8 Bacon – T. A. Webinar – 7 March 2012 Help was somehow to be provided for the student who did not know how to start the problem. For clarity, this was offered as a separate entity. Giving help We also wanted to give the student some control over the amount of detail that was to be given, i.e. the level of help that was required. Transforming Assessment with Adaptive Questions

9. 9 Bacon – T. A. Webinar – 7 March 2012 Transforming Assessment with Adaptive Questions Q 3.2

10. 10 Bacon – T. A. Webinar – 7 March 2012 Describe how to tackle the problem [-2 marks].. Show how a solution can be obtained [-4 marks].. Split the problem into smaller parts [-6 marks].. discusses the principles involved in obtaining a solution describes in more detail a process by which a solution can be obtained breaks the problem down, getting the student to solve the components until the solution is obtained. Then it usually re-poses the original problem. Transforming Assessment with Adaptive Questions

11. 11 Bacon – T. A. Webinar – 7 March 2012 no yes Explain n th part and pose question. Right ? 1 st time ? Explain in simpler terms and usually give a numeric expression for the answer Address the student’s inability to answer the question, but leave remediation to the student. Then continue to next part. Dealing with feedback for the problem-parts help. no Transforming Assessment with Adaptive Questions

12. 12 Bacon – T. A. Webinar – 7 March 2012 In a five option MCQ, the probability of getting it right in three tries and thus gaining some marks without knowing the subject at all, is 0.7. Thus, except for cases where there are many more options, a “don’t know” option can be the more attractive way of encouraging a student to engage. Feedback can be made more detailed, or can offer a different viewpoint, each time ‘don’t know’ is selected. As the amount of help increases, the final mark is reduced. Other question types Transforming Assessment with Adaptive Questions

13. 13 Bacon – T. A. Webinar – 7 March 2012 Other question types Two questions used the “don’t know” model (one single choice, one multiple selection) Between them, 59 right 7 part right 2 part wrong 2 wrong 2 “don’t know”s, then wrong Transforming Assessment with Adaptive Questions

14. 14 Bacon – T. A. Webinar – 7 March 2012 Q1.2

15. 15 Bacon – T. A. Webinar – 7 March 2012 Other question types Two questions used the “don’t know” model (one single choice, one multiple selection) Between them, 59 right 7 part right 2 part wrong 2 wrong 2 “don’t know”s, then wrong One randomised single choice (from 6) offered multiple tries, 31 right 5 right second try 2 right third try 3 wrong third try Transforming Assessment with Adaptive Questions

16. 16 Bacon – T. A. Webinar – 7 March 2012 Q4.4

17. 17 Bacon – T. A. Webinar – 7 March 2012 32right 4malright 4wrongright 3wrong right 11wrong 4 2mal1mal1/2wrong 2 malwrong 1mal Q 4.4.a/b Voltage gain in LR filter Transforming Assessment with Adaptive Questions

18. 18 Bacon – T. A. Webinar – 7 March 2012 Transforming Assessment with Adaptive Questions Q 3.2

19. 19 Bacon – T. A. Webinar – 7 March 2012 16right 2mal-ruleright 2mal-rule right 1wrongright 2wrong right 1mal-rule wrong 1 1 help 1right 1wronghelp 1right 1help 1mal-rulewrongright 1wrong help 2wrong 1help 2wrong 1help 3right 1help 3mal-ruleright Charges on two capacitors in series Multiple answers, with mal-rules and feedback Transforming Assessment with Adaptive Questions

20. 20 Bacon – T. A. Webinar – 7 March 2012 This chart refers only to those who would have scored zero. Transforming Assessment with Adaptive Questions

21. 21 Bacon – T. A. Webinar – 7 March 2012 This chart refers only to those who would have scored zero. Transforming Assessment with Adaptive Questions

22. 22 Bacon – T. A. Webinar – 7 March 2012 Transforming Assessment with Adaptive Questions 727 first attempts 421 correct 306 wrong 306 given help, hints and further attempts 198 correct in the end 108 wrong in the end Student × question instances Without feedback the probability of getting it right was = 0.579 With the feedback and multiple tries the probability became = 0.851 These are distinct at the 99.999% confidence level.

23. 23 Bacon – T. A. Webinar – 7 March 2012 Transforming Assessment with Adaptive Questions There were 23 instances of help being requested, 156 mal-rule instances and 316 wrong instances. 15631623

24. 24 Bacon – T. A. Webinar – 7 March 2012 Student comments: “Really liked the opportunity to have another go at questions with hints if you were unsuccessful in the first attempt.” Best feature ? “Giving hints if the answer entered is wrong.” “The hints given as to how to do the question if you got it wrong in the first instance, and the chance to have another go, albeit for less marks.” “the hints - they often nudge my attention to a key fact” “the overall presentation was clear and easy to understand what was required of us” “the ability to see your mark immediately afterwards “ Transforming Assessment with Adaptive Questions

25. 25 Bacon – T. A. Webinar – 7 March 2012 Student comments: Worst feature ? “Difficult to know how to enter data with powers.” “would like a marks summary” “The inability to review closed questions.” “sometimes the hints when you got it wrong were totally irrelevant” “Not very reliable - we experienced a few crashes and disruptments with the system over the year” Transforming Assessment with Adaptive Questions

26. 26 Bacon – T. A. Webinar – 7 March 2012 “sometimes the hints when you got it wrong were totally irrelevant” With multiple randomised variables, two different mal rules may well have very similar values, and the condition used for mal rule testing only checked within 5% (to cater for precision and rounding effects). I have addressed this by checking for all mal rules when marking, and presenting all appropriate feedback for the mal rules that have matching values. Transforming Assessment with Adaptive Questions

27. 27 Bacon – T. A. Webinar – 7 March 2012 “sometimes the hints when you got it wrong were totally irrelevant” Otherwise, the feedback may indeed be badly written – so as to be unrecognisable by students with particular misconceptions, for example. This will need a detailed analysis of what a student needs to know to be able to solve the problem (concept mapping). It might be addressed by starting a dialogue (within the feedback) to establish what it is that each student needs to be told. Transforming Assessment with Adaptive Questions

28. 28 Bacon – T. A. Webinar – 7 March 2012 Conclusions: ► looking for more mal-rules ► improving some of the mal-rule feedback text ► making ‘wrong’ feedback progressive, or interactive 2. That out of a total of 758 student responses, 205 were corrected by a combination of targeted and general feedback and online help, leaving only 111 as incorrect. 1. The questions need further development, specifically.. ► improving some of the help text ► providing more mark information for users ? Transforming Assessment with Adaptive Questions

29. 29 Bacon – T. A. Webinar – 7 March 2012 The end. If you are interested in getting involved in creating or using these sorts of questions, then do contact me: r.bacon@surrey.ac.uk If you are interested in seeing what is available in QTI v2.1 systems, Then have a look at the JISC project that has been collecting resources and helping finalise the current specification, http://cloud.niallbarr.me.uk/http://cloud.niallbarr.me.uk/. To find out more about the SToMP system, look at http://www.stomp.ac.ukhttp://www.stomp.ac.uk. To locate e-questions for a variety of assessment systems (including QTI v1 & 2) look at the Physical Sciences Question Bank, http://www.heacademy.ac.uk/physsci/ (follow Q for Question bank) or (better) from a link in the SToMP site. (The HEA site is a legacy site, but the SToMP site is maintained) Transforming Assessment with Adaptive Questions