1. Objective assessment Bob Rotheram Reader in Assessment, Learning & Teaching b.rotheram@leedsmet.ac.uk

2. Intended learning outcomes (Biggs 2003) write good quality objective test items; answer (mostly) correctly some questions about today’s subject material; identify what you need to do next.

3. Tales of a dissatisfied student Maths Social work

4. The importance of… Assessment Constructive alignment (Biggs 2003) Feedback

5. Seven pillars of assessment wisdom (Elton 2003) “The drunkard looks under the lamp post for the keys which he has dropped, not because that is where he has dropped them, but because that is the place where there is light.” Reliability pursued at expense of validity. Students should get a profile. Assessments to be valid, reliability treated within constraints of validity.

6. Constructive alignment (Biggs) State intended learning outcomes (ILOs) Find verbs students must perform to achieve outcomes Assessment tasks require students to enact the verbs Course activities to be aligned with enabling achievement of ILOs

7. Feedback Potentially very powerful (Brown G, 2001) Best if: timely perceived as relevant meaningful suggests ways of improvement (within student’s grasp)

8. Computer-assisted assessment (CAA): a demo Social Policy Question Resource (SPQR) 10-question demo Social Policy Question Resource (SPQR) 10-question demo

9. SPQR: some uses Summative assessment Diagnostic tests Formative tests Informal quizzes Revision aid Convey what social policy is Individual questions v. flexible, ‘granular’, ‘learning objects’: insert into lectures gather further info via web links preparation for classes …

10. Resource-based learning (& CAA) best with… Important background ‘Need to know before…’ ‘Remedial’ ‘Nice to know’ ‘Much-repeated’ ‘Learned by doing’ Individual feedback needed Stuff you don’t like to teach! Hard to grasp first time May be needed later, at short notice (Race 2007)

11. CAA in your discipline Use? Potential/advantages? Limitations/disadvantages?

12. CAA: key points Much ‘up-front’ effort needed Best where: large numbers of students on introductory courses questions have long ‘shelf-life’ teams can produce shared item banks

13. Objective tests “require a user to choose or provide a response to a question whose correct answer is predetermined” (Bull & McKenna, 2001) “only as objective as the test designer makes it” (CAA Centre) Reputation mixed Best for lower-level skills???

14. Bloom’s Taxonomy (1956) knowledge comprehension application analysis synthesis evaluation

15. Some question types multiple-choice multiple response matching text match response/fill in blank extended matching set ‘sore finger’ graphical ‘hotspot’ true/false, yes/no multiple true/false ranking Likert scale numeric assertion/reason essay

16. Knowledge-testing questions What word means the same as…? What is the most important difference between…? Which one of the following sequences shows the correct order of…? What are the major classifications of…? Which method is the most useful for…? What evidence best supports the theory of…?

17. Comprehension-testing questions (if not taught or discussed) Which one of the following is closest in meaning to the term…? The statement ‘…’ means that… (Various facts are presented.) Which of the reasons listed below best explains this?

18. Feedback: minimum statement on whether chosen option correct or not; (if incorrect) statement giving correct answer and why correct; clear up possible misunderstanding about wrong choice; source of info on correct answer; language clear and not discouraging.

19. Quality control Editorial board? Evaluate in use: Level of difficulty Discrimination – appropriate?

20. Sources of items Students Colleagues Staff development events Question-setting workshops Publishers’ websites

21. E.g. Oxford UP Test Banks, ‘VLE cartridges’Test Banks E.g. Bryman, social research methods:Bryman, social research methods 260 MCQs Check terms of use

22. Item-writing See ‘PASS-IT’ Guide, Section 2 Sample MCQs: ‘PASS-IT’ guide, Appendix A

23. Item-writing exercise (alone) create 1 MCQ, 1 matching Q, answers, feedback (2s/3s) ‘shred’ + refine items (whole group) further consideration of some items

24. Further issues Constructing assessments ‘Content-free test’ (Race) ‘Content-free test’ Scoring Computerisation, inc. interoperability Metadata Item banks

25. 1. Who, according to Bob Rotheram, said this: “Assessment is the most important thing we do for HE students.”? Lewis Elton John Biggs George Brown Sally Brown Phil Race

26. 2. Which of the following statements best reflects John Biggs’ concept of constructive alignment? A.State the learning outcome we want students to achieve. Find a verb that students need to perform to properly achieve the outcome. Assessment tasks should require students to enact that verb. B.Don’t just tell students about topics and then give them exams on what we have told them about problem-solving. C.Constructive alignment is joined-up programme design, making sure that all the components fit together properly. D.Constructive alignment is used in quality enhancement/assurance in university teaching.

27. 3. Which of the following does Brown G (2001) recommend as components of good feedback on student work? A.Meaningful B.Perceived as relevant C.Extensive D.Suggesting ways of improvement (within the student’s grasp) E.Timely F.Brief

28. 4. Which two levels of Bloom’s Taxonomy is this question most obviously testing?

29. 5. In the two lists below, match the questions with their types Multiple-choiceWhich of these items is the odd one out? Multiple-responseWhich of the following best explains…? Extended matching setGive the proper name for… Text match response/fill in blankOn this diagram, where is the…? Graphical hotspotHere are 10 possible answers and 6 questions. For each question, choose the correct answer(s). Sore fingerWhich, if any, of these statements is true?

30. 6. In each case below, indicate whether the statement is true or false. If software is ‘IMS QTI-compliant’, it separates questions from presentation. Computerised questions need to be IMS QTI-compliant to export them from one CAA package and import them into another. In 2005, Questionmark Perception did not fully meet the IMS QTI specifications. Respondus is fully IMS QTI-compliant. The ‘TOIA’ project and ‘JORUM’ repository take IMS QTI- compliance into account. The ‘RELOAD’ project does not take IMS QTI-compliance into account. The chance of getting all the answers to this question correct purely by guessing is one in seven.

31. Intended learning outcomes (Biggs 2003) write good quality objective test items; answer (mostly) correctly some questions about today’s subject material; identify what you need to do next.

32. Where next? Your next steps?

33. Sources (1) Biggs, J (2003) Teaching for Quality Learning at University (2 nd ed.), Maidenhead, Open University Press. Bull, J and McKenna, C 2001, Blueprint for Computer-Assisted Assessment: Chapter 3 and Appendix I (Republished 2004 by Routledge Falmer) Good Practice Guide in Question and Test Design. http://www.pass- it.org.uk/resources/031112- goodpracticeguide-hw.pdfhttp://www.pass- it.org.uk/resources/031112- goodpracticeguide-hw.pdf

34. Sources (2) Brown G, Bull J and Pendlebury M, (1997) Assessing Student Learning in Higher Education, London, Routledge. Elton L, (2003) Principles for a fair and honest approach to assessing and representing students’ learning and achievement, HE Academy website. Race P, (2007) The Lecturer’s Toolkit, (3 rd ed.) London, Routledge.