1. INQUIRE: Interactive Questions Reinforcing Education Sophie Clarke and Katharine Lindsay Academic Computing Development Team (Learning Technologies Group) Chris McKenna and Steve New Saïd Business School

2. The INQUIRE Project To what extent can on-line multiple choice questions provide a useful support mechanism to undergraduate teaching at Oxford? Small scale investigative pilot

3. A complex learning environment Introduction to Management Lectures, Tutorials, Reading, Essays MCQ Tests primarily based on the lectures

4. How lectures fit in… A complex learning environment

5. Our MCQ model Formative assessment Carefully crafted questions Pointers to other material embedded Low emphasis on technology 6 lectures 6 tests

6. Our MCQ model

10. Research Questions (1) Reinforcement of Learning from other pedagogic elements –Other factors to consider Cementing Students understanding –Simple test of knowledge Research Questions

11. Research Questions (2) Deepening Students knowledge –New topics –Credible resources Framing educational expectations –Confirm expectations Research Questions

12. Evaluation strategy Consulted experts & case studies on evaluation 6 possibilities - system logs, automated tracking of link following, confidence logs, 3 minute papers, video observation, focus groups. 3 chosen methods – mix of qualitative & quantitative. Not cast-iron study

13. Evaluation - Focus Groups 30 students recruited on second day of term £5 HMV voucher Group A: online tests Group B: paper tests Group C: control Focus groups held in week 6 Evaluation Strategy

14. Evaluation - Technology based Following links - simple PHP script to see which students followed which links (anonymous) Login times and frequency recorded by participant. Quia Evaluation Strategy

15. Revisiting the Research Questions (1) Reinforcement of learning from other pedagogic elements. –Paper tests not popular –Light usage of online tests –Trend of late night quizzing Student5: It’s like something you can do that you feel like you’re doing something useful... Student1: And that, it’s kind of like in between time, when you’re doing, after dinner, doing work and maybe you go out later to meet your friends or something.

16. (2) Cementing Students Understanding –In general patchy retention of the details of the lectures. Especially in the paper & control groups. –Online group had greater retention The tests jogged their memories to reinforce ideas and facts that had been discussed in the lectures...it kind of triggered your memory, I waited about a week, two until after the lectures to do it, I remembered it better in my mind, doing that. Images played an important part in recalling the lecture. Revisiting the Research Questions

17. (3) Deepening students knowledge –No groups followed the lecture reading list. –Paper Group did not follow links or references. Not at a computer when doing the tests, too cumbersome to type out URLs, too much reading. –Online Group - trend of click-and-glance. Contradiction between logs and what the students said. Interviewer:... for the ones [questions] which had the links, which could take you off and give you something more, long to read, did anyone look at those? Student1: No, I didn’t. Student5:I read some, I just didn’t include them. Student3:Depends how long they were. Student3:Sometimes I read through them but sometimes I thought, ooh that’s too long, for the moment. Revisiting the Research Questions

18. (4) Framing Educational expectations –Students realised that should be remembering the lectures and return to the handouts. –Acknowledge that they should be reading more. –Realisation that spoon-feeding is not a part of the course. Revisiting the Research Questions I’d like more encouragement as to what we’re actually meant to be learning.

19. Rigid study structure v optional online learning – hard for the students to see how it fitted in with the course requirements and blend their learning. Talismanic status of the lecture handout. Low threshold of convenience (passwords/ internet speed/ computer availability) Given reading for the lectures as well as tutorials but often this is not referred to. Some General Findings Oxford is like living in a bubble - I’m living within it and everything else is completely outside

20. Conclusions Small-scale evaluation project. Worked to reinforce the lectures, cement student understanding and frame educational expectations. Value appears to be in ‘jogging effect’ of re-living lecture, and in returning students to lecture notes/handouts Additional resources must be engaging & convenient. We need to look more closely at how we teach.

21. Does use of MCQs assist exam performance? Would the use of MCQs be different nearer to the exams? How can we fit the MCQs more into the traditional Oxford system. Look more into motivating students to deepen their knowledge through wider reading. Suitable software - reusability (QTI) The return on gold-plating questions is high but time consuming – would the outcome be as good if widened out across the department? Further Investigation