1. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Improving performance on a Marketing module through the use of Formative Computer-Assisted Assessment

2. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 This paper was first presented at the 2nd European Conference on The First Year Experience Gothenburg University, 9th - 11th May 2007

3. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Work in progress

4. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 The delivery mode at MK1004 – Marketing Principles follows very much the traditional lecture model. Students have two in-class examinations. The first one is made of two parts: a MCQ (Multiple Choice Questions) test, consisting of 30 questions, and five open questions. The MCQ test accounts for 25% of the final student grade for this module. Introduction

5. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 There are two main reasons that pressed the author to conduct this research. First - the University of Wolverhampton is moving to a more virtual and self-learning environment through the use of WOLF. Second - the week before the first examination students are presented with an in-class formative MCQ test. Introduction

6. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 There are a number of ways in which a VLE can be used to assess students. The most widespread one is computer- assisted-assessment (CAA). One of the tools supplied by a VLE is self-assessment, through the use of a databank of questions, linked to automated marking and instant feedback; other frequent tool is the tracking of the students’ use of materials within the VLE. Both facilities are present in WOLF. Literature review

7. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 The correlation between the use of formative CAA and exam results have been researched by several authors and a positive link has been found. However, according to Clarke et al. (2004) research into the use of online formative MCQ still remains scarce and more studies are needed. Literature review

8. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007

9. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007

10. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Evaluation Methodology Reeves (1993) suggest that using a combination of quantitative and qualitative approaches is more suitable to assess learning supported by a VLE. In order to improve the validity of our study a number of sources – access logs, survey (with close and open end questions), automated tracking and analysis of in-class test results was used.

11. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Evaluation Methodology Access logs give us the opportunity to find out about accessing patterns of each student, in relation to date, hour and number of times a MCQ test was accessed and number of times it was completed. Automated tracking of results allowed the researcher to find out about the marks gained for each test.

12. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Table 4 – Tracking for first and last day of the online MCQ tests Total of 3512 page views and 1014 interactions

13. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Evaluation Methodology Analysis of in-class test results allow for historical comparisons with the three last cohorts of students (2005-06 Semester 1 & 2 and 2006-07 Semester 1) as well as within this specific cohort - by comparing students who did take the online MCQ tests with those that did not take them and within the group that did take the tests, by comparing formative grades with summative ones.

14. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Evaluation Methodology Finally, surveys were chosen as a way to get students’ opinions regarding the usefulness of the online MCQ tasks towards their learning and final results. Questionnaire: - 38 closed questions and 5 open questions - paper-based and completed in-class after the test - applied to 50% of the students enrolled in the module - a total of 114 questionnaires were received.

15. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Qualitative analysis The overall conclusion is that the online MCQ tests were a valuable learning resource for many students, helping them prepare for the summative in-class test. “the online MCQ tests motivated me to learn and to keep trying” (86%) “the online MCQ tests forced me to study more” (81%) “the marks helped me to access how I was doing on my learning process” (80%) “as a method of learning I enjoyed doing the online MCQ tests” (76%) “the online MCQ tests were useful in revising the content of the module” (76%)

16. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007

17. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Qualitative analysis Question 3 asked the students if they have enjoyed the online MCQ tests as a method of learning. Two students who did not access the programme agreed with the questions and one student who accessed it just once and spent only 4:04 minutes strongly agreed. One may ask why he didn’t spend more time in such an enjoyable activity!

18. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Qualitative analysis Question 4 asked the students if the online MCQ tests were well organised and structured. One student who accessed the task only once and for just 15 seconds did agree. We are delight to see that in just 15 seconds the student was still able to appreciate the organisation and structure of the task. Relying on a questionnaire alone seems, from these findings, to be highly problematic

19. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Quantitative analysis The last three cohorts had average grades of 16.58 (sd 3.75), 18.49 (sd 4.1), and 18.04 (sd 4.53). The current cohort had an average of 16.94 (sd 3.83)*. *sample of 64 students But

20. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Quantitative analysis If we take into consideration only the students that fully used the online tool (12 tests ) then the average goes up to 19.67 (sd 4.5). AVERAGESTDNb Students (%) HIGH (9,10,11,12) 19.313.9920.3% MEDIUM (5,6,7,8) 17.144.1921.9% LOW (1,2,3,4) 16.252.9537.5% NIL (0) 15.624.0520.3%

21. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 236 observations

22. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Model Nb of page views Nb of tests taken Average time spent per test Total time spent on tests CAA results Summative result R² =.002 R² =.103 R² =.109 R² =.097 β =.046* β =.321 β =.330 β =.312 R² =.304 β =.551 R² =.002 β =.041* R² =.066 β =.257 R² =.021 β =.144* R² =.291 β =.539

23. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 Conclusion and further research

24. Miguel Martins, Senior Lecturer – Marketing Department eLearning Celebration 22 July 2007 If you want to receive a copy of the final report please e- mail me at: mmartins@wlv.ac.uk Thank you Work in progress