Evaluating the Impact of a VLE on Learning and Teaching? Dr Barbara Newland Learning Design Studio Academic Services Bournemouth University

Overview Aims of evaluation Durham context Methodology Results Conclusions

Aims of evaluation International widespread adoption of VLEs Longitudinal study of impact of a VLE on learning and teaching Qualitative and quantitative methods

University of Durham 670 academic and 280 academic-related staff 9400 undergraduates and 2400 postgraduates Founded in 1832 Split campus Research led MIS – outsourced to Unisys Learning Technologies Team (LTT)

Images of Durham

Learning Technologies Team aims to encourage, support and develop elearning throughout the University by: providing advice, information and training supporting the University's learning environment - duo (Durham University Online) developing interactive materials 4 members of staff – Juliette Pavey, Kate Boardman, Malcolm Murray, Victoria Boyd

Duo Durham University Online

Uptake of duo Estimated use by Xmas 2000 Actual use by Xmas 2000 Use at June students 3 departments 4286 students 21 departments 211 modules 10,000 students 700 academics 700 modules

Implementation Varied patterns of departmental implementation All courses All first year courses Individual academics Administration courses Duo is now embedded into learning and teaching

Methodology for duo surveys Quantitative and qualitative Online surveys Analyse factors such as gender, year of study and faculty 3 faculties – Arts and Humanities, Science, Social Science and Health Relate to C&IT skills in learning and teaching Data collected from students and staff in all faculties and both campuses

Response rate All students and staff are in duo so hard to know the response rate in relation to active users which has increased over the years StaffApril 01 (n = 91) May 02 (n = 153) May 03 (n = 283, c. 40%) Students May 01 (n = 793) May 02 (n = 2435) May 03 (n = 3668, c. 37%)

Survey questions Similar questions for staff and students in all 3 years to enable comparisons : between years between staff and students Software ease of use levels of confidence features used

Survey questions Frequency of use and access during term time and vacations Learning and teaching contribution effect

Student responses – support, frequency and access Help documents 19%, 15%, 10% Small and decreasing use Accessed duo at least once a week 70%, 71%, 71% Consistent pattern but increasing numbers over time Accessed duo out of term time 60%, 61%, 64% Consistent pattern but increasing numbers over time

Students: Overall, how do you rate the contribution duo has made to your learning?

Staff: Overall, how do you rate the effectiveness of duo as a learning resource?

Staff: Has duo affected your approach to teaching?

Students: If duo is not currently being used for all your modules, would you recommend that it should be?

Staff use of duo features - content 62%, 49% staff have used more features 79%, 76% staff have developed content since the previous year Course information – 74%, 84%, 92% Course documents – 77%, 89%, 95% External links – 42%, 56%, 55% Bibliography – 35%, 53%, 43% (plus 17%, 21%, 21% link to OPAC) Staff information – 49%, 63%, 72%

Staff use of duo features - communication Group work – 25%, 23%, 26% – 72%, 77%, 80% Announcements – 75%, 84%, 86% Discussion board – 16%, 32%, 31% Chat – 10%, 4%, 6%

Staff use of duo features - assessment Assignments – 53%, 63%, 67% Quizzes – 11%, 20%, 26% Surveys - 15%, 13%, 21% Digital drop box – 10%, 6%, 15%

Reflect current ways of teaching Context – relevant, flexible (start from where they are at) Collis also found at the University of Twente that “instructors are most likely to begin by choosing aspects of a system that reflect their current ways of teaching, and then gradually move to new instructional approaches and new features.” (Collis and Messing, 2001)

Lecture notes - staff Is it good practice to put lecture notes online? Depends on what the lecturer does with them 2/3 put lecture summaries, handouts, full notes and/or Powerpoint slides after the lecture

Lecture notes - students Depends how students use them 73% listen more 81% clarify information 72% improve lectures 93% study for exams 2/3 (64%, 64%) students note-taking had stayed the same but for ¼ (27%, 26%) it had decreased Only 1%, 1% stated no longer attended lectures

Gender No statistical significance between gender and Staff with effectiveness as learning resource Students with contribution to learning Both with levels of confidence Higher percentage of females think duo helps them to manage their learning

Staff by faculty – skills More Science faculty found it very easy/easy to use compared to Arts Some Arts staff took longer to gain confidence (6-10 visits) Fewer Science staff chose to have training Reflects the levels of staff confidence in C&IT skills found in Audit June 2000

Staff by faculty – teaching no significant difference 2003 Basic approach is changing Science and social science 12% compared to 3% arts Basic approach has not changed but duo helps me do certain things better Science 79%, social science 76% compared to 98% arts

Students by faculty Contribution that duo has made to their learning overall no significant difference 2003 – higher percentage of science (69%) and social science (52%) rate “good” or “excellent” compared to 42% arts students Higher percentage of social science students think duo helps them to manage their learning “greatly” or “quite a lot” compared to science students

Students by year group Contribution that duo has made to their learning overall 2001 and nd and 3 rd year students rates more higher than 1 st years (1 st years have always had duo) 2003 – 1 st year students rate slightly higher than 2 nd and 3 rd years

C&IT skills survey of new undergraduates – 2002 n = % of students perceive themselves to be confident/very confident computer users 60% have brought a computer with them to University 69% have been using a computer for 5+ years (including 21% for over 10 years) and only 2% have been using a computer for <2 years

C&IT skills survey of new undergraduates % perceive themselves to have good/advanced skills 65% perceive themselves to have good/advanced web browser skills 70% stated a preference for a combination of web and paper based for learning support materials

C&IT skills survey of new undergraduates % own a mobile phone 5% have a PDA 38% take part in online discussions 59% take part in online chat 49% have a qualification in IT 54% regularly switch between several applications when using a computer

C&IT skills survey of academic staff n = 576 (53%) 90% perceive themselves to have good/advanced skills 67% perceive themselves to have good/advanced web browser skills only 17% of staff had taken part in online discussions only 12% of staff had taken part in online chat

Catalyst Introduction encouraged continual reflection on teaching Postgraduate Certificate in HE compulsory for new academics from Jan 2001 Academics working together within courses

Factors influencing enhancement How used How integrated into curriculum Quality of the parts eg content, assessment, communication (earlier research) Enhance learning if the parts enhance learning Flexibility allows lecturers to choose parts

Is the whole greater than the sum of its parts? But is a VLE more than the pieces put together?

Evaluating the impact of a VLE on learning and teaching Incredible uptake of duo which has become embedded into the University Staff and students believe duo has enhanced teaching and learning Academics are gradually using more features so this may result in more interactive and innovative teaching Students expectations are increasing as the basic use of a VLE is assumed Further analysis is being undertaken

References Collis, B. and Messing, J. (2001) Usage, attitudes and workload implications for a web-based learning environment, Association of Learning Technology Journal 9 (1) Collis, B. and Moonen, J. (2001) Flexible Learning in a Digital World, London, Kogan Page Laurillard, D. (1993). Rethinking University Teaching: a Framework for the Effective Use of Educational Technology. London: Routledge. Richardson, J., & Turner, A. (2000). A large-scale ‘local’ evaluation of students’ learning experiences using virtual learning environments. Educational Technology and Society, 3 (4). Salmon, G. (2000). E-moderating: the Key to Teaching and Learning Online. London: Kogan Page.