1 Evaluating the Quality of the e-Learning Experience in Higher Education Anne Jelfs and Keir Thorpe, Institute of Educational Technology (IET), The Open University

2 Introduction  Complexity of student experience  Problems facing evaluators  Different research methodologies  Audits promote evaluation  Holistic view – why? how? when? and what? of evaluation

3 Pressures  Develop transferable student abilities  Pressure within the UK and worldwide  Governmental and employer pressure  Requirement – learners to participate in a networked information society  Emphasis on student-centred learning

4 Evaluators  Experienced evaluators needed in production, usability and effectiveness of online delivery  The Internet and ‘blended learning’ with ICT elements now widespread

5 Two Sides of e-Learning  Promoted by government initiatives  Cost effectiveness of e-Learning  Permits tailored education  Can develop in ways which impoverish education  Time & production costs of development  Students & tutors lacking skills to use

6 The Impact of Quality Assurance  Parallel to increased use of ICT is rising interest in quality assurance strategies  Lighter QAA touch & emphasis on stakeholders – students, employers, funders, society  Emphasis on outcomes rather than processes  QAA looking for an institution’s strategy

7 Why Evaluate the Learner’s Experience?  Subject review & institutional audits  Act as: educational ‘connoisseurs’ with approaches and tools to use in order to act as experts informing policy  Require: awareness and understanding of what is experienced  Require: personal judgement  Require: ability to articulate subtleties to a wider audience

8 Questions to Ask yourself  Is the student experience exceptional?  Are there defects in the system or delivery?  Fit for purpose?  Value for money or adds costs?  Transformative/Value adding?  Students’ perceptions?  Effect on student numbers?

9 Metrics for e-Learning Evaluation 1.Efficiency of the teaching and learning process 2.Readability & usability of materials 3.Communication 4.Social relationships 5.Costs

10 Quality of the e-Learning Experience 1.Efficiency of the teaching and learning process  Accurate student data  Drop out rates  Student assessment results  Student satisfaction ratings  Training of tutors

11 Quality of the e-Learning Experience To best understand the potential for e-Learning:  Start with 1-2 courses  Is this course needed by students?  Is the assessment electronic?  Collection of computer log data 1.Efficiency of the teaching and learning process

12 2. Readability & usability of materials  Peer review of materials  Reuse as learning objects  Online exercises  Study skills  e-Library and e-journal access 3. Communication  Communication with tutors & other students  Skills in using technology  Access from home/campus/3 rd party  Group work Materials and Communication

13 4. Social Relationships  Hours working online  Student motivation  Student support & interaction with other students 5. Costs  Costs per student of recruitment  Training of tutors  Costs of electronic access  Reuse of learning materials & learning objects Social Relationships and Costs

14 Robson’s 3 Approaches to Human- Computer Interaction 1.Learning theory is dominant – how students and their interaction with computers fit together 2.How students learn when using computers 3.Using the technology to teach in ways that are novel and unique to the environment

15 How are we going to Evaluate the e-Learning Experience?  Evaluation goals need to be clearly defined  Is this a formative or summative evaluation?  How should the data be gathered?  Should quantitative or qualitative methods be used?

16 How are we going to Evaluate the e-Learning Experience? Methods include:  Surveys  Observations  Semi-structured interviews  Focus groups  Cognitive walk-throughs  Qualitative studies can provide examples of usability difficulties

17 How are we going to Evaluate the e-Learning Experience? Surveying students’ views:  detects feelings of overload  informs necessary course adjustments Survey methods:  paper based with  telephone supplementary interviews  increased use of electronic surveys  focus groups

18 When to Evaluate the e-Learning Experience?  Dependent on whether the evaluation is formative or summative?  When is the student deemed to have completed the course?  Will formative evaluation during the course engage students more?  Harvesting evaluation information from student interaction with the course

19 What does the Evaluation provide?  Audio/video record of the computer screen, the user’s comments and keystrokes Baseline data to:  make adjustments, possibly ‘instant’ feedback  for external audit  shape e-learning strategy  input into associated resources – library and information literacy provision

20 Data Protection  Students and tutors need to be kept informed about how the data they provide is being used  Students and tutors retain the right to remove what they have provided at any time  Evaluators have to ensure secure access to the data  Evaluators must get consent especially if using the data in a different context

21 Conclusion  The Open University is interested in the quality of the student experience  To improve that experience needs understanding of student and staff needs  Here we have aimed to provide a framework in which to conduct evaluations  Current work on modelling courses will provide data on points of concern that arise in blended and e-learning

22 To praise & ask queries, contact Anne: To complain, contact Keir: Contact