ALT-C 2004 Evaluating the e-Learning Experience Prof Tony Toole University of Glamorgan September 2004

ALT-C 2004 Summary Based on a JISC project evaluating the effectiveness of the e-College Wales MLE {part of the MLEs for Lifelong Learning programme} Based on a JISC project evaluating the effectiveness of the e-College Wales MLE {part of the MLEs for Lifelong Learning programme} Compared the learning journey taken by the students with that intended by the learning designers Compared the learning journey taken by the students with that intended by the learning designers Found that a ‘one size fits all’ model does not work for the lifelong learner Found that a ‘one size fits all’ model does not work for the lifelong learner Led to a new learning design model based on student choice to be developed and piloted by the Wales e- Training Networks Led to a new learning design model based on student choice to be developed and piloted by the Wales e- Training Networks

ALT-C 2004 Evaluation Methodology Focus group meetings and semi-structured interviews with all key stakeholders Focus group meetings and semi-structured interviews with all key stakeholders Instructional designers, content authors, multimedia specialists, web programmers Instructional designers, content authors, multimedia specialists, web programmers Programme managers, e-moderators, customer support team Programme managers, e-moderators, customer support team Students at all institutions in delivery network, including those at level 1, 2 & 3 of the degree programme and a substantial number of those who had withdrawn Students at all institutions in delivery network, including those at level 1, 2 & 3 of the degree programme and a substantial number of those who had withdrawn

ALT-C 2004 The Learning Journey Structured materials Navigation Structured tasks Discussion Forum Assessment

ALT-C 2004 The Learning Journey The learning was intended to be linear, the students working through the materials topic by topic The learning was intended to be linear, the students working through the materials topic by topic The students were expected to complete the tasks associated with each topic and contribute to the discussion about it The students were expected to complete the tasks associated with each topic and contribute to the discussion about it The students would then complete the module assessment and progress The students would then complete the module assessment and progress

ALT-C 2004 But …. Our lifelong learners turned out to be transactional students with different levels of experience, expertise, learning styles and reasons for studying Our lifelong learners turned out to be transactional students with different levels of experience, expertise, learning styles and reasons for studying As a result many didn’t take the route we had designed for them & chose a path that matched their way of doing things As a result many didn’t take the route we had designed for them & chose a path that matched their way of doing things The route they took depended on their motivation for taking the course and what personal benefits they wanted it to deliver The route they took depended on their motivation for taking the course and what personal benefits they wanted it to deliver

ALT-C 2004 Our typical experienced student with an eye on the qualification would … Go straight to the assessment to find out what was required Go straight to the assessment to find out what was required Scan the materials to identify the scope and depth of the module Scan the materials to identify the scope and depth of the module Examine the tasks to get a better feel for what the assessment outcomes were expected to be Examine the tasks to get a better feel for what the assessment outcomes were expected to be Participate in the discussion forums for the purpose of confirming the conclusions drawn Participate in the discussion forums for the purpose of confirming the conclusions drawn Target the topics where skills and knowledge reinforcement was required Target the topics where skills and knowledge reinforcement was required Complete the assessment when the skills and knowledge necessary for the desired grade were judged to have been attained Complete the assessment when the skills and knowledge necessary for the desired grade were judged to have been attained

ALT-C 2004 Our typical experienced student with the intention of developing their skills would … Scan the materials to identify the areas of useful new knowledge Scan the materials to identify the areas of useful new knowledge Read thoroughly through the materials they had chosen, including additional resources, and carry out relevant tasks Read thoroughly through the materials they had chosen, including additional resources, and carry out relevant tasks Participate in the discussion forums where this contributed to their learning goals Participate in the discussion forums where this contributed to their learning goals Complete the assessment when the deadline approached. The main objective here was to progress to the next module with the minimum effort rather than achieve a good grade Complete the assessment when the deadline approached. The main objective here was to progress to the next module with the minimum effort rather than achieve a good grade

ALT-C 2004 Our typical inexperienced student with the intention of developing their skills would … Largely follow the linear structure of the course and read through all materials Largely follow the linear structure of the course and read through all materials Complete the tasks and participate in the discussion forums up to the point where no-one else was contributing Complete the tasks and participate in the discussion forums up to the point where no-one else was contributing Attempt the assessment when the deadline was approaching Attempt the assessment when the deadline was approaching Welcome f2f tutorials/informal student meetings to compare notes and reinforce level of attainment Welcome f2f tutorials/informal student meetings to compare notes and reinforce level of attainment

ALT-C 2004 Lessons we have learned …. We need to provide multiple routes through the materials and allow students to make their own choices We need to provide multiple routes through the materials and allow students to make their own choices We need to communicate proactively with the students, not reactively We need to communicate proactively with the students, not reactively We need to increase the number of assessment points and provide formative feedback We need to increase the number of assessment points and provide formative feedback We need to exploit the skills of experienced students for the benefit of those with no experience We need to exploit the skills of experienced students for the benefit of those with no experience We need to provide plenty of opportunities to communicate for those who are most comfortable in a community of learners, whilst allowing those who wish to plough a lone furrow to do so We need to provide plenty of opportunities to communicate for those who are most comfortable in a community of learners, whilst allowing those who wish to plough a lone furrow to do so

ALT-C 2004 A New Learning Design Model Based on constructivist, situated learning approach employing discovery learning activities in communities of practice Based on constructivist, situated learning approach employing discovery learning activities in communities of practice Modular with clusters of credit rated learning outcomes. Three levels of learning support resources Modular with clusters of credit rated learning outcomes. Three levels of learning support resources Student chooses own mix of learning outcomes to attain, level of support required and where & when the learning will take place Student chooses own mix of learning outcomes to attain, level of support required and where & when the learning will take place Negotiated learning agreement with module tutor Negotiated learning agreement with module tutor

ALT-C 2004 Module Descriptor {This will explain the learning objectives of the module and the number of activities to be carried out} Learning Activities {Learning Activity 1. Description, learning outcomes, student choices, examples}. {Learning Activity N. Description, learning outcomes, student choices, examples} Choose Learning Activity {choose personal application, agree with tutor} Need Learning Materials? Level 1 Materials {Bullet point explanations, assumes knowledge of software, menu structures and all basic functionality} {bullet point 1: explanation}. {bullet point N: explanation } Level 2 Materials {Further explanation of topic, example, exercise) Need Further Materials? Level 3 Materials {Full topic coverage, examples, exercises, tutorial) Need Further Materials? Y Y Y N N N Complete Learning Activity {discuss drafts with tutor} {agree final version} {submit} Learning Activities Complete? Y N End Module {discuss with tutor} Start Module {discuss with tutor}

ALT-C 2004

The Wales On-line Foundation Degree Collaboration of all post-16 institutions in Wales Collaboration of all post-16 institutions in Wales Shared development of FD modules Shared development of FD modules Common learning design model Common learning design model Shared digital repository Shared digital repository Validated at NVQ level 4 Validated at NVQ level 4 All institutions able to exploit materials in whatever way they wish All institutions able to exploit materials in whatever way they wish

ALT-C 2004 For further information on: E-College Wales The Welsh e-Training Networks Contact: