Evaluating E-Learning TECH665

Evaluation Evaluation defined Evaluation defined Evaluation defined Evaluation defined Formative evaluation Formative evaluation Formative evaluation Formative evaluation Summative evaluation Summative evaluation Summative evaluation Summative evaluation Illuminative evaluation Illuminative evaluation Illuminative evaluation Illuminative evaluation Difference between assessment and evaluation? Difference between assessment and evaluation? Difference between assessment and evaluation? Difference between assessment and evaluation?

Quantitative, Qualitative Quantitative Quantitative –Focuses on measurement, externally directed, value-free –80% of enrollments still active –Initial reaction to method of delivery was negative due to complex nature of WebCT/BB used at this time – resulting in 30% drop-out rate

Quantitative/Qualitative Qualitative Qualitative –Focuses on educational process –Directed by user and takes account of values –Student comments and evidence for reasons for change »Stop/Start/Continue method What should we stop doing and why? What should we stop doing and why? What should we start doing and why? What should we start doing and why? What should we keep doing and how are these things useful to you? What should we keep doing and how are these things useful to you?

Evaluation and Assessment Assessment – process whereby teachers set specific tasks related to the learning outcomes which students undertake to do. Their success in these tasks provides evidence of how effective their learning has been. Assessment – process whereby teachers set specific tasks related to the learning outcomes which students undertake to do. Their success in these tasks provides evidence of how effective their learning has been. Assessment data is one of the inputs to evaluation Assessment data is one of the inputs to evaluation Evaluation looks at a broad range of evidence in order to gauge the effectiveness of a project. Evaluation looks at a broad range of evidence in order to gauge the effectiveness of a project.

Illuminative Evaluation Observational approach to evaluation inspired by ethnographic research and methods. Observational approach to evaluation inspired by ethnographic research and methods. Aim – to discover the factors and issues that are important to the participants in a particular situation rather than how well an innovation performs against standard measures of evaluation. Aim – to discover the factors and issues that are important to the participants in a particular situation rather than how well an innovation performs against standard measures of evaluation.

Summative Evaluation When the dinner guest tastes the soup, it is summative evaluation (Harvey, 1998). When the dinner guest tastes the soup, it is summative evaluation (Harvey, 1998). Describes the evaluation of course materials or learning environments with the objective of providing information on the outcomes of implementation and use by students. Describes the evaluation of course materials or learning environments with the objective of providing information on the outcomes of implementation and use by students.

Formative Evaluation When a cook tastes the soup, it is formative evaluation (Harvey, 1998). When a cook tastes the soup, it is formative evaluation (Harvey, 1998). Describes the evaluation of course materials or learning environments with the objective of providing information for improvement during the design and implementation phases. Describes the evaluation of course materials or learning environments with the objective of providing information for improvement during the design and implementation phases.

Evaluation Defined Process through which we examine the learning opportunities and experiences we offer our students and make judgments about their effectiveness and value Process through which we examine the learning opportunities and experiences we offer our students and make judgments about their effectiveness and value

Characteristics Flexible Flexible –Works for a variety of modules; formative and summative Simple, easy to implement Simple, easy to implement –Everyone involved understands the process Reliable, accurately measures results Reliable, accurately measures results –Can you believe the results? Consistent from project to project? Minimal cost Minimal cost –Limited effects on budget, schedule, and staffing requirements

Breadth and Depth Microscopic Microscopic –A course component –A specific course –Whole curriculum Macroscopic Macroscopic –Institution –Industry –Society

Timing Can be carried out at different times, for different audiences, and for different purposes Can be carried out at different times, for different audiences, and for different purposes Three points Three points –During planning process »Predict the success of the proposed project –Between development and deployment »Rate the completed project –After delivery »Tabulate the success of the project

Kirkpatrick Classic four levels of evaluation Classic four levels of evaluation 4 – Results: How well did the organization meet its business goals? Profitable? 4 – Results: How well did the organization meet its business goals? Profitable? 3 – Performance: How much is job performance improved? What skills can be applied? 3 – Performance: How much is job performance improved? What skills can be applied? 2 – Learning: What skills and knowledge were acquired? 2 – Learning: What skills and knowledge were acquired? 1 – Response: Did learners like the training? Did they complete it? 1 – Response: Did learners like the training? Did they complete it?

Kirkpatrick Level 1 – frequently done, smileys Level 1 – frequently done, smileys Levels 2, 3, and 4 completed less frequently in that order even though value increases in that order Levels 2, 3, and 4 completed less frequently in that order even though value increases in that order Why do you think that is? Why do you think that is?