1. Evaluating the Effectiveness of Online Learning Programs ITEC 860 Fall 2003 Aniruddh Mukerji

2. Evaluation The systematic investigation of the worth or merit of an object Evaluation in the ISD Model Formative Evaluation / Internal Evaluation Summative Evaluation / External Evaluation The E in ADDIE 2

3. Kirkpatrick’s Model for Summative Evaluation 3 Level One: Reaction Level Two: Learning Level Three: Transfer Level Four: Results

4. Level One: Reaction Measures learner’s satisfaction with the course. Survey at the end of a Online training program asking questions about what learners liked and did not like about the program What?How? 4

5. Level One: Reaction Could go beyond the ‘smile sheet’ Add questions like: 1. The relevance of the objectives 2. The ability of the course to maintain interest 3. Amount and appropriateness of interactive exercises 4. The ease of navigation 5. The perceived value and transferability of the training to the workplace 5

6. Level Two: Learning Measures how much of the content a learner has mastered based on the course objectives Learning can be measured by pre- and post tests, making sure that test items or questions are truly written to the learning objectives What?How? 6

7. Level Three: Transfer Measures the amount of knowledge, skills and attitude learners are able to apply on the job several weeks or months after the course Observation surveys, sometimes called behavioral scorecard can be completed by the learner, the learner’s manager/supervisor or customers What?How? 7

8. Level Four: Results Measures the financial improvement to the organization’s bottom line several months or a year after the course Generally applies to training that seeks to overcome a business problem caused by lack of knowledge or skill. Isolate a representative control group within a larger student population and then rollout training, complete the evaluation and compare the results against a business evaluation of the non- trained group. What?How? 8

9. Level Four: Results Sales Training: Measure change in sales volume, customer retention, length of sales cycle Technical Training: Measure reduction in calls to help desk; reduced time to complete reports, forms or tasks; or improved use of software or systems Quality Training: Measure reduction in number of defects Safety training: Measure reduction in number or severity of accidents Management training: Measure increase in engagement levels of direct reports 9

10. Finally… Evaluation of training programs, both traditional and online, need to be conducted both before, during, and after training to truly assess the effectiveness of training. Like traditional classroom programs, an online learning program can be evaluated using Kirkpatrick’s four levels The number of levels you choose will depend on what your client or funder demands, what your budget allows and what your timeline permits. Most people choose to evaluate at the reaction and learning levels. 10

11. Other related topics Program Evaluation A systematic investigation of the worth of an ongoing or continuing distance education activity. Teaching and Learning at a Distance: Chapter 12 focuses on approaches to evaluation for the purpose of improving distance education and determining the worth of distance education activities

12. Resources e-Learning Centre: Evaluating e-Learning http://www.e-learningcentre.co.uk/eclipse/Resources/evaluating.htm Distance-Educator.com> Site Channels>Evaluators http://www.distance-educator.com/de_ezine/categories.php?op=newindex&catid=23 Return on Investment - An Evaluative Framework http://coe.sdsu.edu/eet/Articles/k4levels/index.htm 12

13. Sources BOOKS Technology Based Training by Kevin Kruse and Jason Keil (2000) Web-Based Training, 2 nd Edition by Margaret Driscoll (2002) Teaching and Learning at a distance by Simonson, Smaldino, Albright and Zvacek (2003) ONLINE Evaluation of the Effectiveness of Training Programs by Debra M. Farmer @ http://research.umbc.edu/~hodell/602rp7.htm Kirkpatrick's Four Levels of Evaluation @ http://coe.sdsu.edu/eet/Articles/k4levels/index.htm