2. Evaluating Training Programs Level 5: Return on Investment Kelly Arthur Richard Gage-Little Dale Munson

3. E-Poster Objectives  Define the fifth level of evaluation  Explain how to find Return on Investment (ROI) for training programs  Identify benefits and disadvantages 2 of 21

4. Building on Kirkpatrick’s Four Levels 1. Reaction 2. Learning 3. Behavior 4. Results 5. Return on investment (ROI) 3 of 21

5. Why Was It Created? The fifth level was created because Kirkpatrick’s four levels only focus on the effectiveness of training events and not monetary benefits. Instructional Developers needed a way to demonstrate cost-value and accountability for their training. 4 of 21

6. What Is the Fifth Level? Known as ROI, the fifth level was published by Jack Phillips to quantify the monetary value of training investments. ROI answers the question, “For every dollar invested in training, how many dollars does the employer get back?” In other words, show me the money. 5 of 21

7. What Does it Do? ROI converts Kirkpatrick’s fourth level data (business results) into monetary values and compares it with the cost of training. ROI is only a financial measure of training’s value, it does not provide information to improve training. ROI% = Benefits – Costs of Training x 100 Costs of Training 6 of 21

8. Why Should ROI Be Used? “Many programs have failed to deliver what was expected; consequently, program sponsors have requested ROI calculations/justifications.” “The concern for accountability in all functions in organizations is increasing; thus, the training and development function becomes one of many support efforts under scrutiny.” Jack Phillips 7 of 21

9. Isolate training effects 2 Convert data into $$$ 3 Tabulate program costs 4 Calculate ROI 5 Collect data 1 Identify intangible benefits 6 How to Determine ROI (animation) 8 of 21

10. Step 1: Collect Data  Hard data – objective and easy to measure –Output (units produced, items sold, etc.) –Quality (scrap, waste, rework, product defects) –Time (equipment downtime, employee overtime)  Soft data – subjective and difficult to measure –Work habits (employee absenteeism, tardiness) –Work climate (employee grievances, turnover) –Attitudes (employee loyalty, self-confidence) –New skills (decisions made, problems solved) 9 of 21

11. Step 2: Isolate Training Effects  Because many factors influence performance, it is necessary to determine the amount of improvement that is directly attributable to training.  Example: After attending a training program, sales people sold 20 percent more widgets over the next six months. Was it the training? Or, maybe it was the new reorganization or new incentive program. 10 of 21

12. Step 3: Convert Data into $$ Values In this step, level four data are converted into monetary values and compared against program costs.  Level four measurements: –Documented improvements in the standard of work and/or care delivered to clients. –Service targets being met more effectively. –Work being accepted by others (repeat business, testimonials, complementary letters). –Greater measurable productivity, improved safety records, etc. 11 of 21

13. Step 4: Tabulate Program Costs Costs can be categorized into three types: 1. One-time costs: –Analysis and design –Program development –Equipment and software 2. Cost per offering: –Facility rental –Instructor's salary 3. Cost per participant: –Travel –Per diem expenses –Tuition 12 of 21

14. Step 5: Calculate ROI Formulas 1. ROI = Total Cost of Training Number of Students 2. ROI (%) = Training Costs - Benefits x 100 Training Costs 13 of 21

15. Step 6: Identify Intangible Benefits Intangible benefits are activities or conditions that are extremely difficult to quantify. Examples:  Improved teamwork  Improved customer service  Increased job satisfaction  Reduction in customer complaints  Improved communications  Increased flexibility  Employee morale 14 of 21

16. The ROI Formula in Action Krispy Kreme’s evaluation research determined that there was a 10% increase in the number of donuts sold after a new selling skills training program was conducted. Other data revealed that each 1% increase in sales is equal to increased annual revenue of $25,000 for a total of $250,000 in benefits. The cost of training was $75,000. What was the ROI? Benefits - Costs of training x 100 = %ROI Costs of training 15 of 21

17. Let’s Work it Out (animation) ($250,000 - $75,000)(100) $75,000 = ($175,000)(100) $75,000 ($17,500,000) $75,000 = 233% So, for every $1 invested in training, the organization realized a benefit of $2.33. Benefits - Costs of training x 100 = %ROI Cost of training 16 of 21

18. Some Benefits of ROI Evaluations ROI also provides:  A comprehensive measure of training programs  Comparability between training programs  Benefits and cost in percentage terms 17 of 21

19. Some Limitations of ROI Evaluations It is estimated that only 14% of organizations actually evaluate ROI. Why?  Many companies don’t perform level four evaluations  Difficult to isolate training benefits  ROI formula is only an estimation 18 of 21

20. Summary  ROI is the fifth level of evaluation  It provides a monetary value for training investments  ROI is only concerned with how much a training is “worth”  The ROI formula can be used to determine if the benefits of training outweigh the costs of training  ROI is difficult to determine effectively 19 of 21

21. Helpful URLs http://www.learnativity.com/roi-learning.html http://www.performancexpress.org/0205/mainframe0205.html http://www1.astd.org/news_letter/October/Links/phillips.html http://www.performancexpress.org/0208/mainframe0208.html http://www.arches.uga.edu/~rali/edit8350/ROI/default.htm 20 of 21

22. References Long, Larry. (1999). ROI: Capturing the Big Picture. Technical Training. Retrieved November 12, 2002 from the World Wide Web http://www.astd.org/virtual_community/comm_evaluation/long.pdf Martin, John. (2000). ROI: Do It & Prove It. ASTD. Retrieved November 12, 2002 from the World Wide Web http:www.astd.org/CMS/templates/index.html?template_id=1&articleid=2 6100 Phillips, Jack. (1996) Measuring ROI: The Fifth Level of Evaluation. Technical & Skills Training. Retrieved November 12, 2002 from the World Wide Web www.astd.org/virtual_community/ comm_evaluation/phillips.pdf 21 of 21