1. Reasons for Evaluating Training Companies are investing millions of dollars in training programs to help gain a competitive advantage. To justify the costs To validate training as a business tool To determine whether or not the training contributed to the achievement of organizational goals

2. Reasons for Evaluating Training (continued) To determine how well the training met the needs identified To determine how well the learners mastered the training content To determine whether or not the methods used helped the learners achieve the learning objectives To help improve the design of training

3. Training evaluation involves: Formative evaluation – Formative evaluation – evaluation conducted to improve the training process. Summative evaluation – Summative evaluation – evaluation conducted to determine the extent to which trainees have changed (learned) as a result of participating in the training program. Cost/Benefit Evaluation/ROI - extent to which the benefits exceed the costs of training

4. Outcomes Used in Evaluating Training Programs: Cognitive Outcomes Skill-Based Outcomes Affective Outcomes Results Return on Investment

5. Outcomes Used in Evaluating Training Programs: (continued) Cognitive Outcomes Cognitive Outcomes Determine the degree to which trainees are familiar with the principles, facts, techniques, procedures, or processes emphasized in the training program. Measure what knowledge trainees learned in the program (e.g., safety rules, steps in a behavioral interview) Skill-Based Outcomes Skill-Based Outcomes Assess the level of technical or motor skills & behaviors Include acquisition or learning of skills and use of skills on the job (e.g., use a HRIS system, coaching skills)

6. Outcomes Used in Evaluating Training Programs: (continued) Affective Outcomes Affective Outcomes Include attitudes and motivation. Trainees’ reactions to or perceptions of the program including the facilities, trainers, and content (e.g., satisfaction with training or the instruction, tolerance for diversity, motivation to learn, customer service orientation) Results Results Determine the training program’s payoff for the company (e.g., absenteeism data, turnover data, accident reports, sales data)

7. Outcomes Used in Evaluating Training Programs: (continued) Return on Investment (ROI) Comparing the training’s monetary benefits with the cost of the training. Direct costs Indirect costs Benefits

8. Training Outcomes: Kirkpatrick’s Four-Level Framework of Evaluation Criteria LevelCriteriaFocus 1ReactionsTrainee satisfaction, perceptions of the training experience Done right after training 2LearningWhat knowledge, skills, attitudes, have changed and by how much? Done right after training 3BehaviorImprovement of specific job behavior or job performance Do employees actually use on the job what they learned? First time transfer of training is measured. 4ResultsDid the change in behavior positively affect the org.? Can include costs related to accidents, turnover, product quality, customer service, sales, # of grievances, just to name a few. Data collected before and after training, then compare differences. ROI can be included here.

9. Measuring Kirkpatrick’s Levels Level 1 – surveys or questionnaires measuring perceptions and satisfaction with all aspects of the training. Examples include the materials & equipment used, the instructor, the content, the room etc. Level 2 – accomplishment of the learning objectives are measured by a case, a test, an activity etc. Often the needs assessment can become a pretest and then given again as the evaluation of this level. Level 3 – often measured by observation and observer ratings of the extent the employee applies what was learned to their job. Level 4 – measurement of specific business results often collected before and after the training. Company records or reports are usually used.

10. Most CommonTypes of Costs in Training Development Costs: Needs assessment costs, program design, any pilot testing done, computer programming, program purchase fee Directs Costs: costs directly attributable to the delivery of the training. (If training were cancelled the day before it was to begin, the cost would not be incurred). Travel, facilities, food, beverages, equipment rental, trainer compensation Indirect Costs: non-developmental items that would be incurred even if training were cancelled. Copying of training materials, marketing expenses, administrative and clerical support, materials already sent to trainees before the actual training

11. Costs Continued Overhead Costs – The training program’s share of the general operating costs. For a large org. the program’s share of the purchase and maintenance of training equipment, clerical and administrative support, training facilities etc. For a small org. may not be any overhead costs if training if provided by an external source. Otherwise it’s often figured at 10% of direct, indirect, and development costs. Compensation for trainees – while attending training, their salaries, benefits should be included as a cost of training or the cost of replacing those employees (based on time away from the job). Org. have policies on this.

12. ROI ROI tells you the % return you have made over a specified period (usually a year) as a result of investing in a training program % ROI = Net Benefits(usually for a year)/Costs X 100 Or, ROI ratio = Net Benefits/Costs A ratio over 1 is the dollar return expected for each dollar invested 

13. Other Indicators Cost per trainee = Total cost of training divided by the number of trainees Payback Period = Costs / Monthly Benefits How many months it will take before the benefits of the training match the costs and the training pays for itself.

14. Example A particular company has an average of 90 grievances a year. 70 percent or 63 of them go to the 3 rd step before settlement. The average time that management must deal with a grievance is 10 hours. Management wages are $50/hr on average. That’s $500 for each grievance. Union representatives spend an average of 7.5 hrs at $25/hr or 187.50 per grievance. Mgmt cost + union cost = a total cost of $687.50 per grievance.

15. Example Con’t Total cost for 63 grievances is (687.50 x 63 = $43,312.50) prior to training Total training costs estimated to be $32,070. This includes Development, Direct, Indirect, Participant Compensation, etc. After training, in the first year only 8 grievances went to step 3 at a cost of (687.50 x 8 = $5,500) Training reduced the cost of grievances by $37,812.50 (Net Benefit)

16. Example con’t Net Benefit of $37,812.50 minus costs of $32,070 = $5,742.50 savings to the org. % ROI Divide the Net Benefit of $37,812.50 by the costs of $32,070 x 100 = 117.9% return. ROI ratio A return on investment ratio of 1.179. For every dollar spent, the training returns $1.18. A ratio of 1.0 means the training broke even. Lower than that it costs more than it benefits.

17. Evaluation Designs: Threats to Validity Threats to validity Threats to validity refer to a factor that will lead one to question either: (internal validity) The believability of the evaluation results, are they correct (internal validity), or (external validity) The extent to which the evaluation results are generalizable to other groups of trainees and situations or will the training be effective for other groups who go through it (external validity)

18. Methods to Control for Threats to Validity Pre- and Posttests Use of Comparison Groups Random Assignment Multiple Methods

19. Types of Evaluation Designs Posttest – only x T2 Pretest / posttest T1 x T2 Posttest – only with Comparison group x T2 T2 Pretest/posttest with comparison group T1 x T2 T1 T2 Times Series T1 T2 T3 x T5 T6 T7 T1 T2 T3 T5 T6 T7

20. Conditions for choosing a rigorous evaluation design Training is ongoing and has the potential to affect many employees and customers The training involves multiple classes and a large number of trainees The evaluation data will be used to change the program You have the expertise to design a rigorous evaluation and analyze the data Your interested in measuring the change from pre- training levels or comparing two or more different training programs Cost