Training Evaluation

Training evaluation provides the data needed to demonstrate that training does provide benefits to the company.

What are the differences among: Training effectiveness Training outcomes Training evaluation Evaluation design Training effectiveness refers to the benefits that the company and the trainees receive from training Training outcomes or criteria refer to measures that the trainer and the company use to evaluate training programs Training evaluation refers to the process of collecting the outcomes needed to determine if training is effective Evaluation design refers to from whom, what, when, and how information needed for determining the effectiveness of the training program will be collected

Types of Evaluation Formative Summative Formative evaluation – evaluation conducted to improve the training process Helps to ensure that: the training program is well organized and runs smoothly trainees learn and are satisfied with the program Provides information about how to make the program better Summative evaluation – evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program May also measure the return on investment (ROI) that the company receives from the training program

Why Evaluate Training Programs? Objectives Satisfaction Benefits Comparison Why Evaluate Training Programs Objectives to determine whether the program is meeting objectives, is enhancing learning, and is resulting in transfer of training to the job Satisfaction to determine whether trainees believe that the content and administration of the program were satisfactory Benefits to determine the financial benefits and costs of the program Comparison to compare the costs and benefits of different training programs to choose the best program

Why Should A Training Program Be Evaluated? (1 of 2) To identify the program’s strengths and weaknesses To assess whether content, organization, and administration of the program contribute to learning and the use of training content on the job To identify which trainees benefited most or least from the program

Why Should A Training Program Be Evaluated? (2 of 2) To gather data to assist in marketing training programs To determine the financial benefits and costs of the programs To compare the costs and benefits of training versus non-training investments To compare the costs and benefits of different training programs to choose the best program

Objectives = Foundation Terminal behavior Conditions under which terminal behavior is expected The standard below which performance is unacceptable --> criteria by which the trainee is judged

Training Program Objectives and Their Implications for Evaluation: Learning Transfer Outcomes Did company benefit through sales, quality, productivity, reduced accidents, and complaints? Performance on work equipment Results: Performance on a work sample Skill-Based: Trainees’ motivation or job attitudes Affective: Pencil-and-paper tests Cognitive: Ratings by peers or managers based on observation of behavior Did trainees like the program? Did the environment help learning? Was material meaningful? Reactions:

The Evaluation Process Conduct a Needs Analysis Develop Measurable Learning Outcomes and Analyze Transfer of Training Develop Outcome Measures Choose an Evaluation Strategy Plan and Execute the Evaluation

Training Outcomes: Kirkpatrick’s Four-Level Framework of Evaluation Criteria Business results achieved by trainees Results 4 Improvement of behavior on the job; aka skills Behavior 3 Acquisition of knowledge, skills, attitudes, behavior; aka cognitive Learning 2 Trainee satisfaction; aka affective Reactions 1 Focus Criteria Level

Training Outcomes Cognitive (Knowledge) Skills (Behaviors) Affect (Attitudes & Motivation) Reactions Results reaction outcomes relate to what trainees thought of the program, including the facilities, trainers, & content >survey of participants asking what they thought about the program learning outcomes are used to determine the extent to which trainees are familiar with principles, facts, techniques, skills, or processes presented in the training program >administration of an exam over training materials behavior outcomes are used to determine the extent to which the trainees’ behavior changes on the job as a result of participation in the program >recording behavior on the job after training and comparing it to behavior on the job before training results outcomes are used to determine the payoff the training program had for the company >measuring productivity after training to see if it increased

Which Outcome is Represented by the Evaluation Form for Your Training Module?

Outcomes Used in Evaluating Training Programs: (1 of 4) Affective Outcomes Results Return on Investment Cognitive Skill-Based The following suggestions specify the types of changes in jobs that are most likely to lead to improvements in each of the five core dimensions. (1) Combine tasks - managers should put existing fractionalized tasks back together to form a new, larger module of work. This increases skill variety and task identify. (2) Create natural work units - managers should design tasks that form an identifiable and meaningful whole. This increases employee “ownership” of the work and encourages employees to view their work as meaningful and important rather than as irrelevant and boring. (3) Establish client relationships - the client is the user of the product or service that the employee works on. Whenever possible, managers should establish direct relationships between workers and their clients. This increases skill variety, autonomy, and feedback for the employee. (4) Expand jobs vertically - vertical expansion means giving employees responsibilities and controls that were formerly reserved for management. It partially closes the gap between the “doing” and “controlling” aspects of the job, and it increases employee autonomy. (5) Open feedback channels - by increasing feedback, employees not only learn how well they are performing their jobs but also whether their performances are improving, deteriorating, or remaining at a constant level. Ideally, employees should receive performance feedback directly as they do their jobs rather than from management on an occasional basis. 9

Outcomes Used in Evaluating Training Programs: (2 of 4) Cognitive Outcomes Determine the degree to which trainees are familiar with the principles, facts, techniques, procedures, or processes emphasized in the training program Measure what knowledge trainees learned in the program Skill-Based Outcomes Assess the level of technical or motor skills Include acquisition or learning of skills and use of skills on the job

Outcomes Used in Evaluating Training Programs: (3 of 4) Affective Outcomes Include attitudes and motivation Trainees’ perceptions of the program including the facilities, trainers, and content Results Determine the training program’s payoff for the company

Outcomes Used in Evaluating Training Programs: (4 of 4) Return on Investment (ROI) Comparing the training’s monetary benefits with the cost of the training direct costs indirect costs benefits

How do you know if your outcomes are good? Good training outcomes need to be: Relevant Reliable Discriminative Practical

Good Outcomes: Relevance Criteria relevance – the extent to which training programs are related to learned capabilities emphasized in the training program Criterion contamination – extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions Criterion deficiency – failure to measure training outcomes that were emphasized in the training objectives

Criterion deficiency, relevance, and contamination: Outcomes Identified by Needs Assessment and Included in Training Objectives Outcomes Related to Training Objectives Outcomes Measured in Evaluation Contamination Relevance Deficiency

Good Outcomes (continued) Reliability – degree to which outcomes can be measured consistently over time Discrimination – degree to which trainee’s performances on the outcome actually reflect true differences in performance Practicality – refers to the ease with which the outcomes measures can be collected

Training Evaluation Practices Percentage of Courses Using Outcome Outcomes

How do Fidelity & Motorola evaluate their training programs? How do their measures of success compare with those advocated by the text?

Evaluation Designs: Threats to Validity Threats to validity refer to a factor that will lead one to question either: The believability of the study results (internal validity), or The extent to which the evaluation results are generalizable to other groups of trainees and situations (external validity)

Threats to Validity Threats To External Validity Company Threats To Internal Validity Company Persons Outcome Measures Threats To External Validity Reaction to pretest Reaction to evaluation Interaction of selection and training Interaction of methods

Methods to Control for Threats to Validity Pre- and Posttests Use of Comparison Groups Random Assignment

Evaluation Procedures

Utility

[(Ns)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)] Ns = number of applicants selected T = tenure of selected group in years r = correlation between predictor and job performance (VALIDITY) SDy = standard deviation of job performance Zs = average standard predictor score of selected group N = number of applicants C = cost per applicant

[(Nc)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)] Nc = number of trainees who complete program T = duration of training benefit r = correlation between training criterion and job performance (VALIDITY) SDy = standard deviation of job performance Zs = average standard criterion score of trainees N = total number of trainees enrolled C = cost per trainee

Training Costs Direct Indirect Development Overhead Compensation for Trainees Direct instructor (daily rate x # days, fringe benefits) travel expenses, materials x # trainees, classroom space & av equipment, refreshments (X #days X # trainees) Indirect mgmt, clerical & admin salaries + fringes; postage, shipping, telephone, pre & post learning materials (x # trainees) Development fee for program purchase, instructor training, registration fee, travel & lodging, salary, benefits Overhead general org support, top mgmt time (10% of d, I & d) Compensation for Trainees trainees salaries & benefits (based on time away from the job)

For On the Job Training $81,000 50 = Ns = number of trainees who complete program 1 = T = duration of training benefit .50 = r = correlation between training criterion and job performance (VALIDITY) 4800 = SDy = standard deviation of job performance (assume 40% of base pay . . . $12,000 * .40) .80 = Zs = average standard criterion score of trainees 100 = N = total number of trainees enrolled 150 = C = cost per trainee [(Ns)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)] (50 * 1 * .50 * 4800 * .8) - (100 * 150)

Experimental Designs

Experimental Designs Choices Pretest/posttest Control Groups

Experimental Designs 1: 1 group, posttest only 2: 1 group, pretest/posttest 3: Pretest/posttest control group 4: Solomon four-group 5: Time-series 6: Nonequivalent control group

Experimental Designs Validity Internal External

Experimental Designs Threats to Internal Validity History Maturation Testing Instrumentation Regression toward the mean Differential selection Experimental mortality Interactions Diffusion/imitation of treatments Compensatory equalization of treatments Rivalry/desirability of treatments Demoralization

Experimental Designs Threats to External Validity Reactive effect of pretesting Interaction of selection & treatment Reactive effects of experimental settings Multiple-treatment interference

Issues in Training Validity Transfer validity Intra-organizational validity Inter-organizational validity