1. Training Evaluation

2. Training evaluation provides the data needed to demonstrate that training does provide benefits to the company.

3. What are the differences among:
Training effectiveness
Training outcomes
Training evaluation
Evaluation design
Training effectiveness refers to the benefits that the company and the trainees receive from training
Training outcomes or criteria refer to measures that the trainer and the company use to evaluate training programs
Training evaluation refers to the process of collecting the outcomes needed to determine if training is effective
Evaluation design refers to from whom, what, when, and how information needed for determining the effectiveness of the training program will be collected

4. Types of Evaluation Formative Summative
Formative evaluation – evaluation conducted to improve the training process
Helps to ensure that:
the training program is well organized and runs smoothly
trainees learn and are satisfied with the program
Provides information about how to make the program better
Summative evaluation – evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program
May also measure the return on investment (ROI) that the company receives from the training program

5. Why Evaluate Training Programs?
Objectives
Satisfaction
Benefits
Comparison
Why Evaluate Training Programs
Objectives
to determine whether the program is meeting objectives, is enhancing learning, and is resulting in transfer of training to the job
Satisfaction
to determine whether trainees believe that the content and administration of the program were satisfactory
Benefits
to determine the financial benefits and costs of the program
Comparison
to compare the costs and benefits of different training programs to choose the best program

6. Why Should A Training Program Be Evaluated? (1 of 2)
To identify the program’s strengths and weaknesses
To assess whether content, organization, and administration of the program contribute to learning and the use of training content on the job
To identify which trainees benefited most or least from the program

7. Why Should A Training Program Be Evaluated? (2 of 2)
To gather data to assist in marketing training programs
To determine the financial benefits and costs of the programs
To compare the costs and benefits of training versus non-training investments
To compare the costs and benefits of different training programs to choose the best program

8. Objectives = Foundation
Terminal behavior
Conditions under which terminal behavior is expected
The standard below which performance is unacceptable
--> criteria by which the trainee is judged

9. Training Program Objectives and Their Implications for Evaluation:
Learning
Transfer
Outcomes
Did company benefit through sales, quality, productivity, reduced accidents, and complaints?
Performance on work equipment
Results:
Performance on a work sample
Skill-Based:
Trainees’ motivation or job attitudes
Affective:
Pencil-and-paper tests
Cognitive:
Ratings by peers or managers based on observation of behavior
Did trainees like the program?
Did the environment help learning?
Was material meaningful?
Reactions:

10. The Evaluation Process
Conduct a Needs Analysis
Develop Measurable Learning Outcomes and Analyze Transfer of Training
Develop Outcome Measures
Choose an Evaluation Strategy
Plan and Execute the Evaluation

11. Training Outcomes: Kirkpatrick’s Four-Level Framework of Evaluation Criteria
Business results achieved by trainees
Results 4
Improvement of behavior on the job; aka skills
Behavior 3
Acquisition of knowledge, skills, attitudes, behavior; aka cognitive
Learning 2
Trainee satisfaction; aka affective
Reactions 1
Focus
Criteria
Level

12. Training Outcomes Cognitive (Knowledge) Skills (Behaviors)
Affect (Attitudes & Motivation)
Reactions
Results
reaction outcomes relate to what trainees thought of the program, including the facilities, trainers, & content
>survey of participants asking what they thought about the program
learning outcomes are used to determine the extent to which trainees are familiar with principles, facts, techniques, skills, or processes presented in the training program
>administration of an exam over training materials
behavior outcomes are used to determine the extent to which the trainees’ behavior changes on the job as a result of participation in the program
>recording behavior on the job after training and comparing it to behavior on the job before training
results outcomes are used to determine the payoff the training program had for the company
>measuring productivity after training to see if it increased

13. Which Outcome is Represented by the Evaluation Form for Your Training Module?

14. Outcomes Used in Evaluating Training Programs: (1 of 4)
Affective
Outcomes
Results
Return on
Investment
Cognitive
Skill-Based
The following suggestions specify the types of changes in jobs that are most likely to lead to improvements in each of the five core dimensions.
(1) Combine tasks - managers should put existing fractionalized tasks back together to form a new, larger module of work. This increases skill variety and task identify.
(2) Create natural work units - managers should design tasks that form an identifiable and meaningful whole. This increases employee “ownership” of the work and encourages employees to view their work as meaningful and important rather than as irrelevant and boring.
(3) Establish client relationships - the client is the user of the product or service that the employee works on. Whenever possible, managers should establish direct relationships between workers and their clients. This increases skill variety, autonomy, and feedback for the employee.
(4) Expand jobs vertically - vertical expansion means giving employees responsibilities and controls that were formerly reserved for management. It partially closes the gap between the “doing” and “controlling” aspects of the job, and it increases employee autonomy.
(5) Open feedback channels - by increasing feedback, employees not only learn how well they are performing their jobs but also whether their performances are improving, deteriorating, or remaining at a constant level. Ideally, employees should receive performance feedback directly as they do their jobs rather than from management on an occasional basis. 9

15. Outcomes Used in Evaluating Training Programs: (2 of 4)
Cognitive Outcomes
Determine the degree to which trainees are familiar with the principles, facts, techniques, procedures, or processes emphasized in the training program
Measure what knowledge trainees learned in the program
Skill-Based Outcomes
Assess the level of technical or motor skills
Include acquisition or learning of skills and use of skills on the job

16. Outcomes Used in Evaluating Training Programs: (3 of 4)
Affective Outcomes
Include attitudes and motivation
Trainees’ perceptions of the program including the facilities, trainers, and content
Results
Determine the training program’s payoff for the company

17. Outcomes Used in Evaluating Training Programs: (4 of 4)
Return on Investment (ROI)
Comparing the training’s monetary benefits with the cost of the training
direct costs
indirect costs
benefits

18. How do you know if your outcomes are good?
Good training outcomes need to be:
Relevant
Reliable
Discriminative
Practical

19. Good Outcomes: Relevance
Criteria relevance – the extent to which training programs are related to learned capabilities emphasized in the training program
Criterion contamination – extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions
Criterion deficiency – failure to measure training outcomes that were emphasized in the training objectives

20. Criterion deficiency, relevance, and contamination:
Outcomes Identified by Needs Assessment and Included in Training Objectives
Outcomes Related to Training Objectives
Outcomes Measured in Evaluation
Contamination
Relevance
Deficiency

21. Good Outcomes (continued)
Reliability – degree to which outcomes can be measured consistently over time
Discrimination – degree to which trainee’s performances on the outcome actually reflect true differences in performance
Practicality – refers to the ease with which the outcomes measures can be collected

22. Training Evaluation Practices
Percentage of Courses Using Outcome
Outcomes

23. How do Fidelity & Motorola evaluate their training programs?
How do their measures of success compare with those advocated by the text?

24. Evaluation Designs: Threats to Validity
Threats to validity refer to a factor that will lead one to question either:
The believability of the study results (internal validity), or
The extent to which the evaluation results are generalizable to other groups of trainees and situations (external validity)

25. Threats to Validity Threats To External Validity Company
Threats To Internal Validity
Company
Persons
Outcome Measures
Threats To External Validity
Reaction to pretest
Reaction to evaluation
Interaction of selection and training
Interaction of methods

26. Methods to Control for Threats to Validity
Pre- and Posttests
Use of Comparison Groups
Random Assignment

27. Evaluation Procedures

28. Utility

29. [(Ns)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)]
Ns = number of applicants selected
T = tenure of selected group in years
r = correlation between predictor and job performance (VALIDITY)
SDy = standard deviation of job performance
Zs = average standard predictor score of selected group
N = number of applicants
C = cost per applicant

30. [(Nc)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)]
Nc = number of trainees who complete program
T = duration of training benefit
r = correlation between training criterion and job performance (VALIDITY)
SDy = standard deviation of job performance
Zs = average standard criterion score of trainees
N = total number of trainees enrolled
C = cost per trainee

31. Training Costs Direct Indirect Development Overhead
Compensation for Trainees
Direct
instructor (daily rate x # days, fringe benefits) travel expenses,
materials x # trainees, classroom space & av equipment,
refreshments (X #days X # trainees)
Indirect
mgmt, clerical & admin salaries + fringes; postage, shipping, telephone,
pre & post learning materials (x # trainees)
Development
fee for program purchase, instructor training, registration fee,
travel & lodging, salary, benefits
Overhead
general org support, top mgmt time (10% of d, I & d)
Compensation for Trainees
trainees salaries & benefits (based on time away from the job)

32. For On the Job Training $81,000
50 = Ns = number of trainees who complete program
1 = T = duration of training benefit
.50 = r = correlation between training criterion and job performance (VALIDITY)
4800 = SDy = standard deviation of job performance (assume 40% of base pay $12,000 * .40)
.80 = Zs = average standard criterion score of trainees
100 = N = total number of trainees enrolled
150 = C = cost per trainee
[(Ns)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)]
(50 * 1 * .50 * 4800 * .8) - (100 * 150)

33. Experimental Designs

34. Experimental Designs Choices
Pretest/posttest
Control Groups

35. Experimental Designs 1: 1 group, posttest only
2: 1 group, pretest/posttest
3: Pretest/posttest control group
4: Solomon four-group
5: Time-series
6: Nonequivalent control group

36. Experimental Designs Validity
Internal
External

37. Experimental Designs Threats to Internal Validity
History
Maturation
Testing
Instrumentation
Regression toward the mean
Differential selection
Experimental mortality
Interactions
Diffusion/imitation of treatments
Compensatory equalization of treatments
Rivalry/desirability of treatments
Demoralization

38. Experimental Designs Threats to External Validity
Reactive effect of pretesting
Interaction of selection & treatment
Reactive effects of experimental settings
Multiple-treatment interference

39. Issues in Training Validity
Transfer validity
Intra-organizational validity
Inter-organizational validity