1. Effective Training: Strategies, Systems and Practices, 2 nd Edition Chapter Eight Evaluation of Training

2. Chapter 82 Session Overview Define evaluation Describe why many organizations don’t evaluate training Explain who is interested in evaluation data Describe the various levels of training evaluation

3. Chapter 83 Evaluation Defined Evaluation: judgment of training programs worth or value Formative Evaluation (process) examines how the training was designed, developed, and carried out Summative Evaluation (outcome) determines if training accomplished its objectives

4. Chapter 84 Resistance to Training Evaluation A few reasons why training managers don’t evaluate training: Nothing to evaluate No one really cares Evaluation is a threat to my job

5. Chapter 85 Types of Evaluation Data The following are types of evaluation Data that should be collected: Reaction Questionnaire Learning Job Behavior Organizational Results

6. Chapter 86 Who Is Interested Outcome Data ReactionLearningBehaviorResults Training Department TrainerYes No Other TrainersPerhaps No Training ManagerYes Customers TraineesYes Perhaps Trainees’ SupervisorNot reallyOnly if no transfer Yes Upper ManagementNo PerhapsYes

7. Chapter 87 Kirkpatrick Level 1: Reaction Questionnaire Reaction Questionnaires are: “Used to determine what the trainees thought about the training.” (p. 353) Affective Questionnaire: - measures general feelings about training Utility Questionnaire: - measures the trainee's belief about the value of the training

8. Chapter 88 Reaction Questionnaire: Types of Questions The following is obtained from a reaction questionnaire: Facilities issues Trainer issues Perceived value of training Relevance of training (example: pp. 355-356)

9. Chapter 89 Timing of Reaction Questionnaire The following times can be used to administer a reaction questionnaire? During training (after each segment) Immediately following training One or two months after training

10. Chapter 810 Steps in Developing a Reaction Questionnaire (1 of 2) 1. Determine what you want to find out (consider training objectives). 2. Develop a written set of questions to obtain the information. 3. Develop a scale to quantify respondents’ data. 4. Make forms anonymous so participants will feel free to respond honestly. 5. Ask for information useful in determining differences in reactions by subgroups (e.g., young vs. old; minority vs. non-minority). This could be valuable in determining effectiveness of training for different cultures, for example, which may be lost in an overall assessment.

11. Chapter 811 Steps in Developing a Reaction Questionnaire (2 of 2) 6. Allow space for “Additional Comments” in order to allow participants the opportunity to mention things you might not have considered. 7. Decide the best time to give the questionnaire to get the information you want. a. If right after training, have someone other than the instructor administer and pick up the information. b. If some time later, develop a mechanism for obtaining a high response rate (e.g., encourage the supervisor to allow trainees to complete the questionnaire on company time).

12. Chapter 812 Kirkpatrick Level 2: Learning Assessing Learning: Declarative knowledge – paper/pencil multiple choice test Procedural knowledge – paired comparison method (p. 359; 8-4) 1. Determining relationships among a number of concepts Strategic knowledge – Probed Protocol Analysis 1. Define a problem 2. Explain step-by-step how to solve problem

13. Chapter 813 Kirkpatrick Level 3: Job Behavior Job Behavior: “Measuring if training actually transferred to the job”

14. Chapter 814 Kirkpatrick Level 3: Job Behavior Data collection approaches: Behavior data Interviews Direct observation Archival records of performance Scripted situation - a process for assessing transfer of training to the job by providing rater with a number of responses to rate trainee (p. 363) Attitudes – attitudinal surveys

15. Chapter 815 Kirkpatrick Level 4: Organizational Results Organizational Results: “This measurement would be used to determine any change after training was complete” (p. 364) Did grievances go down? Did scrap rates diminish?

16. Chapter 816 Kirkpatrick Level 4: Organizational Results Data collection: Consistent tracking of organizational performance should take place at intervals before, throughout, & after training. Scrap rate after training – scrap rate before training = results

17. Chapter 817 Evaluation: Kirkpatrick Perceived match between trainee Exceptions and what training provided Reactions Trainee readiness for the course Trainee motivation to learn Design, materials, and content Trainer(s) behaviors Learning Transfer of training Motivational forces in the job setting Opportunity to applying training on the job KSAs Job Behavior External environment of the organization: Economy, regulations, suppliers, etc Internal environment of the organization: Policies, procedures, systems Employee performance, KSAs, and needs Organizational Results