Effective Training: Strategies, Systems and Practices, 3 rd Edition P. Nick Blanchard and James W. Thacker Chapter 9 Training Evaluation

Evaluation Phase Chapter 82 Input Process Output Evaluation Objectives Design Issues Organizational Constraints Evaluation Strategy and Design (includes process evaluation and outcome evaluation) Process Measures Outcome Measures - Reaction - Learning - Behavior - Results

Purpose of Training Evaluation Determine if training accomplished objectives Identify strengths and weaknesses of training program Cost-benefit analysis (i.e. is the training program feasible) Who should participate and who benefited most (to find out focus of the training program) Database for future decision making about training 3

Training Evaluation Here, the training program is evaluated to see whether it has actually helped the organization achieve its improvement objectives. Two kinds of evaluation:- 1. Process Evaluation- evaluation of the training process itself, i.e. whether the training followed the curriculum originally set out. Ex- whether a teacher followed the course outline throughout the course

Training Evaluation 2. Outcome Evaluation Whether the training resulted in any real benefits or improvements for the firm. Here, the results of training are compared with the original objectives of the training program to see whether there are any improvements in performance. If yes, the training program is a success. If no, then it is not worth it. Also, organizational constraints affect training. Even if a training program is successful, it might not be feasible if it involves a lot of time, is very expensive, or requires extreme involvement from the employees. All these might not be possible in a tight, competitive business environment.

Who Is Interested in Process Evaluation Data 6 Training Department (YES, because they need to know if the training program was followed and implemented properly, i.e. all modules and topics covered, chance for practice given, trainer explained all problems and confusion etc. ) Basically, whether the training program was followed properly Trainer:Yes, to determine what does and does not work well. Other trainers: To the extent the process generalizes to their training, and therefore, is useful to them Training manager: Only if training is not successful, or if there is a problem with a particular trainer, i.e. for troubleshooting purposes Customers of the Training program (i.e. anyone interested in the output of the training program)- NO, because they are more concerned with outcome data (i.e. whether training resulted in any real benefits or improvements for them). They are usually not concerned whether the training followed due process correctly. Trainees:No Trainees’ supervisor:No Upper management:No

Outcome Evaluation Data This is basically analyzing the results of the 4 types of training objectives mentioned earlier Reaction Outcomes – How trainees felt about the training program, i.e. did trainees like the program or dislike it, did they find it boring or interesting, did they have positive or negative attitude throughout the training etc. Low reaction scores mean training was boring, high reaction scores mean training was engaging and interesting Learning Outcomes – How well the trainees have understood and learnt the training content, the topics, the subject matter etc.

Outcome Evaluation Data Job Behavior (Transfer) Outcomes – How well the learned behaviours have actually transferred to the job Organizational Results Outcomes – How much effect training has had on organizational results, i.e. has training improved inefficiency, low productivity, poor performance etc. (these are the reasons the training was developed in the first place). Basically, the effect of training on bottom line organizational performance is analyzed

Example of Outcome Evaluation 1. Reaction Outcomes: 2. Learning Outcomes: Training: SPSS Software Training Measuring Tool: Reaction Questionnaire Reaction Outcome: Trainees feel interested, engaged and motivated….they have a positive attitude throughout and are not bored in the class, so reaction outcome is positive. Move on to evaluating learning objective Training: SPSS Software Training Measuring Tool: Computer Data Analyzing Test Learning Outcome: Trainees pass computer- based data analyzing test using SPSS software which means they know how to use the software properly. E mployee can use SPSS software to perform correlation, regression and data mapping functions (learning outcome successful)

Example of Outcome Evaluation 3. Job Behaviour (Transfer) Outcomes 4. Organizational Results Outcomes Training: SPSS Software Training Measuring Tool: Observation; SPSS Data Report Submission, SPSS Analysis Report Submission Transfer Outcome: Trainees are properly using the skills learned in training in the workplace. They are using SPSS software to correctly predict the relationship between marketing expenditure and sales, and organizational training and productivity. Therefore, transfer of training is successful (Job Behaviour (transfer) outcome positive) Training: SPSS Software Training Measuring Tool: Profitability Analysis, Productivity Ratios etc Results Outcome: An analysis of results show that as a result of correct forecasting with SPSS software, profitability for the third quarter has increased by 4%. Thus, SPSS software training has resulted in actual benefits for the company (Organizational results Outcome Positive)

Method of Collecting Data for Reaction Outcomes The method of collecting data for reaction outcomes is by using a reaction questionnaire A reaction questionnaire is used to assess how trainees feel toward the training program, i.e. their attitudes, level of interest and motivation, level of engagement (or boredom) etc. A reaction questionnaire gets trainees’ opinions about: – Training Relevance i.e. do trainees see the training as important, and useful for their career. It is important that they do, because if they don’t, it can seriously hamper the effectiveness of the training program (even a well-designed one) – Training Materials and Exercises How trainees feel about the materials and exercises used in the training. Any suggestions for improvements that they might have

Method of Collecting Data for Reaction Outcomes – Reactions to the Trainer How well the trainees liked (or disliked) the trainer. Questions should be specific and address key components of the trainer’s behaviour (avoid general questions). Such as “ did the trainer give proper, detailed announcements before taking exams? “ and not “was the trainer enjoyable ?” – Facilities and Procedures How well the trainees liked (or disliked) the facility, room, environment, refreshments served, and other regulations of the training program (no of coffee breaks etc)

Method of Collecting Data for Reaction Outcomes Timing of Reaction Questionnaire – Immediately after training Advantage: – Training memory is fresh in trainees’ mind so they can give more accurate responses – Audience is captive (they have to submit responses) Weakness: – Trainees may be anxious to leave, so responses will be hurried and not accurate – Trainees don’t know transfer of training outcome yet, so their feelings are not complete – Some time after training (a couple of days or months) Advantage: – Trainees are aware of transfer outcome (i.e. whether training is useful to the job) so they can give more informed responses Weakness: – Trainees might forget about training specifics or cannot remember all the details properly – They are not captive, so response is not guaranteed

Method of Collecting Data for Reaction Outcomes – During Training (after separate segments or parts of training) Advantage: – Real time responses obtained, so training can be modified accordingly – Audience is still captive, so responses guaranteed Weakness: – Requires quicker analysis and implementation cycle of data – More expensive and time consuming to implement (separate questionnaires for separate days)

Reaction Questionnaire for the Trainer – Part 1 of 2 Chapter 815 Please circle the number that reflects the degree to which you agree or disagree with the following statements. 1= Strongly disagree, 2= Disagree, 3= Neither agree nor disagree 4= Agree 5 =Strongly agree 1. The trainer did a good job of stating the objectives at the beginning of training ……….………..………………… The trainer made good use of visual aids (easel, white board) when making the presentations ……………....……… The trainer was good at keeping everyone interested in the topics ……………………………………………..…… The trainer encouraged questions and participation from trainees …………………………….………………

Reaction Questionnaire for the Trainer – Part 2 of 2 Chapter The trainer made sure everyone understood the concepts before moving on to the next topic………………………… The trainer summarized important concepts before moving to the next module…………………………………… Overall, how would you rate this trainer (check one) ____1. Poor; I would not recommend this trainer to others. ____2. Adequate; I would recommend this trainer only if no others were available. ____3. Average ____4. Good; I would recommend this trainer above most others. ____5. Excellent; this trainer is among the very best I’ve ever worked with.

Steps to Consider in Developing a Reaction Questionnaire – Part 1 of 2 Chapter Determine what you want to find out (consider training objectives). 2. Develop a written set of questions to obtain the information. 3. Develop a scale to quantify respondents’ data. 4. Make forms anonymous so participants will feel free to respond honestly. 5. Ask for information useful in determining differences in reactions by subgroups (e.g., young vs. old; minority vs. non-minority). This could be valuable in determining effectiveness of training for different cultures, for example, which may be lost in an overall assessment.

Steps to Consider in Developing a Reaction Questionnaire – Part 2 of 2 Chapter Allow space for “Additional Comments” in order to allow participants the opportunity to mention things you might not have considered. a. If right after training, have someone other than the instructor administer and pick up the information. b. If some time later, develop a mechanism for obtaining a high response rate (e.g., encourage the supervisor to allow trainees to complete the questionnaire on company time).

Method of Collecting Data for Learning Outcomes Learning outcomes can be of three types: Knowledge, Skills and Attitudes (KSA) Therefore, they can be tested by: – Knowledge Tests Written Tests (usually in MCQ format) Advantage: Easy to administer, easy to score, can test detailed knowledge if designed properly Weakness: Difficult to construct MCQ questions (requires some skill)

Method of Collecting Data for Learning Outcomes – Skill Tests Different types of skills require different types of tests. There is no one way to test For example, for SPSS software training…….a computer data analyzing test; for communication skill training……a presentation test – Attitude Tests Trainee attitudes can be tested by using a attitudinal scale (ex- a questionnaire which asks and ranks trainee responses regarding attitude toward the training objective) This is different from a reaction questionnaire (a reaction questionnaire identified trainee attitude towards the training program, but an attitude test identifies trainee attitude towards the training objective- whatever that might be Ex- if the purpose of the training was to create positive attitudes towards employees of different racial backgrounds (i.e. diversity awareness training), only an attitude test can measure if it was done (not skill or knowledge)

Example of an Attitudinal Measure Chapter 821 Attitudes Toward Empowerment Please indicate the degree to which you agree or disagree with the following statements. 1 = Strongly disagree 2 = Disagree 3 = Neither agree nor disagree 4 = Agree 5 = Strongly agree 1. Empowering employees is just another way to get more work done with fewer people. (reverse scored)………… Empowering of employees allows everyone to contribute their ideas to the betterment of the company… The empowerment program has improved relationship with my supervisor……………………………………………… Empowerment has brought more meaning to my life at this company…………………………………………………… Empowerment interventions should be introduced in other plants in this company………………………………………… The empowerment process has been a positive influence in labor-management relations……………………………………

Procedures for Developing a Multiple Choice Test Chapter Examine objectives to determine the content area you wish to test. 2. Write the questions in a clear manner. Shorter is better. 3.Choose alternatives to the correct response from typical errors made during training. Make alternatives realistic. 4.Avoid making the correct response always longer than incorrect responses. 5.Four alternatives are usually enough. 6.Pretest items. Ask for feedback on clarity. Note questions that most get wrong.

Paper & Pencil Test for Evaluation of Learning Chapter 823 Evaluation of learning There is no specific time limit on this test, but most should be able to finish in about one hour. Answers to the questions should be written in the booklet provided. Please read each question carefully as some of the questions have more than one part to them. 1.List four types of active listening and provide an example for each. 2.List the steps in the Conflict resolution model. After each step, provide a relevant example of a phrase that would represent that step. 3.And so forth…..

Method of Collecting Data for Job Behaviour (Transfer) Outcomes Data about training transfer outcomes can be obtained by using: – Interviews – Questionnaires Most often used Advantages: – Large sample can be surveyed – Quick responses – Anonymity can be provided to trainees (as an option) so better chance of getting honest answers (compared to say interview) A variation of the questionnaire is the scripted situation questionnaire which basically presents a situation and gives a no. of choices that the trainee might take. The rater is then asked to make a choice based on his knowledge about the trainee – Observation – 360 Degree Performance Review Everyone around the trainee evaluates him and his job performance (transfer of training) including his boss, colleagues, subordinates, even customers etc If done pre-training and post-training, can enable us to see the transfer of training (if there is any)

Guidelines for Writing Effective Questionnaires – Part 1 of 4 Chapter Ask one question at a time. Bad: Both the organization’s goals and my role within the organization are clear. 1. Write simply and clearly, and make the meaning obvious. Bad: To what extent do supervisors provide information regarding the quality of performance of people at your level? Good: How often does your boss give you feedback on your job? Good:The organization’s goals are clear. My role within the organization is clear.

Guidelines for Writing Effective Questionnaires – Part 2 of 4 Chapter 826 Good: During the past three months how often did you receive feedback on your work? not once 1–3 times about once 2-4 times once a a month a week a week day or more 3. Provide discrete response options. Bad: During the past three months how often did you receive feedback on your work? rarely occasionally frequently

Guidelines for Writing Effective Questionnaires – Part 3 of 4 Chapter Limit the number of response options. Bad: What percent of the time are you sure of what your compensation will be? –10% 11–20% 21–30% 31–40% 41–50% 51–60% 61–70% 71–80% 81–90% 91–100% Good: What percent of the time are you sure of what your compensation will be? –20% 21–40% 41–60% 61–80% 81–100%

Guidelines for Writing Effective Questionnaires – Part 4 of 4 Chapter Match the response mode to the question. Bad: To what extent are you satisfied with your job? strongly disagree agree strongly disagree agree Good: To what extent are you satisfied with your job? not at all a little bit some quite a lot very much

Example of 360-Degree Feedback Results Chapter 829 Skills being Assessed Very Low LowAverageHighVery High Listening SkillsSP, BE Managing ConflictSP, BE OrganizingS, PB, E Written CommunicationS, PBE S = Subordinates P = Peers B = Supervisor (Boss) E = Employee being assessed

Scripted Situation Item for Evaluation of a School Superintendent Chapter 830 The following is a scenario about a school superintendent. Read the scenario and place an X next to the behavior you believe your superintendent would follow. The administrator receives a letter from a parent objecting to the content of the science section on reproduction. The parent strongly objects to his daughter having exposure to such materials and demands something be done. The effective administrator would most likely: (check one) ____ Ask the teacher to provide handouts, materials, and curriculum content for review. ____ Check the science curriculum for the board-approved approach to reproduction, and compare board guidelines to course content. ____ Ask the head of the science department for his or her opinion about the teacher’s lesson plan. ____ Check to see if the parent has made similar complaints in the past.

Method of Collecting Data for Organizational Results Outcomes Organizational Results outcomes can be obtained from different sources depending on the type of training Ex- Customer service training can be evaluated by looking at……….level of customer satisfaction and loyalty toward the firm; factory worker training can be evaluated by looking at………..quality standards (no. of defects produced/day) etc It is important to connect the links between organizational results and training, because an increase in organizational performance can happen for some reason other than training Ex- customer satisfaction can be positive for some reason other than training, such as good promotional efforts (discounts, gifts etc) or a low price compared to competitors (manufacturing advantage), and not as a result of good customer service training

Types of Outcomes and Examples of Factors Affecting those Outcomes Chapter 832 Perceived match between trainee Exceptions and what training provided Reactions Trainee readiness for the course Trainee motivation to learn Design, materials, and content Trainer(s) behaviors Learning Transfer of training Motivational forces in the job setting Opportunity to applying training on the job KSAs Job Behavior External environment of the organization: Economy, regulations, suppliers, etc Internal environment of the organization: Policies, procedures, systems Employee performance, KSAs, and needs Organizational Results

Training Cost Evaluation Sometimes it doesn’t matter how good or successful a training program is in terms of improving learning or KSA’s, but it has to be feasible financially Two types of cost evaluation: – Cost-Benefit Evaluation- Cost-benefit evaluation compares the monetary cost of training with the non- monetary benefits obtained from training such as positive attitude and better working relationships. These are non-financial gains obtained from training and are difficult to assess numerically Ex- Conflict management training given to managers results in peace and happiness within the department (no arguments or tension) – Cost-Effectiveness Evaluation Cost-effectiveness evaluation compares the monetary cost of training with the financial benefits obtained from the training such as increased profits etc Two types of cost-effectiveness evaluation: Cost-Savings Analysis Utility Analysis

Training Cost Evaluation Cost-Savings Analysis Cost-savings analysis looks at how the training will reduce costs for the firm by fixing or correcting the problem only Ex- As a result of conflict management training, there will be less work stoppages/day, so more working hours/day. Each hour contributes $250 to revenue. So, a fixed amount will be saved Utility Analysis Utility analysis looks at all the ways the training will benefit the firm financially, not just by correcting the problem only Ex- As a result of conflict management training, not only will there be less work stoppages/day but since employees are also not arguing with each other (there is a friendly relationship), they will be more motivated than normal and produce at a higher rate. Also there will be less absenteeism. All these have financial benefits for the firm

Who Is Interested in the Outcome Data Chapter 835 Outcome Data ReactionLearningBehaviorResults Training Department TrainerYes No Other TrainersPerhaps No Training ManagerYes Customers TraineesYes Perhaps Trainees’ SupervisorNot reallyOnly if no transfer Yes Upper ManagementNo PerhapsYes