Training Evaluation

Evaluation of Training Since huge sums of money are spent on training and development, how far the programme has been useful must be determined. The question of what to evaluate is crucial to evaluation strategy. The answers depend on the type of training programme, the organization and the purposes of evaluation.

Objectives of evaluation 1.To determine success in accomplishing Programme Objectives 2.To identify the strengths and weaknesses in the training process 3.To compare the costs to the benefits of a Training Programme 4.To decide who should participate in future programmes 5.To test the clarity and validity of tests, cases and exercises

6. To identify which participants were the most successful with the programme 7. To reinforce major points made to the participant 8. To gather data to assist in marketing future programmes 9. To determine if the programme was the appropriate solution for the specific need 10. To establish a database that can assist management in making decisions

Criteria of Evaluation 1.Training validity- did the trainees learn during training? 2.Transfer validity- what has been learnt in the training, has it been transferred on the job? 3.Intra-organizational validity- is performance of the new group of trainees, consistent with the performance of the original training group? 4.Inter- organizational validity- can a training program validated in one organization can be used successfully in another organization?

Feedback: It helps in giving feedback to the candidates by defining the objectives and linking it to learning outcomes. Research: It helps in ascertaining the relationship between acquired knowledge, transfer of knowledge at the work place, and training.

Control: It helps in controlling the training program because if the training is not effective, then it can be dealt with accordingly. Power games: At times, the top management (higher authoritative employee) uses the evaluative data to manipulate it for their own benefits. Intervention: It helps in determining that whether the actual outcomes are aligned with the expected outcomes

Before Training: The learner's skills and knowledge are assessed before the training program. During the start of training, candidates generally perceive it as a waste of resources because at most of the times candidates are unaware of the objectives and learning outcomes of the program. Once aware, they are asked to give their opinions on the methods used and whether those methods confirm to the candidates preferences and learning style.

During Training: It is the phase at which instruction is started. This phase usually consist of short tests at regular intervals

After Training: It is the phase when learner’s skills and knowledge are assessed again to measure the effectiveness of the training. This phase is designed to determine whether training has had the desired effect at individual department and organizational levels. There are various evaluation techniques for this phase.

The various methods of training evaluation are: Observation Questionnaire Interview Self diaries Self recording of specific incidents

The Four Levels Reaction Learning Behavior Results

All about Kirkpatrick In 1959, Kirkpatrick wrote four articles describing the four levels for evaluating training programs. He was working on his dissertation for a Ph.D. when he came up with the idea of defining evaluation. Evaluation, as according to Kirkpatrick, seems to have multiple meanings to training and developmental professionals. Some think evaluation is a change in behavior, or the determination of the final results.

KirkPatrick’s four levels of evaluation of training impact Level Reaction Learning Behaviour Results Questions Were the employees pleased with the programme? What did the participants learn in the programme? Did the participants change their behaviour based on what was learned? Did the change in behaviour positively affect the organization?

“The Four Levels represent a sequence of ways to evaluate (training) programs….As you move from one level to the next, the process becomes more difficult and time-consuming, but it also provides more valuable information.”

Reaction – is the measuring of the reaction of the participants in the training program. what participants thought of the program, including materials, instructors, facilities, methodology, content etc.

Learning – is the change in the participants’ attitudes, or an increase in knowledge, or greater skills received, as a result of the participation of the program. learning evaluation is concerned with measuring the extent to which principles, facts, techniques and skills have been acquired

Behavior –how much transfer of knowledge, skills, and attitude occurs after the training. behavioral change is measured to determine the extent to which principles, facts, techniques, and skills have been acquired.

Results – is the most important and difficult of all - determining final results after training evaluation of results involves monitoring organizational improvement such as cost savings, work output changes, and quality changes.

Kaufman’s five levels of evaluation of training impact Some researchers, recognising some shortcomings of Kirkpatrick’s four-level approach, have attempted to modify and add to this basic framework. Kaufman has expanded the definition of level 1 and added a fifth level addressing societal issues.

LevelEvaluation focus 5. Societal outcomes Societal and client responsiveness, consequences and payoffs 4. Organizational output Organizational contributions and payoffs 3. ApplicationIndividual and small group (products) utilization within the organization 2. AcquisitionIndividual and small group mastery and competency 1b. ReactionMethods, means, and process acceptability and efficiency 1a. EnablingAvailability and quality of human, financial, and physical resources input

At level 1, the factor of the concept of enabling addressees the availability of various resource inputs necessary for a successful intervention. Level 5 is the evaluation of societal and client responsiveness. This moves evaluation beyond the organization, and examines the extent to which the performance improvement programme has enhanced society and the environment surrounding the organization.

The CIRO Approach Developed by War, Bird, and Racham, is a unique way to classify evaluation process. Four general categories of evaluation are described, which form the letters CIRO 1. Context evaluation 2. Input evaluation 3. Reaction evaluation 4. Outcome evaluation

Context evaluation Context evaluation involves collecting information about a performance deficiency, assessing the information to establish training needs and, on the basis of those findings, setting objectives. Context evaluation involves obtaining and using information about the current operational situation (or context) to determine training needs and objectives.

Three types of objectives may be evaluated- 1.Ultimate objectives (the particular deficiency in the organization that the programme will eliminate or overcome) 2.Intermediate objectives (the changes in employees’ work behavior that will be necessary for the ultimate objectives to be attained) 3.Immediate objectives (the new knowledge, skills, or attitudes that employees must acquire to change their behavior and reach the intermediate objective)

Input Evaluation Input Evaluation involves obtaining and using information about possible training resources to choose between alternative inputs to training. It involves analysing the resources available (both internal and external) and determining how they can be deployed so that there is a maximum chance of achieving the desired objectives.

Input evaluation refers to the process of collecting evidence and using it to decide on the training methods

Reaction Evaluation Reaction evaluation involves obtaining and using information about participants’ reactions to improve the HRD process. The distinguishing feature of this type of evaluation is that it relies on the subjective input of the participants. Their views can prove extremely helpful when collected and used in a systematic manner.

Outcome Evaluation involves obtaining and using information about the results or outcomes of training, and is usually regarded as the most important part of evaluation. If outcome evaluation is to be successful, it requires careful preparation before the programme begins.

Stages of outcome evaluation- 1.Defining trend objectives 2.Selecting or constructing some measures of those objectives 3.Making the measurements at the appropriate time 4.Assessing the results and using them to improve later programmes.

A successful training programme produces some initial change in a participant, an immediate outcome, which is reflected in changes of knowledge, skills, or attitudes. These changes can be measured during or at the end of the programme.

The Philip’s Five Level ROI Framework The ROI process adds a fifth level to the four levels of evaluation developed by Kirkpatrick.

LevelBrief Description 1. Reaction and planned action 2. Learning 3. Job applications 4. Business results 5. Return on investment

1.Reaction and planned action Programme participants’ satisfaction is measured, along with a listing of how they plan to apply what they have learned. This level of evaluation is important as a participant satisfaction measure, a favorable reaction does not ensure that participants have learned new skills or knowledge.

2. Learning Measurements focus on what participants learned during the programme using tests, skill practices, role plays, simulations, group evaluations, and other assessment tools. A learning check is helpful to ensure that participants have absorbed the material and know how to use it.

3. Job application A variety of follow up methods are used to determine if participants apply what they learned on the job. Level 3 evaluations are important to gauge the success of the programme’s application, it still does not guarantee that there will be a positive impact in the organization.

4. Business results The measurement focuses on actual results programme participants achieve as they successfully apply the programme material. Typical level 4 measures include output, quality, costs, time, and customer satisfaction.

5. Return on Investment Comparing the monetary benefits from the programme with its costs. It is usually presented as a percent or cost/benefit ratio. Very few organizations actually conduct evaluations at ROI level, perhaps because ROI evaluation is often characterised as a difficult and expensive process.

Steps to calculate return on investment (ROI),: 1. Identify outcome(s) (e.g., quality, accidents) 2. Place a value on the outcome(s) 3. Determine the change in performance after eliminating other potential influences on training results. 4. Obtain an annual amount of benefits (operational results) from training by comparing results after training to results before training

Steps to calculate return on investment (ROI), : (continued) 5. Determine training costs (direct costs + indirect costs + development costs + overhead costs + compensation for trainees) 6. Calculate the total savings by subtracting the training costs from benefits (operational results) 7. Calculate the ROI by dividing benefits (operational results) by costs.  The ROI gives you an estimate of the rupee return expected from each rupee invested in training

Although business results and ROI are desired, it is important to evaluate the other levels. A chain of impact should occur through the levels as the skills and knowledge learned (level2) are applied on the job (level3) to produce business results (level4).

Outcomes Used in Evaluating Training Programs: Cognitive Outcomes Skill-Based Outcomes Affective Outcomes Results Return on Investment

Outcomes Used in Evaluating Training Programs: (continued)  Cognitive Outcomes  Determine the degree to which trainees are familiar with the principles, facts, techniques, procedures, or processes emphasized in the training program.  Measure what knowledge trainees learned in the program.  Skill-Based Outcomes  Assess the level of technical or motor skills.  Include acquisition or learning of skills and use of skills on the job

Outcomes Used in Evaluating Training Programs: (continued)  Affective Outcomes  Include attitudes and motivation.  Trainees’ perceptions of the program including the facilities, trainers, and content.  Results  Determine the training program’s payoff for the company

Outcomes Used in Evaluating Training Programs: (continued) Return on Investment (ROI) –Comparing the training’s monetary benefits with the cost of the training. Direct costs Indirect costs Benefits

How do you know if your outcomes are good? Good training outcomes need to be: Relevant Reliable Discriminate Practical

Importance of Training Cost Information  To understand total expenditures for training, including direct and indirect costs.  To compare costs of alternative training programs.  To evaluate the proportion of money spent on training development, administration, and evaluation as well as to compare money spent on training for different groups of employees.  To control costs