Assessments and the Kirkpatrick Model Using Assessments to Evaluate Training Welcome to our tutorial on Kirkpatrick's Four Levels of Evaluation.

Donald Kirkpatrick Kirkpatrick developed a model of training evaluation in 1959 Arguably the most widely used approach Simple, Flexible and Complete 4-level model Donald Kirkpatrick developed a model of training evaluation in 1959 that has served the training community like no other. This 4-level model is arguably the most widely used approach in the world-even today. It's simple. Flexible. Complete. It presents four types of evaluations.those for reaction, learning, behavior and results.

The Four Levels Level I: Evaluate Reaction Level II: Evaluate Learning Level III: Evaluate Behavior Level IV: Evaluate Results Fifth level was recently “added” for return on investment (“ROI”) but this was not in Kirkpatrick’s original model It is important to remember that each level is different. Level 1 evaluates Reaction; Level 2 measures Learning; Level 3 refers to Behavior. Level 4 looks at Results. Let’s take a look at each of Kirkpatrick’s level in detail.

Relationship Between Levels Level 4 - Results Was it worth it? Level 3 - Behavior KSA being used on the job? Each subsequent level is predicated upon doing evaluation at lower level A Level 3 will be of marginal use, if a Level 2 evaluation is not conducted Level 2 - Knowledge Did they learn anything Level 1 - Reaction Was the environment suitable for learning?

Only by assessing each level can we yield actionable results Level 4 - Results Was it worth it? Level 3 - Behavior KSA being used on the job? Check Requirements, Systems and Processes Level 2 - Knowledge Did they learn anything Check Performance Environment Level 1 - Reaction Was the environment suitable for learning? Improve Knowledge/Skill transfer Go from top down What if we did nothing? Applications of percentage – level 1, level 2, level 3 etc. Improve Learning Environment

Types of Assessments Used at Each Level Level 4 - Results Was it worth it? Type Form Summative Correlation of business results with other assessment results Observation of Performance 360° Survey Diagnostic Self-assessment Test Reaction Formative Survey Real-time Polling Quizzing Level 3 - Behavior KSA being used on the job? Level 2 - Knowledge Did they learn anything Level 1 - Reaction Was the environment suitable for learning?

Reaction - What Is It? Level 1 How favorably participants react to the training (“Customer satisfaction”) Collects reactions to instructor, course, and learning environment Communicates to trainees that their feedback is valued Can provide quantitative information A Level One evaluation measures audience reaction. Kirkpatrick says that evaluating reaction is the same thing as measuring customer satisfaction in that: It gives you feedback so you know how customers are reacting right now; and It shows you care how they react. Level One also provides quantitative data on customer satisfaction.

Reaction - What It Looks Like Level 1 Questionnaire - Most common collection tool Content Methods Media Trainer style Facilities Course materials Most evaluators use the questionnaire for Level Two evaluations. A questionnaire can identify what to keep, delete, or improve. As a result, changes may be made in the areas of content, methods, media, trainer style, facilities, and/or course materials.

Reaction - How to Perform Level 1 Determine what you want to find out Design a form to collect/quantify reactions Do Immediately Develop acceptable scoring standards Follow-up as appropriate In order to perform a Level One evaluation, first determine what you want to find out. Then, design a form to collect or quantify learner reactions. You want to do this immediately, so that you capture everyone’s responses. Develop acceptable scoring standards. Know what you will accept, and what you won’t. Follow-up as appropriate, communicating the results of your evaluation with those who need to know.

Learning - What Is It? Level 2 Knowledge Skills Attitudes Let’s review Level Two --”Learning”. Instructors can affect three cognitive areas during training--they can teach new knowledge, skills, and attitudes. Therefore, when we measuring learning, we determine if one or more of the following has occurred: Was knowledge learned or gained? Were new skills or ways to improve existing skills gained? Were attitudes changed?

Learning - What It Looks Like Level 2 Media used to measure learning: Text Voice Demonstration Methods used to measure learning: Interviews Surveys Tests (pre-/post-) Observations Combinations Media used to help measure learning include the written word, voice, and demonstration. Specific methods that may be employed to check for learning include interviews, surveys, pre- and post-tests, observations, or any combinations of these. To view actual examples of learning evaluation tools, be sure to view the QuestionMark web site examples.

Learning - How to Perform Level 2 Use a control group, if feasible Evaluate knowledge, skills, and/or attitudes before and after Get 100% participation or use statistical sample Follow-up as appropriate If practical, use a control group to compare knowledge changes. Design the tests around your objectives. You may want to evaluate learning both before and after the training, too. Try to survey 100% of the participants to get useful data. If that’s not feasible, use a statistical sampling. Finally, follow through and communicate with those who need to know.

Behavior - What Is It? Level 3 Transfer of knowledge, skills, and/or attitude to the real world Measure achievement of performance objectives Let’s move to Level Three, Behavior. Kirkpatrick says this level identifies behavior change. Level Three focuses on the learner’s ability to transfer the learning to where it is actually needed--the real world. Therefore, evaluators measure achievement of performance objectives in real-world settings.

Behavior - What It Looks Like Level 3 Observe performer, first-hand Survey key people who observe performer Use checklists, questionnaires, interviews, or combinations To measure behavior changes, Level Three evaluators collect data from the setting where the learner must exhibit the new behavior. Evaluators observe the performer and might gather data from other key individuals, such as subordinates, supervisors, and any others who are credible witnesses of the trainee’s behavior. Besides observation checklists, evaluators might use questionnaires or interviews to gather data.

Behavior - How to Perform Level 3 Evaluate before and after training Allow ample time before observing Survey key people Consider cost vs. benefits 100% participation or a sampling Repeated evaluations at appropriate intervals Use of a control group Observation is the primary method used in a Level Three evaluation, whether you choose to: Evaluate before and after training, or Survey key people who observe the trainee in action Allow ample time before observing, and consider the cost-to-benefits of your tactics, such as: 100 percent participation or a sampling Repeated evaluations Use of a control group

Results - What Is It? Level 4 Assesses “bottom line,” final results Definition of “results” dependent upon the goal of the training program Let’s turn our attention to Level Four--”Evaluate Results.” Here we are looking at the final results that occur because the participants attended and participated in the training. Of course, when we speak of the term “results” we must often times consider the goal of the training program. Therefore we must be careful to identify what results we’re seeking as we evaluate different programs. Results can be financial. Results should be quantifiable.

Results - What It Looks Like Level 4 Depends upon objectives of training program Quantify Proof vs. Evidence Proof is concrete Evidence is soft Level Four evaluations capture quantifiable results. There are two ways to get the numbers: Prove it, or find strong evidence. Proof is concrete. The results relate to the training in direct, indisputable ways. In some situations, the cost of getting proof is monumental. Therefore, evidence--a softer form of correlating results to training--will suffice.

Results - How to Perform Level 4 Use a control group Allow time for results to be realized Measure before and after the program Consider cost versus benefits Be satisfied with evidence when proof is not possible To perform this measurement, use a control group. The control group can provide important data, such as the effect of that advertising on increased sales levels mentioned in the last slide. Allow ample time for results to be achieved. Collect data both before and after the training program to get additional perspective on the trainee’s performance. Consider the cost versus the benefits of performing this evaluation. Finally, be prepared to accept evidence when proof is not possible.

Summary Level I: Evaluate Reaction Level II: Evaluate Learning Level III: Evaluate Behavior Level IV: Evaluate Results Congratulations. You’ve made it through Kirkpatrick’s Four Levels of evaluation. Now you know that: Level One measures participant reaction Level Two measures new learning Level Three measures behaviors in the real world; and Level Four measures the results. But we’re not going to measure your learning today--We’ll leave that up to you!

Readings to Consider Kirkpatrick, Donald L. (1998). Evaluating Training Programs: The Four Levels. Berrett-Koehler Publishers. Worthen, Blaine R, James R. Sanders, Jody L. Fitzpatrick (1997). Program Evaluation: Alternative Approaches and Practical Guidelines (Second Edition).Addison, Wesley, Longman, Inc. Kirkpatrick, Donald L. (1998). Another Look at Evaluating Training Programs. American Society for Training & Development. Sieloff, Debra A. (1999). The Bridge Evaluation Model. International Society for Performance Improvement.