1. Kirkpatrick’s Levels of Evaluation
Presentation by:
Debra Sieloff
Welcome to our tutorial on Kirkpatrick's Four Levels of Evaluation.

2. Donald Kirkpatrick
Kirkpatrick developed a model of training evaluation in 1959
Arguably the most widely used approach
Simple, Flexible and Complete
4-level model
Donald Kirkpatrick developed a model of training evaluation in 1959 that has served the training community like no other.
This 4-level model is arguably the most widely used approach in the world-even today. It's simple. Flexible. Complete.
It presents four types of evaluations.those for reaction, learning, behavior and results.

3. The Four Levels Level I: Evaluate Reaction Level II: Evaluate Learning
Level III: Evaluate Behavior
Level IV: Evaluate Results
Fifth level was recently “added” for return on investment (“ROI”) but this was not in Kirkpatrick’s original model
It is important to remember that each level is different.
Level 1 evaluates Reaction;
Level 2 measures Learning;
Level 3 refers to Behavior.
Level 4 looks at Results.
Let’s take a look at each of Kirkpatrick’s level in detail.

4. Reaction - What Is It? Level 1
How favorably participants react to the training (“Customer satisfaction”)
Collects reactions to instructor, course, and learning environment
Communicates to trainees that their feedback is valued
Can provide quantitative information
A Level One evaluation measures audience reaction.
Kirkpatrick says that evaluating reaction is the same thing as measuring customer satisfaction in that:
It gives you feedback so you know how customers are reacting
right now; and
It shows you care how they react.
Level One also provides quantitative data on customer satisfaction.

5. Reaction - What It Looks Like
Level 1
Reaction - What It Looks Like
Questionnaire - Most common collection tool
Content
Methods
Media
Trainer style
Facilities
Course materials
Most evaluators use the questionnaire for Level Two evaluations.
A questionnaire can identify what to keep, delete, or improve.
As a result, changes may be made in the areas of content, methods, media, trainer style, facilities, and/or course materials.

6. Reaction: Connection to Other Levels
The Connection - Can ask trainees if they:
Will use new skill(s) or information (Level II)
Plan to change behavior (Level III)
Expect improvements in results (Level IV)
The Disconnection - Does not:
Measure what was learned (Level II)
Guarantee behavioral change (Level III)
Quantify results from learning (Level IV)
At Level One, you can ask trainees if they plan to use the newly learned skills or information. You may also ask if they plan to change their behavior or expect performance improvements.
Conversely, Level One cannot serve as a substitute for measuring actual learning, behavior or results. Those are properties of Level Two, Three, and Four evaluations.

7. Reaction - How to Perform
Level 1
Reaction - How to Perform
Determine what you want to find out
Design a form to collect/quantify reactions
Do Immediately
Develop acceptable scoring standards
Follow-up as appropriate
In order to perform a Level One evaluation, first determine what
you want to find out.
Then, design a form to collect or quantify learner reactions.
You want to do this immediately, so that you capture everyone’s responses.
Develop acceptable scoring standards. Know what you will accept, and
what you won’t.
Follow-up as appropriate, communicating the results of your evaluation
with those who need to know.

8. Learning - What Is It? Level 2 Knowledge Skills Attitudes
Let’s review Level Two --”Learning”.
Instructors can affect three cognitive areas during training--they can teach new knowledge, skills, and attitudes.
Therefore, when we measuring learning, we determine if one or more of the following has occurred:
Was knowledge learned or gained?
Were new skills or ways to improve existing skills gained?
Were attitudes changed?

9. Learning: Connection to Other Levels
The Connection - People who learn can:
Experience pride (Level I)
Experiment with new behaviors (Level III)
Achieve better results (Level IV)
The Disconnection - It doesn’t ensure they:
Liked training program (Level I)
Will behave differently (Level III)
Will get expected results (Level IV)
Level Two can provide information that relates to the other levels.
As participants gain new knowledge or skills, they may experience pride from learning new things. They may be encouraged to experiment with new behaviors. They may even achieve better outcomes as they apply them.
However, just because understanding can be proven, does not mean that the participants liked the training, that they will behave differently, or will achieve expected results in the real world.

10. Learning - What It Looks Like
Level 2
Learning - What It Looks Like
Media used to measure learning:
Text
Voice
Demonstration
Methods used to measure learning:
Interviews
Surveys
Tests (pre-/post-)
Observations
Combinations
Media used to help measure learning include the written word, voice, and demonstration.
Specific methods that may be employed to check for learning include interviews, surveys, pre- and post-tests, observations, or any combinations of these.
To view actual examples of learning evaluation tools, be sure to view the QuestionMark web site examples.

11. Learning - How to Perform
Level 2
Learning - How to Perform
Use a control group, if feasible
Evaluate knowledge, skills, and/or attitudes before and after
Get 100% participation or use statistical sample
Follow-up as appropriate
If practical, use a control group to compare knowledge changes.
Design the tests around your objectives. You may want to evaluate learning both before and after the training, too.
Try to survey 100% of the participants to get useful data. If that’s not feasible, use a statistical sampling.
Finally, follow through and communicate with those who need to know.

12. Behavior - What Is It? Level 3
Transfer of knowledge, skills, and/or attitude to the real world
Measure achievement of performance objectives
Let’s move to Level Three, Behavior.
Kirkpatrick says this level identifies behavior change.
Level Three focuses on the learner’s ability to transfer the learning to where it is actually needed--the real world.
Therefore, evaluators measure achievement of performance objectives in real-world settings.

13. Behavior - What It Looks Like
Level 3
Behavior - What It Looks Like
Observe performer, first-hand
Survey key people who observe performer
Use checklists, questionnaires, interviews, or combinations
To measure behavior changes, Level Three evaluators collect data from the setting where the learner must exhibit the new behavior.
Evaluators observe the performer and might gather data from other key individuals, such as subordinates, supervisors, and any others who are credible witnesses of the trainee’s behavior.
Besides observation checklists, evaluators might use questionnaires or interviews to gather data.

14. Behavior: Connection to Other Levels
The Connection - Can determine:
Degree to which learning transfers to the post-training environment (Level II)
The Disconnection - Cannot determine if:
Participants like the training (Level I)
Participants understand (Level II)
Behaviors accomplish results (Level IV)
A Level Three does connect slightly to Level Two issues. For example, a well designed Level Three evaluation will identify the degree to which learning transfers to the post-training environment as a learned behavior.
The emphasis is on learned behavior--not reaction, understanding, or results.

15. Behavior - How to Perform
Level 3
Behavior - How to Perform
Evaluate before and after training
Allow ample time before observing
Survey key people
Consider cost vs. benefits
100% participation or a sampling
Repeated evaluations at appropriate intervals
Use of a control group
Observation is the primary method used in a Level Three evaluation, whether you choose to:
Evaluate before and after training, or
Survey key people who observe the trainee in action
Allow ample time before observing, and consider the cost-to-benefits of your tactics, such as:
100 percent participation or a sampling
Repeated evaluations
Use of a control group

16. Results - What Is It? Level 4 Assesses “bottom line,” final results
Definition of “results” dependent upon the goal of the training program
Let’s turn our attention to Level Four--”Evaluate Results.”
Here we are looking at the final results that occur because the participants attended and participated in the training.
Of course, when we speak of the term “results” we must often times consider the goal of the training program.
Therefore we must be careful to identify what results we’re seeking as we evaluate different programs. Results can be financial. Results should be quantifiable.

17. Results - What It Looks Like
Level 4
Results - What It Looks Like
Depends upon objectives of training program
Quantify
Proof vs. Evidence
Proof is concrete
Evidence is soft
Level Four evaluations capture quantifiable results.
There are two ways to get the numbers: Prove it, or find strong evidence.
Proof is concrete. The results relate to the training in direct, indisputable ways.
In some situations, the cost of getting proof is monumental.
Therefore, evidence--a softer form of correlating results to training--will suffice.

18. Results: Connection to Other Levels
The Connection
Positive Levels 1, 2, 3 evaluations results can provide positive Level 4 evidence
The Disconnection - Does not:
Tell if participants liked training (Level I)
Prove trainees understand (Level II)
Prove use of preferred behaviors (Level III)
Positive results from Levels One, Two, and Three evaluations can provide positive Level Four evidence.
However, Level Four doesn’t tell if participants liked the training or if they gained satisfactory knowledge.
Be careful. Level Four doesn’t always directly correlate to desired behaviors. For example, an increase in sales calls may be due to an increase in company sponsored advertisements rather than a salesperson’s improved performance.

19. Results - How to Perform
Level 4
Results - How to Perform
Use a control group
Allow time for results to be realized
Measure before and after the program
Consider cost versus benefits
Be satisfied with evidence when proof is not possible
To perform this measurement, use a control group. The control group can provide important data, such as the effect of that advertising on increased sales levels mentioned in the last slide.
Allow ample time for results to be achieved.
Collect data both before and after the training program to get additional perspective on the trainee’s performance.
Consider the cost versus the benefits of performing this evaluation.
Finally, be prepared to accept evidence when proof is not possible.

20. Summary Level I: Evaluate Reaction Level II: Evaluate Learning
Level III: Evaluate Behavior
Level IV: Evaluate Results
Congratulations. You’ve made it through Kirkpatrick’s Four Levels of evaluation. Now you know that:
Level One measures participant reaction
Level Two measures new learning
Level Three measures behaviors in the real world; and
Level Four measures the results.
But we’re not going to measure your learning today--We’ll leave
that up to you!

21. Readings to Consider
Kirkpatrick, Donald L. (1998). Evaluating Training Programs: The Four Levels. Berrett-Koehler Publishers.
Worthen, Blaine R, James R. Sanders, Jody L. Fitzpatrick (1997). Program Evaluation: Alternative Approaches and Practical Guidelines (Second Edition).Addison, Wesley, Longman, Inc.
Kirkpatrick, Donald L. (1998). Another Look at Evaluating Training Programs. American Society for Training & Development.
Sieloff, Debra A. (1999). The Bridge Evaluation Model. International Society for Performance Improvement.

22. Thank you for your attention
Thank you for your time and attention. We hope you have enjoyed learning about Kirkpatrick’s Levels of Evaluation.
We at Question Mark strive to improve our products and services. To let us know how you felt about this presentation, visit our North American home page and complete the Learning Center Survey:
Question Mark provides numerous presentations to help our customers, and prospective customers, successfully deploy computerized assessments. We provide public “web-seminars” that use a combination of conference calls and web technology to discuss computerized assessment issues. You can register for a public seminar by going to:
For other material that you might find useful, please visit our Learning Center at:
Thank you once again. This is the end of the presentation.