1. Assessments and the Kirkpatrick Model
Using Assessments to Evaluate Training
Welcome to our tutorial on Kirkpatrick's Four Levels of Evaluation.

2. Donald Kirkpatrick
Kirkpatrick developed a model of training evaluation in 1959
Arguably the most widely used approach
Simple, Flexible and Complete
4-level model
Donald Kirkpatrick developed a model of training evaluation in 1959 that has served the training community like no other.
This 4-level model is arguably the most widely used approach in the world-even today. It's simple. Flexible. Complete.
It presents four types of evaluations.those for reaction, learning, behavior and results.

3. The Four Levels Level I: Evaluate Reaction Level II: Evaluate Learning
Level III: Evaluate Behavior
Level IV: Evaluate Results
Fifth level was recently “added” for return on investment (“ROI”) but this was not in Kirkpatrick’s original model
It is important to remember that each level is different.
Level 1 evaluates Reaction;
Level 2 measures Learning;
Level 3 refers to Behavior.
Level 4 looks at Results.
Let’s take a look at each of Kirkpatrick’s level in detail.

4. Relationship Between Levels
Level 4 - Results Was it worth it?
Level 3 - Behavior KSA being used on the job?
Each subsequent level is predicated upon doing evaluation at lower level
A Level 3 will be of marginal use, if a Level 2 evaluation is not conducted
Level 2 - Knowledge Did they learn anything
Level 1 - Reaction
Was the environment suitable for learning?

5. Only by assessing each level can we yield actionable results
Level 4 - Results Was it worth it?
Level 3 - Behavior KSA being used on the job?
Check Requirements, Systems and Processes
Level 2 - Knowledge Did they learn anything
Check Performance Environment
Level 1 - Reaction
Was the environment suitable for learning?
Improve Knowledge/Skill transfer
Go from top down
What if we did nothing?
Applications of percentage – level 1, level 2, level 3 etc.
Improve Learning Environment

6. Types of Assessments Used at Each Level
Level 4 - Results Was it worth it?
Type
Form
Summative
Correlation of business results with other assessment results
Observation of Performance
360° Survey
Diagnostic
Self-assessment
Test
Reaction
Formative
Survey
Real-time Polling
Quizzing
Level 3 - Behavior KSA being used on the job?
Level 2 - Knowledge Did they learn anything
Level 1 - Reaction
Was the environment suitable for learning?

7. Reaction - What Is It? Level 1
How favorably participants react to the training (“Customer satisfaction”)
Collects reactions to instructor, course, and learning environment
Communicates to trainees that their feedback is valued
Can provide quantitative information
A Level One evaluation measures audience reaction.
Kirkpatrick says that evaluating reaction is the same thing as measuring customer satisfaction in that:
It gives you feedback so you know how customers are reacting
right now; and
It shows you care how they react.
Level One also provides quantitative data on customer satisfaction.

8. Reaction - What It Looks Like
Level 1
Questionnaire - Most common collection tool
Content
Methods
Media
Trainer style
Facilities
Course materials
Most evaluators use the questionnaire for Level Two evaluations.
A questionnaire can identify what to keep, delete, or improve.
As a result, changes may be made in the areas of content, methods, media, trainer style, facilities, and/or course materials.

9. Reaction - How to Perform
Level 1
Determine what you want to find out
Design a form to collect/quantify reactions
Do Immediately
Develop acceptable scoring standards
Follow-up as appropriate
In order to perform a Level One evaluation, first determine what
you want to find out.
Then, design a form to collect or quantify learner reactions.
You want to do this immediately, so that you capture everyone’s responses.
Develop acceptable scoring standards. Know what you will accept, and
what you won’t.
Follow-up as appropriate, communicating the results of your evaluation
with those who need to know.

10. Learning - What Is It? Level 2 Knowledge Skills Attitudes
Let’s review Level Two --”Learning”.
Instructors can affect three cognitive areas during training--they can teach new knowledge, skills, and attitudes.
Therefore, when we measuring learning, we determine if one or more of the following has occurred:
Was knowledge learned or gained?
Were new skills or ways to improve existing skills gained?
Were attitudes changed?

11. Learning - What It Looks Like
Level 2
Media used to measure learning:
Text
Voice
Demonstration
Methods used to measure learning:
Interviews
Surveys
Tests (pre-/post-)
Observations
Combinations
Media used to help measure learning include the written word, voice, and demonstration.
Specific methods that may be employed to check for learning include interviews, surveys, pre- and post-tests, observations, or any combinations of these.
To view actual examples of learning evaluation tools, be sure to view the QuestionMark web site examples.

12. Learning - How to Perform
Level 2
Use a control group, if feasible
Evaluate knowledge, skills, and/or attitudes before and after
Get 100% participation or use statistical sample
Follow-up as appropriate
If practical, use a control group to compare knowledge changes.
Design the tests around your objectives. You may want to evaluate learning both before and after the training, too.
Try to survey 100% of the participants to get useful data. If that’s not feasible, use a statistical sampling.
Finally, follow through and communicate with those who need to know.

13. Behavior - What Is It? Level 3
Transfer of knowledge, skills, and/or attitude to the real world
Measure achievement of performance objectives
Let’s move to Level Three, Behavior.
Kirkpatrick says this level identifies behavior change.
Level Three focuses on the learner’s ability to transfer the learning to where it is actually needed--the real world.
Therefore, evaluators measure achievement of performance objectives in real-world settings.

14. Behavior - What It Looks Like
Level 3
Observe performer, first-hand
Survey key people who observe performer
Use checklists, questionnaires, interviews, or combinations
To measure behavior changes, Level Three evaluators collect data from the setting where the learner must exhibit the new behavior.
Evaluators observe the performer and might gather data from other key individuals, such as subordinates, supervisors, and any others who are credible witnesses of the trainee’s behavior.
Besides observation checklists, evaluators might use questionnaires or interviews to gather data.

15. Behavior - How to Perform
Level 3
Evaluate before and after training
Allow ample time before observing
Survey key people
Consider cost vs. benefits
100% participation or a sampling
Repeated evaluations at appropriate intervals
Use of a control group
Observation is the primary method used in a Level Three evaluation, whether you choose to:
Evaluate before and after training, or
Survey key people who observe the trainee in action
Allow ample time before observing, and consider the cost-to-benefits of your tactics, such as:
100 percent participation or a sampling
Repeated evaluations
Use of a control group

16. Results - What Is It? Level 4 Assesses “bottom line,” final results
Definition of “results” dependent upon the goal of the training program
Let’s turn our attention to Level Four--”Evaluate Results.”
Here we are looking at the final results that occur because the participants attended and participated in the training.
Of course, when we speak of the term “results” we must often times consider the goal of the training program.
Therefore we must be careful to identify what results we’re seeking as we evaluate different programs. Results can be financial. Results should be quantifiable.

17. Results - What It Looks Like
Level 4
Depends upon objectives of training program
Quantify
Proof vs. Evidence
Proof is concrete
Evidence is soft
Level Four evaluations capture quantifiable results.
There are two ways to get the numbers: Prove it, or find strong evidence.
Proof is concrete. The results relate to the training in direct, indisputable ways.
In some situations, the cost of getting proof is monumental.
Therefore, evidence--a softer form of correlating results to training--will suffice.

18. Results - How to Perform
Level 4
Use a control group
Allow time for results to be realized
Measure before and after the program
Consider cost versus benefits
Be satisfied with evidence when proof is not possible
To perform this measurement, use a control group. The control group can provide important data, such as the effect of that advertising on increased sales levels mentioned in the last slide.
Allow ample time for results to be achieved.
Collect data both before and after the training program to get additional perspective on the trainee’s performance.
Consider the cost versus the benefits of performing this evaluation.
Finally, be prepared to accept evidence when proof is not possible.

19. Summary Level I: Evaluate Reaction Level II: Evaluate Learning
Level III: Evaluate Behavior
Level IV: Evaluate Results
Congratulations. You’ve made it through Kirkpatrick’s Four Levels of evaluation. Now you know that:
Level One measures participant reaction
Level Two measures new learning
Level Three measures behaviors in the real world; and
Level Four measures the results.
But we’re not going to measure your learning today--We’ll leave
that up to you!

20. Readings to Consider
Kirkpatrick, Donald L. (1998). Evaluating Training Programs: The Four Levels. Berrett-Koehler Publishers.
Worthen, Blaine R, James R. Sanders, Jody L. Fitzpatrick (1997). Program Evaluation: Alternative Approaches and Practical Guidelines (Second Edition).Addison, Wesley, Longman, Inc.
Kirkpatrick, Donald L. (1998). Another Look at Evaluating Training Programs. American Society for Training & Development.
Sieloff, Debra A. (1999). The Bridge Evaluation Model. International Society for Performance Improvement.