1. Presenter: Angelia Bendolph Harkcon, Inc.
Evaluation Training Workshop Based on USCG SOP, Vol. 3, Training Evaluation December 2015 Version 5
Presenter: Angelia Bendolph
Harkcon, Inc.

2. Training Agenda Overview Test Your Knowledge Training Content/Practice
Quick Check
Questions/Feedback

3. Learning Objectives
Identify the basis of the USCG’s evaluation program
Recognize Kirkpatrick’s Levels of Evaluation
Describe Level 1 evaluation
Discriminate between learner-centered and trainer-centered evaluations

4. Learning Objectives Describe Level 2 evaluation
Discriminate between pre-test and post-test

5. Knowledge Check Use Kahoot.it interactive game to check your knowledge
Get your smartphone, laptop, or tablet
Go to kahoot.it

6. What is Evaluation?
Evaluation is a process used to measure the value and effectiveness of training
It occurs in every phase of the ADDIE model.
The evaluation process must be evaluated itself (meta-evaluation).
Source: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell

7. Two Types of Evaluation
In instructional design, two types:
Formative evaluation
Summative evaluation
Dick and Carey, The Systematic Design of Instruction;
Graphic from

8. Kirkpatrick’s Levels of Evaluation
The Coast Guard’s evaluation program
Original Four levels (Donald Kirkpatrick -1998)
Modified by Jim and Wendy Kirkpatrick’s New World Kirkpatrick Model
From pp. 2-3 SOP December 2015
Level 1: Reaction Evaluation – Captures the degree to which the student reacts favorably to the training event
Level 2: Learning Evaluation – Assesses the degree to which the student demonstrates mastery of TPOs in the training environment
Level 3: Behavior Evaluation – Measures the degree to which the intervention impacts actual on-the-job performance
Level 4: Results Evaluation - Determines to what degree the targeted outcomes occur as a result of the learning event(s) and subsequent reinforcement. These outcomes just contribute to the organization meeting its purpose and mission.
A Level 4 answers the question, “Is this what the organization exists to do/deliver/contribute?”
Source: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell

9. Kirkpatrick’s Level 1 - Reaction
Source: from

10. Kirkpatrick’s Level 2 - Learning
Source: from

11. Kirkpatrick’s Level 3 - Behavior
Source: from

12. Kirkpatrick’s Level 4 - Results
Source: from

13. Kirkpatrick’s Level 1 Level 1 – Reaction “Smile sheets”
An evaluation of the learner’s impressions of the training
Level of customer satisfaction
Student engagement
Content relevance
“Smile sheets”
Evaluation in the implementation phase
There is a strong correlation between Level 1 and 2 evaluations
Segment timing can be affected by differences in facilitators, materials and equipment. A designer should account for any changes in variables that may affect timing.
Good idea to factor in extra timing
Try several run-throughs to get an average time
Deficiencies in materials: Can range from typographic errors in the copy to offensive graphics or wording
Having materials in the right language
Structure: Flow of the course from both the facilitator and the participants’ viewpoints
Target population: Pilot tests often reveal holes in the population analysis whether undershot or overshot the average learner.
Source: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell and USCG SOP Volume 3 December 2015

14. Kirkpatrick’s Level 1 (cont.)
Trainer-centered
Learner-centered
Activity 1
The aim of each of the level 1 evaluations: survey or focus group, is to discover learners’ reactions to the process.
A good strategy for level 1 evaluation is to determine learners’ initial responses to the experience as they exit the training.
The freshest and most accurate data for a level 1 evaluation comes at the immediate conclusion of the training.
Sources: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell and
Standard Operating Procedures (SOP) for the Coast Guard’s Training System, Vol. 3 , December 2015

15. Kirkpatrick’s Level 2 Level 2: Learning
Knowledge
Skill
Attitude
Confidence
Commitment
Kirkpatrick New World Level 2 =
Knowledge + Skills + Attitude + Confidence + Commitment
Knowledge – “I know it”
Skill – “I can do it right now”
Attitude – “I believe this will be worthwhile to do on the job”
Confidence – “I think I can do it on the job.”
Commitment – “I intend to to it on the job.”
Sources: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell and
Standard Operating Procedures (SOP) for the Coast Guard’s Training System, Vol. 3 , December 2015

16. Kirkpatrick’s Level 2 (cont.)
Level 2: Learning
Pretest results vs. Post-test results
Kirkpatrick New World Level 2 =
Knowledge + Skills + Attitude + Confidence + Commitment
Activity 2
Knowledge – “I know it”
Skill – “I can do it right now”
Attitude – “I believe this will be worthwhile to do on the job”
Confidence – “I think I can do it on the job.”
Commitment – “I intend to to it on the job.”
Sources: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell and
Standard Operating Procedures (SOP) for the Coast Guard’s Training System, Vol. 3 , December 2015

17. Questions

18. Wrap-up
Complete the evaluation
Thanks for your time!