Presenter: Angelia Bendolph Harkcon, Inc. Evaluation Training Workshop Based on USCG SOP, Vol. 3, Training Evaluation December 2015 Version 5 Presenter: Angelia Bendolph Harkcon, Inc.

Training Agenda Overview Test Your Knowledge Training Content/Practice Quick Check Questions/Feedback

Learning Objectives Identify the basis of the USCG’s evaluation program Recognize Kirkpatrick’s Levels of Evaluation Describe Level 1 evaluation Discriminate between learner-centered and trainer-centered evaluations

Learning Objectives Describe Level 2 evaluation Discriminate between pre-test and post-test

Knowledge Check Use Kahoot.it interactive game to check your knowledge Get your smartphone, laptop, or tablet Go to kahoot.it

What is Evaluation? Evaluation is a process used to measure the value and effectiveness of training It occurs in every phase of the ADDIE model. The evaluation process must be evaluated itself (meta-evaluation). Source: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell

Two Types of Evaluation In instructional design, two types: Formative evaluation Summative evaluation Dick and Carey, The Systematic Design of Instruction; Graphic from http://instructionaldesignandtraining.blogspot.com/2011/05/addie-evaluation-phase.html

Kirkpatrick’s Levels of Evaluation The Coast Guard’s evaluation program Original Four levels (Donald Kirkpatrick -1998) Modified by Jim and Wendy Kirkpatrick’s New World Kirkpatrick Model From pp. 2-3 SOP December 2015 Level 1: Reaction Evaluation – Captures the degree to which the student reacts favorably to the training event Level 2: Learning Evaluation – Assesses the degree to which the student demonstrates mastery of TPOs in the training environment Level 3: Behavior Evaluation – Measures the degree to which the intervention impacts actual on-the-job performance Level 4: Results Evaluation - Determines to what degree the targeted outcomes occur as a result of the learning event(s) and subsequent reinforcement. These outcomes just contribute to the organization meeting its purpose and mission. A Level 4 answers the question, “Is this what the organization exists to do/deliver/contribute?” Source: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell

Kirkpatrick’s Level 1 - Reaction Source: from http://summitsafety.wordpress.com

Kirkpatrick’s Level 2 - Learning Source: from http://summitsafety.wordpress.com

Kirkpatrick’s Level 3 - Behavior Source: from http://summitsafety.wordpress.com

Kirkpatrick’s Level 4 - Results Source: from http://summitsafety.wordpress.com

Kirkpatrick’s Level 1 Level 1 – Reaction “Smile sheets” An evaluation of the learner’s impressions of the training Level of customer satisfaction Student engagement Content relevance “Smile sheets” Evaluation in the implementation phase There is a strong correlation between Level 1 and 2 evaluations Segment timing can be affected by differences in facilitators, materials and equipment. A designer should account for any changes in variables that may affect timing. Good idea to factor in extra timing Try several run-throughs to get an average time Deficiencies in materials: Can range from typographic errors in the copy to offensive graphics or wording Having materials in the right language Structure: Flow of the course from both the facilitator and the participants’ viewpoints Target population: Pilot tests often reveal holes in the population analysis whether undershot or overshot the average learner. Source: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell and USCG SOP Volume 3 December 2015

Kirkpatrick’s Level 1 (cont.) Trainer-centered Learner-centered Activity 1 The aim of each of the level 1 evaluations: survey or focus group, is to discover learners’ reactions to the process. A good strategy for level 1 evaluation is to determine learners’ initial responses to the experience as they exit the training. The freshest and most accurate data for a level 1 evaluation comes at the immediate conclusion of the training. Sources: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell and Standard Operating Procedures (SOP) for the Coast Guard’s Training System, Vol. 3 , December 2015

Kirkpatrick’s Level 2 Level 2: Learning Knowledge Skill Attitude Confidence Commitment Kirkpatrick New World Level 2 = Knowledge + Skills + Attitude + Confidence + Commitment Knowledge – “I know it” Skill – “I can do it right now” Attitude – “I believe this will be worthwhile to do on the job” Confidence – “I think I can do it on the job.” Commitment – “I intend to to it on the job.” Sources: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell and Standard Operating Procedures (SOP) for the Coast Guard’s Training System, Vol. 3 , December 2015

Kirkpatrick’s Level 2 (cont.) Level 2: Learning Pretest results vs. Post-test results Kirkpatrick New World Level 2 = Knowledge + Skills + Attitude + Confidence + Commitment Activity 2 Knowledge – “I know it” Skill – “I can do it right now” Attitude – “I believe this will be worthwhile to do on the job” Confidence – “I think I can do it on the job.” Commitment – “I intend to to it on the job.” Sources: ISD from the Ground Up: A No-Nonsense Approach to Instructional Design by Chuck Hodell and Standard Operating Procedures (SOP) for the Coast Guard’s Training System, Vol. 3 , December 2015

Questions

Wrap-up Complete the evaluation Thanks for your time!