Strategic Planning for Training Evaluation Presented by: Nita C. Lescher, M.S.W., L.I.C.S.W. American Humane Association Briefly discuss AHA’s experience with training evaluation. Go around the room, ask participants to introduce themselves and explain their experience with training evaluation.

Before, During, and After Training When to Evaluate? Before, During, and After Training Introduce topic, explain that evaluation is important, not just as an overall topic, but also as concrete activities to be conducted before, during, and after training. Explain that we will discuss the purposes for each phase of evaluation next.

Why Evaluate? Before Training Identify needs or problems. Identify specific job competencies to close gaps. Identify the type of evaluation needed. Be sure to ask trainees, What can evaluation before training do for your organization? Try to generate these responses, if not given go over.

During Development of Training Why Evaluate? During Development of Training Determine which staff should be the targets of training. Provide feedback to curriculum designers on content, methods, and materials. Determine if course delivery is consistent and follows curriculum specifications. Be sure to ask trainees, What can evaluation during the development of training do for your organization? Try to generate these responses, if not given go over.

Why Evaluate? After Training Note changes in participants’ attitudes and reactions to the course. Assess trainees’ mastery of knowledge. Assess trainees’ mastery of skills. Assess transfer of new knowledge and skills to the job. Measure whether or not performance gaps have begun to close. Measure relationship of training to agency goals and client outcomes. Be sure to ask trainees, What can evaluation after training do for your organization? Try to generate these responses, if not given go over. Explain that this category is typically what organizations think about when evaluation is even thought about in the first place.

Strategic Plan for Training Evaluation Why do it? Specify clear relationships among agency goals, resources, purposes of training evaluation and decisions about what to evaluate. Ask, what is the purpose of strategic planning for training evaluation. Provide this definition.

Factors to Consider Purpose of training Cost of evaluation Centrality of subject matter to competent job performance Consequences of poorly developed/delivered training Number of staff to be reached Length of course Legal mandates, public and political pressures Agency goals Go over these factors. Explain that they will help decide on the components of a strategic plan. Ask participants, anything else?

AHA Training Evaluation Model The famous “Apple Model” for a strategic plan for training evaluation.

Apple Peel Levels Level 1 = Course Evaluation of the training: Content Structure Methods Materials Delivery Go over each level. Explain the purpose of each level. Ask participants, What experience have you had with this level?

Level 2 = Satisfaction (Happiness Sheets) Apple Peel Levels Level 2 = Satisfaction (Happiness Sheets) Assesses trainees’ feelings about: Trainer Quality of material presented Methods of presentation Environment (e.g., room temperature) This is the most common method for training evaluation. Ask participants, What experience have you had with this level?

Apple Peel Levels Level 3 = Opinion Assesses trainees’ attitudes/perceptions about: Utilization of training Own learning Training’s value Most higher level evaluation is done at this level. Ask participants, What experience have you had with this level?

Apple Peel Levels Level 4 = Knowledge Acquisition Assesses learning and recalling terms, definitions, and facts Level 5 = Knowledge Comprehension Assesses understanding concepts and relationships, recognizing examples in practice, and problem solving Ask participants, What experience have you had with this level? Ask for examples.

Level 6 = Skill Demonstration Apple Peel Levels Level 6 = Skill Demonstration Assesses application of learned material to new and concrete situations within the training environment Level 7 = Skill Transfer Assesses application of new knowledge and skills outside the classroom and on the job Most organizations may strive for this level, but it is expensive and the effects of training may be confounded by other factors. However, evaluation at this level will tell you whether training made a difference. Ask participants, What experience have you had with this level? Ask for examples.

Level 9 = Client Outcomes Level 10 = Community Impacts Apple Peel Levels Level 8 = Agency Impact Level 9 = Client Outcomes Level 10 = Community Impacts Assesses impact of training at agency, client, and community levels Ask participants, What experience have you had with this level? Ask for examples. Ask participants: Why is it difficult to evaluate at this level? Possible answers: Cost Difficulty of isolating effects Lack of support from agency

Model for Successful Training Course Evaluation Step 1: Identify the purposes of the evaluation. Step 2: Determine the levels of evaluation needed. Step 3: Develop an evaluation design. Step 4: Select or develop instruments. Step 5: Develop a data collection plan. Step 6: Pilot and revise instruments and procedures. Step 7: Collect evaluation data. Step 8: Analyze evaluation data. Step 9: Report results. Step 10: Provide feedback/disseminate results. Go over these steps. Explain that it is a linear process that requires a commitment to the process. When discussing Step 3, refer participants to Handout “Sample Designs for Common Types of Evaluation”. (Need to find electronic version or have this handout scanned.)