Evaluation of Training and Education Activities

Objectives By the end of this presentation, participants will be able to List reasons why evaluation is needed Describe different types of evaluation Discuss how to use different types of evaluation for training and education activities 2

Evaluation (1) As trainers, what do we want to evaluate? 3

Evaluation (2) What have we accomplished? What was the training? Where? How many people were trained? Cost? How well have we done? Did the trainings go as planned? Did the participants like the training? Was the training relevant? What were the strengths and weaknesses? 4

Evaluation (3) How effective is the training? To what extent did the training achieve the objectives? Did the participants increase their knowledge and skills? Did they change their job performance? Did health status improve? How can we improve on it? Are the training objectives appropriate, given the available resources? 5

Systematic Health Education Process 6 DevelopmentPilot-testing (Formative Evaluation) Implementation (Process Evaluation) Assess Effectiveness (Outcome and Impact Evaluation) (Formative Research) Needs Assessment

7 When to Begin Evaluation? Consider evaluation up front (in planning phase of training project) Allocate time Allocate resources Evaluation actually occurs at every stage of development Determine the type of evaluation you need

Levels of Evaluation Formative (before) Identifies strengths and weaknesses of training/product before implementation Process (during) Describes the implementation of the training or distribution/use of the product Summative/Outcome/Impact (after) Assesses the short-term and long-term effects of the training/product 8

Four Types of Evaluation 9 Impact Outcome Process Formative Program implementation ends here. Program implementation occurs here. Summative Evaluation

10 Goals and Objectives Goal Describes the overall mission/purpose of the program Broad general statement that clearly states the problem that will be addressed Objectives State the expected result Specific, measurable, attainable, realistic, time- framed Partial accomplishment of a goal Activities Specific actions or procedures expected to occur to achieve objectives

Formative Evaluation 11 Impact Outcome Process Formative Program implementation ends here. Program implementation occurs here. Summative Evaluation

Formative Evaluation (Pilot Testing) Conducted with the target audience Answers questions about whether trainings/products are Understandable, relevant, attractive, credible, acceptable, effective Identifies strong and weak points Identifies confusing terms or concepts Allows for necessary revision prior to implementation Prevents production of ineffective products 12

Formative Evaluation Methods Focus groups With various types of participants (e.g., experienced, non-experienced) Interviews (key informant/audience) Informal discussion Open-ended questions Understanding Appeal Utility Test the curriculum in the “real” environment Readability testing (literacy and comprehension) 13

Formative Evaluation Measures Ease of use or intuitiveness Readability Reading level Tone Fonts, colors, spacing Comprehension Terminology and illustrations Concepts 14

TB: The Hollywood Movie

Formative Evaluation Screen tests Movie screenings Focus groups TB

Process Evaluation 17 Impact Outcome Process Formative Program implementation ends here. Program implementation occurs here. Summative Evaluation

Process Evaluation (Monitoring of Training/Curriculum) Examines the procedures and tasks involved in implementing a training/product Assesses if the training/product is working as planned Answers questions such as: Are the activities being implemented as planned? Is the material being used as intended? How many participants attended the program? Were the participants satisfied with the educational materials? 18

Process Evaluation Measures Product inventory Hard copies distributed Web pages viewed or downloaded Attendance Checklist and timetable review Budget, $$$$ spent Audience feedback on quality of program/material during implementation (participant reactions) 19

Participant Reactions Do audience/participants think that: Objectives are relevant to participants' work Course content is realistic Training methods are suitable Facilitators are effective Training environment (classrooms, meals, accommodation) contributes to learning 20

Participant Reactions Methods Written feedback – participants write down their reactions Ensures that all participants provide comments, not just the outspoken ones Both open- and closed-ended questions are asked Oral feedback and observations Observations by facilitators Facilitators meet at the end of each day to discuss their opinions and participant comments 21

Participant Reaction Form 22

Process Evaluation Ticket sales Critical reviews Rotten tomatoes TB

Outcome Evaluation 24 Impact Outcome Process Formative Program implementation ends here. Program implementation occurs here. Summative Evaluation

Outcome and Impact Evaluation Answers the questions… What have you accomplished? How well have you done it? How much have you done? How effective have you been? 25

Outcome Evaluation Based on learning objectives Evaluation of learning & performance of skills Assesses immediate or short-term effects End of course or immediately after distribution Helps draw conclusions about effectiveness Answers questions such as: Were learning objectives achieved? What knowledge and skills were gained as a result of the training/product? 26

Outcome Evaluation Effectiveness of Self-Study Modules on Tuberculosis Knowledge Mean Scores Pre-test Post-test Print 74%91% Satellite 81%92% Web 76%92% (p<.01) 27

Impact Evaluation 28 Impact Outcome Process Formative Program implementation ends here. Program implementation occurs here. Summative Evaluation

Impact Evaluation Gathers information about the long-range results of the training/product and improvements in health status 3 months, 6 months, one year following implementation Answers questions such as: Did the user’s job performance improve? What are the changes in job practice of user or participant? What changes were implemented in the program? Costly, time consuming, but important 29

Should You Measure Impact? Ask yourself the following questions: Do you have the time and resources? Do you have support from management and partners? Are you able to reach the audience at this later date? What will motivate the audience to participate? 30

Impact Evaluation Methods 31 Methods Surveys Key-informant interviews Direct observation/performance checklists Document/record reviews/treatment registers Reviewing of program indicators Timeline Depends on the goal of the training; however, generally 1 to 6 months after training

Summative Evaluation Domestic and international box office revenues DVD sales Franchise opportunities TB

Evaluation Reports

Justify program, course, or material to stakeholders Allow for identification of areas for improvement Provide evidence for additional needs Share lessons learned

Most Importantly… Plan for evaluation at the beginning and conduct throughout development process