FORMATIVE EVALUATION Intermediate Injury Prevention Course August 23-26, 2011, Billings, MT

Session Goal To provide participants with a basic knowledge of formative evaluation and its use when developing an injury prevention program.

Describe the purpose of formative evaluation Describe several formative evaluation methods Provide resources for each formative evaluation method Session Objectives

The 4 Stages of Evaluation Review Formative  Will this work? Process  Are we doing what we set out to do? Impact  Can we see any changes? Outcome  Have we reduce injuries?

The 4 Stages of Evaluation Review Process Evaluation Impact Evaluation Outcome Evaluation Long-term: Timely Feedback Baseline Data Collection Project Implementation Pre-TestingMonitoring Health Outcomes Looks at Changes in Target Audience Formative Evaluation Short-term: Knowledge, Attitudes & Practice

Formative Evaluation Defined The process of testing and /or obtaining expert review of program plans, messages, materials, strategies, or modifications, for weaknesses and strengths before they are put into effect.

Formative Evaluation Purpose Get target audience perception of the possible interventions Gain community support Ensure program materials & activities are acceptable or culturally appropriate Identify problems with program concepts or design Used in conjunction with process evaluation

Formative Evaluation When to Conduct New program New materials Existing program is:  modified,  having problems, or  adapted for a new setting, population, problem, or behavior Ongoing at all stages of program

Formative Evaluation Who to Involve - Stakeholders The people with a vested interest in a program and the future use of its evaluation

Formative Evaluation Categories of Stakeholders Implementers Partners Participants Decision makers

Formative Evaluation Stakeholders Example: Car seat distribution program Who should we include? Implementers Partners Participants Decision makers

Formative Evaluation Data Collection Methods 1.Focus Groups 2.Key Informant Interviews 3.Town Hall Meetings 4.Expert Reviews 5.Surveys

Formative Evaluation Focus Groups - Purpose To gather information on a particular topic by conducting an informal discussion between 6 to 10 individuals similar to target population

Formative Evaluation Focus Groups 8 Important Steps Secret Ingredients: -Focus group similar in makeup to target population -Requires an experienced facilitator

Formative Evaluation Key Informant Interviews - Purpose To obtain detailed information by asking broad open-ended questions and by probing interviewee

Formative Evaluation Key Informant Interviews 7 Important Steps Secret Ingredients: -Take care to select informants with various points of view -Remember to first establish rapport

KEY INFORMANT EXERCISE Scenario: Local IPS is seeking to determine attitudes and beliefs about ATV related injuries and possible interventions. They will interview two key community members.

Formative Evaluation Town Hall Meetings - Purpose To gather a diverse group of community members together to think and talk about an issue

Formative Evaluation Town Hall Meetings 8 important steps Special Ingredients  Plan early  Make sure there is plenty of time for interaction between audience and presenters

Formative Evaluation Expert Reviews - Purpose To identify mistakes, concerns, and general areas for improvement.

Formative Evaluation Expert Reviews 7 important steps Special Ingredients  The committee should be continuously involved in the process.  Include detailed instruction in the job description.

Formative Evaluation Surveys - Purpose To obtain SPECIFIC information from a targeted group of people using a set of defined questions.

Formative Evaluation Surveys 7 important steps Special Ingredients  Avoid overloading the survey  Design questionnaire such that its clear and simple.

Formative Evaluation Examples in IHS Sleep Safe and Ride Safe Curriculums – Focus Groups, Key Informant Interviews, Expert Review Animal Control Project - Surveys Child Passenger Safety Training - Expert Review IHS Area Injury Prevention Program Evaluations - Expert Review

Formative Evaluation Conclusion Formative evaluation  Tests program plans, messages, materials, strategies, or modifications for weaknesses and strengths before they are put into effect.  Allows the evaluator to probe the feelings, beliefs, and impressions of participants.  Generates community support for the proposed project or program.