Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006
Overview Warm-up Where does Evaluation fit? Approaches to Evaluation Examples Q&A: Evaluating Your Programs
Warm-Up
Who here is … Evaluator? Service Provider? Administrator?
4 Questions: What are the benefits of evaluating your programs? What are the challenges to evaluating your programs? What needs (besides $) do you have in order to plan/conduct evaluation? What resources do you have/use for evaluation?
Where does Evaluation fit?
Mission Goals Objectives Activities Outcomes Impact Program & Process Evaluation IMPLEMENTATION EvaluationEvaluation Program & Evaluation PLANNING OutcomeOutcome
Mission Provides the vision How this work makes a difference in the world Broadest scope
Goals (Ross & Mico, 1980; McKenzie & Smeltzer 2001) “ A future event toward which a committed endeavor is directed ” Simple & concise 2 basic components “ Who will be affected ” “ What will change as a result of the program ”
Objectives Specific steps that contribute to a goal. Often several objectives per goal. Good objectives are SMART: S S – specific M M – measurable A A – attainable R R – realistic T T – time-bound
Good Objectives Show… (McKenzie & Smeltzer 2001) What will change: Outcome that will be achieved When will it change: Conditions under which the outcomes will be observed How much change: Criteria for deciding whether the outcomes has been achieved Who will change: Target population
Activities Internal: administrative, etc. External: the services you provide to clients Based on your goals/objectives
Outcomes Changes that occur in people being served by your program Attribution: To the best extent possible, show that change is a result of your program (but note…causality is difficult) Standards are typically different for evaluation (vs. research) To assess, you need at least 2 time points (pre- and post-) and/or a comparison group
Impact The scope of the program’s effects, the duration of its outcomes and the extent of its influence on the broader context (for example, HIV incidence) Attribution: Can be more challenging to show causality, because looking for more diffuse effects Usually broad and long-term Typically not in the scope of program evaluation
Approaches to Evaluation
Why do we evaluate? To determine if objectives are being met To improve quality of the program To decide how to change content To identify the effects of the program
Process vs. Outcome Evaluation Process Demographics (Who’s being trained?) Reaction to content (“Smile Sheets”) Service units delivered Outcome Changes in knowledge/attitudes/beliefs Changes in behavior Impact on patients/clients
Process Evaluation can help us… Create better learning environment Improve presentation skills Show accountability Reflect the target populations Track service units
Outcome Evaluation can help us… Show the program’s effects Allow for comparisons over time Provide specific guide points for improving programs Show accountability
Planning Your Evaluation (1) Figure out your questions: What will this be used for? Consider your Resources Staffing Time Materials $$$ Choose Methods Quantitative: Surveys, pre/post tests, etc. Qualitative: Interviews, focus groups, etc.
Planning Your Evaluation (2) Select Indicator(s) Relevant Measurable Improvable Instrument/Tool Development Don’t reinvent the wheel! Analysis: Get answers to your questions Reporting: Formal & Informal
Examples