Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006

Overview  Warm-up  Where does Evaluation fit?  Approaches to Evaluation  Examples  Q&A: Evaluating Your Programs

Warm-Up

Who here is … Evaluator? Service Provider? Administrator?

4 Questions:  What are the benefits of evaluating your programs?  What are the challenges to evaluating your programs?  What needs (besides $) do you have in order to plan/conduct evaluation?  What resources do you have/use for evaluation?

Where does Evaluation fit?

Mission Goals Objectives Activities Outcomes Impact Program & Process Evaluation IMPLEMENTATION EvaluationEvaluation Program & Evaluation PLANNING OutcomeOutcome

Mission  Provides the vision  How this work makes a difference in the world  Broadest scope

Goals (Ross & Mico, 1980; McKenzie & Smeltzer 2001)  “ A future event toward which a committed endeavor is directed ”  Simple & concise  2 basic components “ Who will be affected ” “ What will change as a result of the program ”

Objectives  Specific steps that contribute to a goal. Often several objectives per goal.  Good objectives are SMART: S S – specific M M – measurable A A – attainable R R – realistic T T – time-bound

Good Objectives Show… (McKenzie & Smeltzer 2001)  What will change: Outcome that will be achieved  When will it change: Conditions under which the outcomes will be observed  How much change: Criteria for deciding whether the outcomes has been achieved  Who will change: Target population

Activities  Internal: administrative, etc.  External: the services you provide to clients  Based on your goals/objectives

Outcomes  Changes that occur in people being served by your program  Attribution: To the best extent possible, show that change is a result of your program (but note…causality is difficult)  Standards are typically different for evaluation (vs. research)  To assess, you need at least 2 time points (pre- and post-) and/or a comparison group

Impact  The scope of the program’s effects, the duration of its outcomes and the extent of its influence on the broader context (for example, HIV incidence)  Attribution: Can be more challenging to show causality, because looking for more diffuse effects  Usually broad and long-term  Typically not in the scope of program evaluation

Approaches to Evaluation

Why do we evaluate?  To determine if objectives are being met  To improve quality of the program  To decide how to change content  To identify the effects of the program

Process vs. Outcome Evaluation  Process Demographics (Who’s being trained?) Reaction to content (“Smile Sheets”) Service units delivered  Outcome Changes in knowledge/attitudes/beliefs Changes in behavior Impact on patients/clients

Process Evaluation can help us…  Create better learning environment  Improve presentation skills  Show accountability  Reflect the target populations  Track service units

Outcome Evaluation can help us…  Show the program’s effects  Allow for comparisons over time  Provide specific guide points for improving programs  Show accountability

Planning Your Evaluation (1)  Figure out your questions: What will this be used for?  Consider your Resources Staffing Time Materials $$$  Choose Methods Quantitative: Surveys, pre/post tests, etc. Qualitative: Interviews, focus groups, etc.

Planning Your Evaluation (2)  Select Indicator(s) Relevant Measurable Improvable  Instrument/Tool Development Don’t reinvent the wheel!  Analysis: Get answers to your questions  Reporting: Formal & Informal

Examples