Selecting the Right Evaluation Method

Objectives Why should we evaluate? Which activities should we evaluate? When should we evaluate? How should we evaluate (methods)?

Why Do We Evaluate? Design a program. (or item; e.g., curriculum, poster) Baseline, formative Determine if objectives were met. (best practices) Outcome, process, impact Build an evidence base for ACSM Re-design or refine a program. (lessons learned) Process, outcome

Which Activities Should We Evaluate? Is this a new activity? Is this an activity that may be expanded? Have we invested a lot in this activity? Is something unexpected happening? Does the funder want an evaluation? IF YES ….. EVALUATE!

Case Example Is this a priority program for evaluation? Why or why not? Which objectives or activities are most important to evaluate?

DesignImplementationEnd BASELINE FORMATIVE PROCESS OUTCOME IMPACT When do we evaluate?

FORMATIVE EVALUATION PROCESS EVALUATION OUTCOME EVALUATION IMPACT EVALUATION ACSM ASSESSMENT KAP Surveys Key Informant Interviews Media Scans Focus Groups Analysis of Program Data Secondary Analysis Simple Surveys Client Exit Interviews

When Can We Use Quantitative Methods? BASELINEFORMATIVEPROCESSOUTCOMEIMPACT Routine Surveillance Analysis of NTP Data Analysis of Project Data Simple Surveys Population-Based SurveysPopulation- based Surveys

Use a QUANTITATIVE Method to: Count things or people. Measure your program against a benchmark. Monitor trends over time. Statistically correlate factors: Is there a connection between variables?

When Can We Use Qualitative Methods? BASELINEFORMATIVEPROCESSOUTCOMEIMPACT Focus Groups Exit Interviews Media Scans Informant Interviews Exit Interviews

Use a QUALITATIVE Method to: Find out overall concerns, opinions. Gather initial information or sense of direction. Get details of/reasons for any problems. Test communication materials.

Combine BOTH for Best Results: Identify key issues with a focus group or interviews. Analyze NTP data to look for correlates or to obtain a baseline measurement. Use surveys to measure before and after results. Conduct interviews to assess how your process is going. Link your final outcome data to surveillance or NTP data.

Which Method and When? BASELINEFORMATIVEPROCESSOUTCOMEIMPACT Analysis of Routine Surveillance Focus Groups and In-Depth Interviews KAP Exit Interview Simple Survey Media Scan

Factors to Consider Methods Cost, Generalizability, Rigor Scale MY ACTIVITY OR PROJECT Small, local Low Key informant interviews Focus groups Simple surveys Large, multi-region Medium KAP surveys Exit interviews NTP/Program data analysis NationalHigh Experimental design (research) KAP surveys

What is the purpose of the evaluation? I am at the end of a three-year, district-level ACSM project. Did we produce the expected outputs and short-term outcomes? Analysis of project data, focus group, or KAP survey? I am designing a media campaign to inform my workers that TB treatment is free. What is the best way to get this message to them? What do they already know? Routine surveillance, KAP survey, or focus group?

What is the most rigorous method to answer the evaluation question? The NTP wants an impact evaluation of ACSM because these new strategies have never been used at the national level. Experimental design, secondary data analysis, exit interviews? I am looking for a successful ACSM model that I can replicate in my district. Case study, KAP survey, experimental design? We trained 50 DOTS nurses on interpersonal communication and counseling skills and we want to know if they are using them correctly. Time location survey, exit interviews, secondary data analysis?

How quickly do you need the results? The NTP just announced funding for ACSM to reduce default. I have to design an activity in six weeks! I need to know why defaults are high in my district. KAP survey, simple survey, analyze surveillance data? We have Global Fund money to implement a whole “package” of ACSM interventions to improve case detection over five years. We really need to know if the activities are effective! Research study, informant interviews, KAP survey?

What financial and human resources do I have? I am SO lucky! My donor wants a formal evaluation of ACSM and gave us a lot of money for it. AND we have partners at the School of Public Health to help us! KAP survey, routine surveillance, research study? My donor thinks that radio/TV/newspapers are the best way to reach our population and is giving me a lot of money to improve coverage of TB issues. But they want all of the money to go toward implementation. How can I know if it is effective? KAP survey, media scan, focus group?

Evaluation Template ACSM Objective: ACSM Activity: Evaluation Question Evaluation Category/Method When or How Often Sampling Resources Needed Plan for the Results