Designing Effective Evaluation Strategies for Outreach Programs 6/30/2005 Designing Effective Evaluation Strategies for Outreach Programs Mariela GAbaroni Health Promotion Services Successful Grant Writing

Agenda What is evaluation? Importance of Evaluation 6/30/2005 Agenda What is evaluation? Importance of Evaluation How to design an effective evaluation strategy Data collection sources and methods How to justify conclusions based on evaluation results Successful Grant Writing

What is “evaluation”? Evaluation can be described as the systematic investigation of the merit, worth, or significance of any “object”

Program Evaluation defined 6/30/2005 Program Evaluation defined Evaluation is the systematic application of scientific methods to assess the design, implementation, improvement or outcomes of a program (Rossi & Freeman, 1993; Short, Hennessy, & Campbell, 1996). The term "program" may include any organized action such as media campaigns, service provision, educational services, public policies, research projects, etc. (Center for Disease Control and Prevention [CDC], 1999). Successful Grant Writing

Importance of Evaluation 6/30/2005 Importance of Evaluation Most important part of a program/service Provides data for objective review Allows for Quality Improvement Provides feedback about what was successful and what failed Measures impact the program is making Allows establishment of model programs and best practice Successful Grant Writing

Purpose of Program Evaluation 6/30/2005 Purpose of Program Evaluation Demonstrate program effectiveness Improve the implementation and effectiveness of programs Better manage limited resources Document program accomplishments Justify current program funding Support the need for increased levels of funding Documentation of program development help ensure successful replication Successful Grant Writing

Effective evaluation is not an "event" that occurs at the end of a project, but is an ongoing process which helps decision makers better understand the project; how it is impacting participants, partner agencies and the community; and how it is being influenced/impacted by both internal and external factors. W.K. Kellogg Foundation Evaluation Handbook, p. 3

6/30/2005 Successful Grant Writing

No evaluation is good unless… results are used to make a difference 6/30/2005 No evaluation is good unless… results are used to make a difference Successful Grant Writing

Types of Evaluation Formative (planning) Process Outcome Impact 6/30/2005 Types of Evaluation Formative (planning) Before implementation Is this program appropriate? Feasible? Process Is implementation going as planned? Outcome Was the program effective? Did we achieve the goals/objectives we set? Impact Did the program change behavior long-term? Successful Grant Writing

Formative (Planning) Evaluation 6/30/2005 Formative (Planning) Evaluation Why is the program needed? Identify target population Who needs to be involved in the planning? Identify key players/agencies What are the goals of the program? Identify goals from perspective of various key stakeholders/agencies What resources are necessary? Identify financial resources Identify non-financial resources What is the timeline? Determine timeline to program implementation Successful Grant Writing

Process Evaluation Who was the target audience? What has the program done? When and where did the program activities take place? What are the barriers/facilitators for implementation?

Outcome Evaluation Data – Types and Collection Quantitative Qualitative How to collect data for process evaluation: Interviews Focus groups Observation Questionnaires/Surveys/Pre/Post Tests Analysis of existing documents

Outcome Evaluation Did the program produce the desired effect on the target population? Why? Why not?

Impact Evaluation Data – Types and Collection What was the long-term result of the program The essence of program impact evaluation is comparison (i.e. multiple data collection points to compare). Assessment of change over time. Done through utilization of: Control group Comparison group Pre- and post-test The data collected is most often quantitative but may be qualitative given the nature of the program or project.

6/30/2005 What Method is Best? Select the type of evaluation instrument that will work best for the program/service. Consider: Utility: Who needs the info from this evaluation and what info do they need? Feasibility: How much money, time, and effort can we put into this? Propriety: Who needs to be involved in the evaluation to be ethical? Accuracy: What design will lead to accurate information? Successful Grant Writing

Evaluation Need not be Some evaluation is better than none 6/30/2005 Evaluation Need not be Expensive Complicated Time consuming Some evaluation is better than none Evaluation plan should be meaningful, related to goals and objectives, and be an honest examination of program Successful Grant Writing

Performance Indicator (quantitative or qualitative) 6/30/2005 Evaluation Flowchart Goal Stated Objectives Program/Service Outcome Performance Indicator (quantitative or qualitative) Successful Grant Writing

Program Planning Process 6/30/2005 Program Planning Process Evaluation is part of a larger program planning process. Planning Implementing Evaluating Use of Result Change program Discontinue program plan implement evaluate improve Successful Grant Writing

LOGIC MODEL connection

Designing an Effective Program Plan Describe the program Don’t jump into planning or evaluation without clarity on The big need your program is to address The key target group(s) who need to take action The kinds of actions they need to take (your intended outcomes or objectives) Activities needed to meet those outcomes Causal relationships between activities and outcomes

Investments or Inputs Time Money Partners Equipment Facilities Staff 6/30/2005 Investments or Inputs Time Money Partners Equipment Facilities Staff Successful Grant Writing

Outputs What We Do: Who We Reach Programs Workshops RAW Days 6/30/2005 Outputs What We Do: Programs Workshops RAW Days Cooking Demonstrations Brochures Who We Reach Target Population Customer Clients Patients Participants Students Successful Grant Writing

Outcomes Short Term Outcome (Medium-term) Impact (Long-term) 6/30/2005 Outcomes Short Term Increased knowledge Motivation Awareness Outcome (Medium-term) Change in behavior Practices Policies Impact (Long-term) Environment Change in conditions Social Economic Political Successful Grant Writing

Key Domains in Evaluation Focus Implementation (Process) Is program in place as intended? Effectiveness (Outcome) Is program achieving its intended short-, mid-, and/or long term effects/outcomes Efficiency How much “product” is produced for given level of inputs/resources? Causal Attribution Is progress on outcomes due to your program?

How much did you learn: What is evaluation? Why is evaluation important? When do you use formative evaluation? When would you use impact evaluation? When would you use outcome evalution? What things should you consider when planning a program?

Questions?