Dancing Your Way through Program Evaluation with the CDC 6-Step Evaluation Framework MaryCatherine Jones, MPH, Public Health Consultant, CVH Team Zarina Fershteyn, MPH, Director of Program Evaluation, NACDD

Learning Objectives To identify the steps associated with the CDC Framework for Evaluation To discuss the intersection between logic models and the framework To engage peers in discussing common roadblocks and solutions in evaluating chronic disease and health promotion programs

Course Overview What we will accomplish

Getting Started with Evaluation Common types of evaluation in public health Implementation Effectiveness Efficiency Cost Effectiveness Attribution Others

What is the value in evaluation? Why do we evaluate public health programs? Monitor progress toward goals Determine whether program is producing desired progress on outcomes Permit comparisons across groups Justify need for funding/support Improve program quality Ensure effective allocation of resources

What is the CDC 6 Step Evaluation Framework? Program evaluation is 1 of 10 essential public health services. CDC published this framework in 1999 to help public health programs produce results that get used. Key concepts behind the framework include knowing your audience for your evaluation results and considering their needs in developing your evaluation plan. In other words: Who needs the evaluation results (utility)? What evaluation activities are realistic given our resources and opportunities? (feasibility) Are the key people affected by the program involved in its evaluation (propriety) Will the evaluation results be valid and reliable—can our users TRUST them (accuracy)?

Your Evaluation Team An evaluation team ensures that you have a variety of skills and perspectives that will make your evaluation useable and ensure that your results drive action. Evaluation team members can include: Program staff Partners Funders Policymakers Community members Subject matter experts Others

Your Lead Evaluator Able to work with a variety of stakeholders Able to balance potentially competing needs of stakeholders Able to prioritize and deliver in the face of limited resources Able to innovate

Why Are Stakeholders Important? Who are typical stakeholders? -Those involved in program operations -Those served/affected by the program -The intended users of the evaluation findings—able to take action on the results Stakeholders can help (or hinder) an evaluation before it is conducted, while it is being conducted, and after the results are collected and ready for use. Because so many public health efforts are complex and because public health agencies may be several layers removed from frontline implementation, stakeholders take on particular importance in ensuring that the right evaluation questions are identified and that evaluation results will be used to make a difference. Stakeholders are much more likely to support the evaluation and act on the results and recommendations if they are involved in the evaluation process. Conversely, without stakeholder support, your evaluation may be ignored, criticized, resisted, or even sabotaged. In addition, to be proper/ethical and accurate, you need to include those who participate in the program and are affected by the program or its evaluation. Shortly, we’ll be sharing a worksheet with you to practice identifying stakeholders and understanding their investment in your evaluation results. You will use the worksheet today during our small group activity, but we also encourage you to take it home and try it with your program.

Stakeholder Roles Evaluation Team (planning and implementing the evaluation) Dissemination of evaluation findings Contributing data Analyzing and drawing conclusions from data

Considerations for Stakeholder Engagement Standard Questions Utility Who will use these results? Feasibility How much time and effort can we expect from the stakeholders? How much time/effort do we have to engage them? Propriety Which stakeholders can help our evaluation to be ethical? Accuracy How broadly do we need to engage stakeholders to paint an accurate picture of the program?

Describing the Program

Considerations for Your Description Need Target Population Outcomes Activities Outputs Resources/Inputs Relationship of activities and outcomes Stage of development Context/environmental factors

Sample Logic Model Format Investment Outcomes Inputs Activities Outputs Short-Term Intermediate Term Long Term

Considerations for Your Program Description Standard Questions Utility Is the level of detail appropriate for the intended use? Feasibility Are the activities and at least some of the outcomes within the program’s control? Propriety Does the description include enough detail to examine strengths and weaknesses of the program? Accuracy Is the program description comprehensive? Are likely influences on the program identified?

Case Study and Small Group Activity ?

Group Discussion

Focusing the Evaluation Design Types of Evaluation Implementation/Process Effectiveness/Outcome

Focusing the Evaluation Design Considerations for Design WHO HOW WHAT (we can reasonably evaluate)

Big Details for Your Evaluation Plan Availability of data Timing of data collection Data analysis (who, when, how) Types of conclusions we can make By using a logic model in our evaluation planning, we can focus on these activities, outputs, and outcomes to develop measurements

Case Study (Continued) ? With your team, complete Worksheets 3A and 3B You have 15 minutes for this activity

Overcoming Trouble Spots Umm…Sorry, this made sense until I started explaining it. What are areas that could potentially be tricky for an evaluator to navigate? (list all options) Group vote: Select 2 of the areas identified as trouble spots Group discussion on how to prevent, manage, or otherwise address these.

Summary Worksheets adapted from U.S. Department of Health and Human Services Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2011. Visit the self-study guide for worksheets related to the other Steps!

Our Challenge to You

Our contact information MaryCatherine Jones, mcjones@chronicdisease.org Zarina Fershteyn, zfershteyn@chronicdisease.org