Everyday Program Evaluation Sheena Cretella MSPH SC DHEC Diabetes Division Program Evaluator 1

Today’s Objectives Planning Programs – Logic Models Evaluation – Standards of Evaluation – Steps to Evaluation – Indicators – Moderators – Success Stories Data – Types – Collection – Management 2

PLAN Like Your Program Depends On It! A Logic Model is like a road map Plan your route: START where you are now START where you are now FINISH where you intend to be after the program FINISH where you intend to be after the program 3

Logic Model Pieces Inputs and Resources What do you have? Activities What do you plan to do? Outputs What targets do you have for your activities? Outcomes What changes resulted from your activities? 4

Logic Model Resources Grant Funding Volunteers Church Facilities Activities Construct Walking Trail Conduct Sessions Create Community Garden Outputs Number of Miles Walked Number of Attendees Number of Seed Hills Short Term Outcomes Improved Attitude of Physical Activity Increase in knowledge of Healthy Behaviors Obtain Horticulture Skills Intermediate Outcomes Creation of an environment supporting healthy behaviors Increase in Physical Activity Increase in Fruit and Vegetable Consumption Long Term Outcomes Healthier Community Reduce the Risk of Pre- diabetes and Diabetes and Prevent Complications of Diabetes 5

Logic Model Resources Grant Funding Volunteers Church Facilities Activities Construct Walking Trail Conduct Sessions Community Garden Created Outputs Number of Miles Walked Number of Attendees Number of Seed Hills Short Term Outcomes Improved Attitude of Physical Activity Increase in knowledge of Healthy Behaviors Obtain Horticulture Skills Intermediate Outcomes Creation of an environment supporting healthy behaviors Increase in Physical Activity Increase in Fruit and Vegetable Consumption Long Term Outcomes Healthier community Reduce the Risk of Pre- diabetes and Diabetes and Prevent Complications of Diabetes 6

Program Planning 7

STOP: Check List for Program Planning  Do you and the stakeholders know what is going to be done, with whom, to whom/what?  Is the goal statement outcome oriented?  Are the expectations of stakeholders known to all?  Can the pieces of the logic model be measured?  Are there missing pieces or “gaps” in the logic model/plan?  Is the impact of the program important? Is there evidence? 8

What is Evaluation? 9 “Evaluation is the systematic investigation of merit, worth, or significance of any ‘object.’” - Michael Scriven

Keep In Mind Key Questions Grantees Need to Answer About their Programs: – What role, if any, did my program play in the results? – What role, if any, did the moderators play? – Were there any unintended outcomes? – What will happen if I do not do something? 10

11 Utility Will the evaluation provide relevant information in a timely manner? Feasibility Are the planned evaluation activities realistic given the time, resources, and expertise at hand? Propriety Does the evaluation protect the rights of individuals and protect the welfare of those involved? Accuracy Will the evaluation produce findings that are valuable and reliable? Standards of Evaluation

The Steps to Good Program Evaluation 1 Engage Stakeholders 2 Describe the Program 3 Focus the Evaluation 4 Gather Credible Evidence 5 Justify Conclusions 6 Use Lessons Learned 12

Moderators PoliticalEconomical SocialTechnological 13

Logic Model Resources Grant Funding Volunteers Church Facilities Activities Construct Walking Trail Conduct Sessions Community Garden Created Outputs Number of Miles Walked Number of Attendees Number of Seed Hills Short Term Outcomes Improved Attitude of Physical Activity Increase in knowledge of Healthy Behaviors Obtain Horticulture Skills Intermediate Outcomes Creation of an environment supporting healthy behaviors Increase in Physical Activity Increase in Fruit and Vegetable Consumption Long Term Outcomes Healthier community Reduce the Risk of Pre- diabetes and Diabetes and Prevent Complications of Diabetes 14 Moderators: Weather (drought, heat advisory, snow), safety (snakes/dogs), cost of gas (less volunteers/low attendance), neighborhood safety, inexperienced personnel, local habitat policies.

Measures Each activity and outcome should have a measure. A good measure is: Specific Observable Measurable 15 When choosing measures consider the following: Quality of data Quantity of data Logistics Sources of data collection Primary vs. Secondary

16 Data Management

Data Sources Primary Collecting new data Data is from the people you work with Common forms: – Group discussions – Observation – Document review: logs, journals, meeting minutes, sign-in sheets, etc. – Surveys: telephone, , personal, etc. Secondary Existing data sources Before using Secondary data ensure that they will meet the evaluation needs Large ongoing surveillance systems Routinely collected Not flexible 17

Data Sources Program participants and non-participants Key members Program staff Who would you interview? Meetings Special events / activities What might you observe? Meeting minutes Strategic plans Registration form Photos Which Documents might you review? 18

Success Story Catchy Action verb Title Importance of the problem Evidence Issue Describe program Partner Support Program Point out the most important measured outcomes/benefits of the program Impact 19

Success Story Content – Contact Information – Photos and Logos – Quotes Format – Brief – One Page – Bullets Possible Inclusions: – Testimonials – Promising practices – Lessons Learned – Partner Success – New Partners 20 Make it your own!

21 Issue Contact Impact Title Program Photo

Types of data 22 Make sure you are choosing the right kind of data for your specific evaluation questions. Qualitative Quantitative – Continuous – Categorical Ordinal Nominal

Data Management Physical Records – De-identify records whenever possible – Store records in a safe locked place – Shred personal data upon completion of evaluation Electronic Records – Use unique identification numbers whenever possible – Use a password on your computer – Erase (not delete) personal data upon completion of evaluation. – Back up data. 23

Data Management Id#AgeCountyVariable 3Variable Richland Fairfield Orangeburg Lexington 24 Id#AgeCountyVariable 3Variable Richland Fairfield Orangeburg Lexington Id#AgeCountyVariable 3Variable Richland Fairfield Orangeburg Lexington

Everyday Program Evaluation Thank You! Sheena Cretella MSPH SC DHEC Diabetes Division Program Evaluator