1. Logic Models as Tools for Developing Performance Measures Presented by JoAnn A. Smith, MPH Community Health Administration Grants Monitoring and Program Evaluation Division Data Analysis and Program Evaluation Unit August 26, 2010

2. Presentation Goals  Provide an overview of the development of logic models, evaluation questions and performance measures  Understand how logic models, measurement and evaluation are valuable tools that can be used to improve program performance, communicate performance and programs to funders and stakeholders.

3. Key Questions that should be asked about programs  What am I doing, with whom, to whom/what?  How well am I doing it?  Is anybody (anything) better off?  Short-term  Long-term --------------------------------------------------------------------------  What role, if any, did my program play in the results?  What role, if any, did the context play?  Were there any unintended outcomes, if so why?

4. PERFORMANCE MANAGEMENT Performance management includes activities to ensure that goals are consistently being met in an effective and efficient manner. Performance management tools include logic models, performance measurement and program evaluation. Logic Model Tool/framework that helps identify the program/project resources, activities, outputs customers, and outcomes. Performance Measurement Helps you understand what level of performance is achieved by the program/project. Program Evaluation Helps you understand and explain why you’re seeing the program/project results.

5. Logic models (Understanding your program in preparation for performance and evaluation)

6. Logic Model Defined Basically, a logic model is a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or outcomes (results) you hope to achieve. Source: Adapted from the W.K. Kellogg Foundation Logic Model Development GuideW.K. Kellogg Foundation Logic Model Development Guide From www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf. Accessed on July 29, 2009

7. What are logic models used for Staff and managers can use logic models to…  Develop program/project design  Identify and develop performance measures for their program/project  Support strategic planning  Communicate the priorities of the program/project

8. Putting it all together Examples of Logic Models

9. Simply Put…A Logic Model is  A graphic approach to strategic planning;  A series of connections (“If - Then”) that link problems and/or needs with the actions taken to achieve goals;  Serves as a “roadmap” identifying what a program expects and “how” it will be achieved. Use these resources… For these activities… To produce these outputs… So the target audience can change their ways… Which leads to these outcomes… Leading to these results!

10. Inter- mediate Changes in behavior, practice or decisions. Behavior Inter- mediate Changes in behavior, practice or decisions. Behavior Target Pop User of the products/ services. Target audience the program is designed to reach. Target Pop User of the products/ services. Target audience the program is designed to reach. Activities Things you do– activities you plan to conduct in your program. Activities Things you do– activities you plan to conduct in your program. Outputs Product or service delivery/ implementation targets you aim to produce. Outputs Product or service delivery/ implementation targets you aim to produce. Resources/ Inputs: Programmatic investments available to support the program. Resources/ Inputs: Programmatic investments available to support the program. Short-term Changes in learning, knowledge, skills, understanding Attitudes Short-term Changes in learning, knowledge, skills, understanding Attitudes Long- term Change in condition. Condition Long- term Change in condition. Condition External Influences (Sphere of Influence) Factors outside of your control (positive or negative) that may influence the outcome and impact of your program/project. External Influences (Sphere of Influence) Factors outside of your control (positive or negative) that may influence the outcome and impact of your program/project. Outcomes WHYHOW PROGRAM RESULTS FROM PROGRAM Logic Model Elements What the program actually does Who/what will change?

11. Attend this training session Notes and learning based on attending this presentation Increased understanding of logic models and use in proposal, grant and/ or program development ACTIVITIESOUTPUTSOUTCOMES Example- Simple Logic Model Need: To learn about logic models Target Population: Participants Goal: To understand how logic models are constructed and used. Participants Training Session INPUTS External Factors: Larger conference room is available so more staff can attend or staff may be unable to attend due to schedule conflicts.

12. Case Study Wonderland County’s Teen Mother Prenatal Care Program

13. Wonderland County’s Teen Mother Prenatal Care Program The Problem… In Wonderland County, teen mothers are not delivering healthy babies. The County Health Department identified a lack of prenatal care among this population, including other risk factors. To address this issue, Wonderland County’s City Council released a competitive Request For Proposals (RFP) to fund programs that seek to improve birth outcomes among the teens residing in Wonderland County. In response to this RFP… The Orange Blossom Health Group plans to implement The Teen Mother Prenatal Care Program, a nationally certified prenatal care education program, that was cited by the CDC as a Best Practice program. The group is preparing a logic model as part of their grant application. The intent of this program is to educate teen mothers on the importance of proper prenatal care with the ultimate goal of delivering healthy babies.

14. Funding Staff Educational Materials Space Pregnant Teens Wonderland County’s Teen Mother Prenatal Care Program Logic Model Teens Recruited into program Prenatal health classes held Target POP Pregnant Teens 13-18 yrs Teens are knowledgeable of of prenatal guidelines Teens follow proper prenatal care guidelines Teen mothers deliver healthy babies InputsActivities Outcomes Outputs #of Pregnant teens attend Program Short term Intermediate Long Term Goal: To educate at least X% of the pregnant teen population attending Wonderland County Public High Schools about the importance of Prenatal care and delivering healthy babies. Change in knowledge Change in behavior Change in condition

15. Alice in Wonderland has a dilemma So she seeks advice from her friend the Cheshire Cat….

16. Lewis Carroll 1872 Through the Looking Glass Alice’s Dilemma Alice: I have developed a logic model for my grant funded program but I want to know more about it, is it working, should I change anything etc. Cat: That depends on what you need to measure. Alice: I don’t know what I need to measure! I just want to be able to show my funders that my program works. Cat: Then it doesn’t matter what you measure!! Alice: Well what do I need to do? Please help! Cat: YOU need to develop performance measures!

17. The Logic Model as a Tool for Developing Performance Measures

18. Building upon the logic model – Preparing for Performance Measurement and Evaluation Use the logic model to:  Develop performance measures for your program or project  Help identify which aspects of a program/project to evaluate

19. Performance Measurement The ongoing monitoring and reporting of program process and accomplishments, using pre-selected performance measures.

20. Example Let’s Revisit Wonderland County’s Logic Model

21. Fully Detailed Logic Model With Evaluation Questions and Indicators Staff Education Materials Space Funding Wonderland County Teen Mother Prenatal Care Program Logic Model Prenatal health classes held Pregnant Teens are recruited into the program Pregnant teens Teens are knowledgeable of prenatal health guidelines Teens follow proper Prenatal health guidelines Teen mothers deliver healthy babies Performance Measures $ used #staff trained # of sites # of teens recruited- # of sessions held- #/% of teens completing the program # % demonstrating Increased knowledge skills # % demonstrating changes # % increase delivering healthy babies Inputs Activities Outcomes Outputs #of Teens who attend Program # of sessions Held # of teens completing The program Short term Mid Term Long Term EVALUATION QUESTIONS What amount of time and $ invested? How many sessions were held? What was the quality of the sessions? To what extent did knowledge & skills increase? Why? To what extent did behaviors (proper PNC & nutrition) change? Why? To what extent are teens delivering healthy babies?

22. Measures Across the Logic Model Spectrum AreaDefinitionExample Measure Resources/ Inputs Measure of resources consumed by the organization. Amount of funds, #of FTE; materials; equipment etc. ActivitiesMeasure of products and services provided as a direct result of program activities. # of technical assistance requests responded to; # of products produced/delivered Customer Reached Measure of target population receiving outputs. % of target population trained; # of target population receiving technical assistance Customer Satisfaction Measure of satisfaction with outputs % of customers satisfied with training; % of customers “very satisfied” with assistance received OutcomesAccomplishment of program goals and objectives (short-term and intermediate outcomes, long term outcomes –impacts) % change in knowledge;% change in behavior;% increase/decrease in a condition

23. Good Performance Measures  Provide a way to see if our strategy is working  Allow measurement of accomplishments (qualitative), not just of the work that is performed  Provide a common language for communication  Are explicitly defined in terms of unit of measure, collection frequency, data quality, expected value (targets), and thresholds  Are valid, to ensure measurement of the right things  Are verifiable, to ensure data collection accuracy

24. Program Evaluation  A systematic study that uses measurement and analysis to answer specific questions about how well a program is working to achieve its outcomes and why.

25. Performance Measures vs. Evaluation Performance Measurement  Ongoing monitoring and reporting of accomplishments.  Examines achievement of program objectives.  Describes program achievements in terms of outputs, outcomes in a given time against a pre-established goal.  Early warning to management. Program Evaluation  In-depth, systematic study conducted periodically or on ad- hoc basis.  Examines broader range of information on program performance than is feasible to monitor on an on-going basis.  Explains why the results occurred.  Longer term review of effectiveness.

26. Relationship between Measurement and Evaluation  Performance measurement data helps provide the information needed to conduct the evaluation and assess program performance.  Lack of performance measurement data is a major obstacle to conducting an evaluation

27. Finally, What can Measurement and Evaluation do for your program?  Increase certainly that program goals & objectives are being met  Identify what works well, what does not and why  Identify program areas that need improvement  Determine if allocated resources are yielding the greatest benefit

28. A word of Caution Regarding Data Source Selection  Not all data is good data  Be selective and choosy  Use reputable sources

29. Logic Models and Performance Measures Recap  Using a logic model as part of the evaluation process helps to organize and map the program with outputs and outcomes  Logic models can assist in the development of Performance Measures  Performance Measures help measure project performance  Performance measures help justify what’s working and areas for improvement and can lead to program evaluation