1. Dr. G. Johnson, www.researchdemystified.org1 Program Evaluation and the Logic Model Research Methods for Public Administrators Dr. Gail Johnson

2. Dr. G. Johnson, www.researchdemystified.org2 What to Evaluate? Projects: a single intervention in one location or a single project implemented in several locations. Programs: an intervention comprised of various activities or projects which are intended to contribute to a common goal. Organizations: multiple intervention programs delivered by an organization.

3. Dr. G. Johnson, www.researchdemystified.org3 When to Evaluate? Before program starts: To improve design During the implementation: To improve implementation Identify barriers to be removed Lessons learned about implementation

4. Dr. G. Johnson, www.researchdemystified.org4 When to Evaluate Mid-term evaluation Relevance, effectiveness, efficiency Lessons learned: management tool Impact evaluation Either at the end of the project or a few years after the program has been operating: assessing a mature program Can also look at: effectiveness, efficiency, early signs of impact and sustainability Lessons learned: future projects

5. Dr. G. Johnson, www.researchdemystified.org5 Why Is Evaluation Useful? Feedback Accountability Learning Improvement Results Testing underlying assumptions or theory Funding decisions

6. Dr. G. Johnson, www.researchdemystified.org6 Fear of Evaluation If evaluation is so useful, why do some people fear evaluation?

7. Dr. G. Johnson, www.researchdemystified.org7 Evaluation Questions Compliance/ Accountability Questions Did the promised activities actually take place as they were planned? “ How ” Questions What was the sequence or processes that led to successful (or not) outcomes Impact Questions Did the program achieve the desired results?

8. Dr. G. Johnson, www.researchdemystified.org8 Types of Evaluations Auditing: accounting for money Is the money being spent according to plan? Efficiency and effectiveness. Monitoring: measuring implementation and results Is the intervention producing the intended results? Process: measuring operations and service delivery Are there problems in service delivery?

9. Dr. G. Johnson, www.researchdemystified.org9 Types of Program Evaluations Feasibility evaluations Before the program begins Intended to improve program design Evaluability assessments Assesses potential usefulness of the evaluation Used to test out different strategies for conducting an evaluation What is doable given the situation?

10. Dr. G. Johnson, www.researchdemystified.org10 Evaluability Assessment Helps to define the actual objectives, implementation and management of a program. The actual objectives may differ from those initially planned. Determines the coherence of the program: are goals, activities, program infrastructure linked?

11. Dr. G. Johnson, www.researchdemystified.org11 Evaluability Assessment Key steps in the process : Interview key program staff to actual program mission, goals, objectives and activities. Site visits to observe and get a sense of what is going on. May include interviews with key stakeholders. Observe program delivery.

12. Dr. G. Johnson, www.researchdemystified.org12 Evaluability Assessment Reach agreement as to: Whether to conduct the evaluation. Scope and objectives of the evaluation. The decision could be to not conduct the evaluation.

13. Dr. G. Johnson, www.researchdemystified.org13 Evaluability Assessment: Challenges Key components of the program may not be well defined: Lack of agreement on program objectives. Lack clear, measurable indicators of performance and/or impact. Target group may not be clearly defined. The delivery system is poorly articulated.

14. Dr. G. Johnson, www.researchdemystified.org14 Types of Program Evaluations Formative evaluations During implementation Feedback about operations and processes Used to make mid-course corrections

15. Dr. G. Johnson, www.researchdemystified.org15 Definition: Performance Monitoring  Performance monitoring:  the continuous process of collecting and analyzing data to compare how well a project, program or policy is being implemented against expected results.  Traditional:  focus on inputs, activities and outputs.

16. Dr. G. Johnson, www.researchdemystified.org16 Types of Evaluation:Monitoring On-going review: On-time On-budget On-target Linked with on-going management Measured against established baselines Indicators of progress toward targets

17. Dr. G. Johnson, www.researchdemystified.org17 Types of Program Evaluations Summative Evaluations At the end of the program or after the program has been running long enough to achieve its goals (with “mature” programs) Identify lessons learned Other issues: unintended outcomes, program sustainability, program efficiency, costs and benefits Sometimes called impact evaluations and ex-post evaluations

18. Dr. G. Johnson, www.researchdemystified.org18 Program Evaluation Summative Evaluation Question: Do Public Programs Work? Implied cause-effect relationship Did the program cause a desired outcome?  Performance-based: Focus on outcomes, results, impacts, goal achievement.

19. Dr. G. Johnson, www.researchdemystified.org19 Differences Formative Evaluations Project Monitoring Early Years of Implementation Key Question: Are we doing things right? –Have we hired the right people with the right skills? –Have we marketed the program effectively? –Have we met our strategic objectives? –Have we spent our money according to our plan?

20. Dr. G. Johnson, www.researchdemystified.org20 Differences Summative Evaluations Measuring Results or Impacts A longer time before results or impacts are visible Key Question: Are we doing the right thing? This gets back to the theory or underlying assumptions of the program: –We can do an excellent job at training people but if the problem is not about the larger structural economic issues, a training program, no matter how well implemented, may show little result.

21. Dr. G. Johnson, www.researchdemystified.org21 Working with Models Visualize a program in context Systems approach, within an environment Identify the relationships between various components Identify cause and effect Identify key assumptions

22. Dr. G. Johnson, www.researchdemystified.org22 Models: Cause and Effect: Did the program cause something to happen? Education Employment

23. Dr. G. Johnson, www.researchdemystified.org23 Reduced Poverty Improved Quality of Life Increased Income Job Training Unemployed Hierarchy of Objectives

24. Dr. G. Johnson, www.researchdemystified.org24 Logic Models The focus is on results or impacts rather than inputs and activities We are not training people just for the sake of training people We believe if we train the chronically unemployed, then there quality of life will be improved and poverty will decrease. Our goal is to reduce poverty Also called Program Outcome Model or Measuring for Results

25. Dr. G. Johnson, www.researchdemystified.org25 Logic Model Inputs Activities Outputs Outcomes Impact

26. Dr. G. Johnson, www.researchdemystified.org26 Elements of the Logic Model Inputs: what resources are used University inputs: budget, number of faculty, number of staff, number of buildings, number of classrooms Activities: what the program does University activities: teaching, research, and service

27. Dr. G. Johnson, www.researchdemystified.org27 Elements of the Logic Model Outputs: the services or products produced University outputs: number of students that graduate, number of articles and books published by faculty Outcomes: what happened: immediate results Graduates are sought after, get good jobs, active alumni who donate big bucks Faculty well-known, obtain big grants, enhance rating of university

28. Dr. G. Johnson, www.researchdemystified.org28 Elements of the Logic Model Impacts: the “so what.” Larger, long term results, usually tied to program goals. A more informed and engaged citizenry, preserves democratic institutions, future leaders. Faculty research contributes to knowledge.

29. Dr. G. Johnson, www.researchdemystified.org29 Logic Model Logical Connections: Inputs to do activities Activities lead to outputs Outputs lead to one or more outcomes Outcomes lead to impacts

30. Dr. G. Johnson, www.researchdemystified.org30 Logic Model: Training Program Inputs Activities Outputs Resources money staff Supplies mentors Training Programs Dress for success coaching Interview coaching Resume assistance Products Number of graduates per training session % graduate rate Outcomes Benefits changes Increased skills % Obtain jobs % Obtain high paying, quality jobs Increased self-esteem Impacts Goals Increased income Self- sufficiency Family stability Reduction in poverty

31. Dr. G. Johnson, www.researchdemystified.org31 Takeaway Lesson Program evaluation takes many forms but all follow the same research planning process. Evaluation of programs have shifted from reporting inputs and activities to attempting to measure results: the difference the program actually made.

32. Dr. G. Johnson, www.researchdemystified.org32 Takeaway Lesson Measuring results is harder than it appears. Program goals and objectives may be fuzzy. Sites may vary in how they have implemented the program. It takes money to collect and analyze data. Results may not be observable for many years. The operating environment may make it hard to see results.

33. Dr. G. Johnson, www.researchdemystified.org33 Takeaway Lesson Measuring results is harder than it appears. But there is much that can be learned from engaging in this process and doing the best job possible. Remember: do not quickly conclude a program does not work just because you cannot measure the result. The research tools available may not be up to the job.

34. Dr. G. Johnson, www.researchdemystified.org34 Creative Commons This powerpoint is meant to be used and shared with attribution Please provide feedback If you make changes, please share freely and send me a copy of changes: Johnsong62@gmail.com Visit www.creativecommons.org for more informationwww.creativecommons.org