1. 1 Using Logic Models to Enhance Evaluation WESTAT Center to Improve Project Performance (CIPP) Office of Special Education Programs Amy A. Germuth, Ph.D. Compass Consulting Group Durham NC

2. 2 Today’s Schedule 12:30-3:30 Workshop 12:30-2:00 Workshop 2:00-2:15Break 2:15-3:30Workshop/Wrap-up Please feel free to ask questions at any time or take a break if you need it.

3. 3 What we Will Cover What a logic model is Why we develop logic models The components of a logic model Vocabulary associated with logic models What is an output versus an outcome Why satisfaction is (almost always) an output versus an outcome How to develop a logic model Simple logic model Complex logic model How logic models can guide formative and summative evaluation efforts Formative versus summative evaluation Connecting evaluation to the logic model

4. 4 Learning Goals 1. Understand what a logic model is and why you might want to develop one for your program. 2. Understand the components of a logic model. 3. How to develop a logic model for your program. 4. How to use your logic model to guide evaluation.

5. 5 What is a Logic Model? A depiction of a program showing what the program will do and what it is to accomplish. A series of “if-then” relationships that suggest (domino effect): If the program is implemented as intended (and designed)desired outcomes will be realized A systematic and visual way to depict the relationship that exists among the resources, activities, and desired results of a project or program. Thus, a completed logic model depicts how the project will work and is sometimes referred to as the program theory.

6. 6 Why are Logic Models used? To describe group work, team work, community- based collaboratives and other complex organizational processes. To guide clarity on sequencing. To guide planning, implementation, evaluation and communication. To help depict the theory of change associated with a program.

7. 7 Constructing Logic Models Examining program descriptions, missions, visions, plans, or extracting them from a narrative (useful if program in place) Forward mapping—starting with activities and asking “so what” to generate expected outputs and outcomes (useful if program in place) Reverse mapping—starting with outcomes and asking “how best to” identify the activity that is most likely to generate them (useful if developing a program) Importance of incorporating Evidence Based Practices

8. 8 Simple Logic Model Inputs → Outputs → Outcomes Example: Inputs → Outputs → Outcomes Books → New vocabulary words → Increased reading comprehension

9. 9 More Complex Logic Models

10. 10

11. 11 Another Logic Model

12. 12 Our Logic Model Framework - 1 Objectives/Goals Inputs→→ Activities→→ Outputs→→ Outcomes→→ Impacts Assumptions External Factors / Context

13. 13 The Visual… Need Goal(s)InputsActivitiesOutputsOutcomesImpacts Evidence-based practices are used to: 1)establish goals as effective and efficient ways to meet needs; 2)establish activities as effective and efficient ways to operationalize goals; 3)establish the direct outcomes likely to result from activities; and/or 4)establish that direct outcomes eventually will lead to impacts in some proportion of the target population.

14. 14 Our Logic Model Framework-2 Inputs = what the program needs to invest in order to provide activities Activities + Outputs = what the program does Outcomes + Impacts = who or what will change and how Needs reflected in Legislation = The legislation that provides resources for a program is grounded in identified or observed needs Assumptions + External Factors / Context = Factors that influence getting to outcomes

15. 15 Logic Model Definitions 1. NEEDS – Inspire legislation. 2. GOALS / OBJECTIVES– The primary changes that are necessary to serve the need. 3. INPUTS - Resources, contributions, investments (financial and other) that go into the program. 4. ACTIVITIES - What you do as part of the program. 5. OUTPUTS - Activities, services, events and products that reach people who participate or who are targeted. These are often discrete. 6. OUTCOMES - Results or measures of change for individuals, groups, communities, organizations, or systems. Can be short-term and long-term. 7. IMPACTS – Overall result of multiple outcomes acting together.

16. 16 Activity 1- Output or Outcome? Directions: Worksheet 1 contains multiple outputs and outcomes selected from a diverse range of policy, personnel, and project / program evaluations. 1. On your own, identify each as an Output or Outcome. 2. Next compare your results with those at your table, and be prepared to present your findings to the broader group.

17. 17 Developing a Logic Model Let’s start by developing a simple logic model (Inputs →Outputs →Outcomes), assuming the situation we are addressing is: Developing and operating a science lab. 1. What are possible inputs? 2. What are possible outputs? 3. What are possible outcomes?

18. 18 Inputs: Developing and Operating a Science Lab Funds to buy lab equipment Lab procedures curriculum Guide to lab procedures and safety rules Safety equipment

19. 19 Outputs: Developing and Operating a Science Lab Number of labs completed Number of different procedures completed Number of times each procedure done Number of safety rules learned Number of safety measures taken

20. 20 Outcomes: Developing and Operating a Science Lab Improved lab skills Increased understanding of safety needs Increased number of safety precautions taken

21. 21 Activity 2 – Another Simple Logic Model Situation: Family vacation. Let’s distinguish between inputs and activities on this one. Inputs – what we invest Activities – what we do Outputs – who we reach and how Outcomes – what results What would a potential logic model look like?

22. 22 Adding More Detail: Short- and Long-term Outcomes The USED Math-Science Partnership Program was developed to increase students’ achievement in math and science. The assumption behind this program is that by providing teachers professional development in math and science content and pedagogy, teachers would be better prepared to help their students attain greater achievements in these areas.

23. 23 MSP Program Logic Model Inputs: $$$, trainers, college professors, materials, technology Activities: Professional development in content and pedagogy, shared planning time, departmental meetings, book discussions Outputs: Attendance at PD, feedback on PD, # persons trained, average number of hours of training received Outcomes: ?????? Impact: Increased student achievement

24. 24 MSP Program Logic Model: Short- and Long-term Outcomes What are some short-term outcomes associated with teachers? What about with students? What are some long-term outcomes associated with teachers? What about with students?

25. 25 Additional Considerations Assumptions - the beliefs we have about the program, the people involved, the context in which the program is operating, and the way we think the program will work. Objectives –our goals / objectives for our program – these are based on our assumptions and guide our inputs and activities. External Factors / Context - the context or environment in which the program exists. This includes external factors that interact with and influence the program action (e.g., need, interest, economic factors, etc.). They may guide your assumptions and choices of inputs and activities, but also may be things that affect outcomes and impacts.

26. 26 Activity 3: Putting it all Together In groups, please review the scenario/ program assigned to you and identify the following: Assumptions External Factors / Context Objectives / Goals Inputs Activities Outputs Outcomes (short-term and long-term) Impacts The more detail / specific, the more useful your logic model will be, particularly for guiding your evaluation!

27. 27 Using Logic Models to Guide Evaluation Logic Models are extremely useful in guiding both formative and summative evaluation efforts. - Formative evaluation is undertaken to assess whether the program is proceeding as planned, the fidelity of implementation to program design, and the degree to which changes need to be made. - Summative evaluation is conducted to asses whether planned outcomes have been achieved and what impacts (intended and unattended) have occurred.

28. 28 Logic Models and Formative Evaluation Logic Models identify the key inputs, activities, and outputs associated with the program. This information in turn identifies data sources and key data that can be used to answer such questions as: 1. Are key inputs being utilized as expected? As designed? 2. How are activities being received by participants? 3. Are key outputs occurring at the desired level? 4. What do participants and stakeholders like about the current program? 5. What changes do they suggest should be made? What other questions might we ask? What other data sources can we consider? What data should we collect?

29. 29 Logic Models and Summative Evaluation Logic Models also identify key outcomes or measurable changes and impacts associated with the program. This information in turn identifies data sources and key data that can be used to answer such questions as: 1. What outcomes (expected and unexpected) have occurred? 2. What expected outcomes have not occurred? 3. Where is change the greatest? 4. What impacts (expected and unexpected) have occurred? 5. What expected impacts have not occurred? 6. To what degree have outcomes occurred? 7. What is the unique contribution of the program to the observed change? 8. What is the cost / benefit of these outcomes? What other questions might we ask? What other data sources can we consider? What data should we collect?

30. 30 Updating our Logic Model These questions, data sources, and data to be collected can easily be added to our logic model.

31. 31 Activity 4 – Adding Evaluation Components Using the same scenario/program from Activity 3, work in your group to identify potential formative and summative evaluation questions, and data / data sources needed to answer them. Be prepared to share these with the broader group.

32. 32 References (Web)  Kellogg Foundation Logic Model Development Guide http://www.wkkf.org  CDC Guide on Logic Model development http://www.ojp.usdoj.gov/BJA/evaluation/guide/docume nts/cdc-logic-model-development.pdf  University of Wisconsin Extension Program Logic Model Development Guide http://www.uwex.edu/ces/pdande/evaluation/evallogicm odel.html  InnoNetTool for Planning and Evaluation http://www.innonet.org/index.php?section_id=64&conte nt_id=185

33. 33 References (Texts) Logic Modeling Methods in Program Evaluation: Research Methods for the Social Sciences - J Frechtling Theory Driven Evaluations – H. Chen

34. 34 The End Questions?

35. 35 CIPP Hotline Information Email: CIPP@westat.com Toll-free: (888) 843 - 4101