1. Overview of Evaluation ED. 883.721 Session 1: 01/28/10

2. Assignment 1 Handouts Review your problem and program for us Discussion Clear problem statements Defining terms Clear program or policy Scope and scale

3. “A program is a theory and an evaluation is its test.” (Rein, 1981 p141)

4. What is a program theory? Theory of action – why should the program work? Explains why the program does what it does and provides the rationale for expecting that doing so will achieve the desired results Explicit or implicit in all programs and policies

5. Why would we want to know the program’s theory of action? Develop knowledge of issues Aid to identify evaluation questions Framework for data collection Help to understand the data Interpret the evidence Make sound recommendations Better reporting Object of evaluation itself

6. Social programs are complex things Can be clearly defined In terms that are clear and reproducible Involves a mechanism that is well understood or visible or “fuzzy” Programs are not “Black Boxes” Y

7. Conceptualizing a Program Define the Boundaries of the Program Program Goals Official, Stated Goals As perceived by staff Distinct from objectives Problems? Assumptions

8. Important Components Program Goals and Objectives Can they be realistically be obtained? Program Functions, Components and Activities Linking Functions, Components and Activities How does everything fit together?

9. Finding out about a Program Review program documents Interview stakeholders Site visits and observations Literature review Corroborate it

11. Program Theory Target Population Program Outcomes

12. Impact Theory Assumptions about the change process actuated by the program Assumptions about the improved outcomes Target Population Program Outcomes

13. Impact Theory Causal – cause and effect relationship Diagrams of the chain of causes and effects Proximal versus Distal Outcomes

14. Service Utilization Theory How to reach targets Provision and sequence services How to conclude services Target Population Program Outcomes

15. Service Utilization Theory Focus on assumptions how and why recipients will become engaged in the program and follow through to desired outcomes Flow charts of paths through program

16. Organizational Theory Program resources, personnel, administration, organization Factors that make it possible to develop and maintain service delivery Target Population Program Outcomes

17. Organizational Theory/Plan Who is in charge of what? What resources are necessary at critical points? Charts preconditions for each critical point Inputs and activities

19. Inputs/Factors Positive and Negative Resources and Barriers Tangible

20. Activities Processes, tools, events, actions Products Curricula, materials Services Education, training Counseling Infrastructure Relationships, capacities

21. Outputs DIRECT results of program activities Speak to size and scope Data about activities Number of classes taught Participation rates Hours of services provided

22. Outcomes & Impacts Outcomes What is expected to result from activities Individual level Impacts Higher level changes that result from program activities

23. Problem Statements

25. Feb 11: Identifying Problems and Asking Researchable Questions RLF Chapters 3 & 4 Light, R., Singer, J., & Willet, J. (1999). What are your questions? In By design: planning research on higher education (pp. 12-40). Cambridge, MA: Harvard University Press. Kerlinger, F. (1986). Problems and hypotheses. In Foundations of behavioral research (pp. 15-25). New York: Harcourt Brace Jovanovich.