Overview of Evaluation ED Session 1: 01/28/10

Assignment 1 Handouts Review your problem and program for us Discussion Clear problem statements Defining terms Clear program or policy Scope and scale

“A program is a theory and an evaluation is its test.” (Rein, 1981 p141)

What is a program theory? Theory of action – why should the program work? Explains why the program does what it does and provides the rationale for expecting that doing so will achieve the desired results Explicit or implicit in all programs and policies

Why would we want to know the program’s theory of action? Develop knowledge of issues Aid to identify evaluation questions Framework for data collection Help to understand the data Interpret the evidence Make sound recommendations Better reporting Object of evaluation itself

Social programs are complex things Can be clearly defined In terms that are clear and reproducible Involves a mechanism that is well understood or visible or “fuzzy” Programs are not “Black Boxes” Y

Conceptualizing a Program Define the Boundaries of the Program Program Goals Official, Stated Goals As perceived by staff Distinct from objectives Problems? Assumptions

Important Components Program Goals and Objectives Can they be realistically be obtained? Program Functions, Components and Activities Linking Functions, Components and Activities How does everything fit together?

Finding out about a Program Review program documents Interview stakeholders Site visits and observations Literature review Corroborate it

Program Theory Target Population Program Outcomes

Impact Theory Assumptions about the change process actuated by the program Assumptions about the improved outcomes Target Population Program Outcomes

Impact Theory Causal – cause and effect relationship Diagrams of the chain of causes and effects Proximal versus Distal Outcomes

Service Utilization Theory How to reach targets Provision and sequence services How to conclude services Target Population Program Outcomes

Service Utilization Theory Focus on assumptions how and why recipients will become engaged in the program and follow through to desired outcomes Flow charts of paths through program

Organizational Theory Program resources, personnel, administration, organization Factors that make it possible to develop and maintain service delivery Target Population Program Outcomes

Organizational Theory/Plan Who is in charge of what? What resources are necessary at critical points? Charts preconditions for each critical point Inputs and activities

Inputs/Factors Positive and Negative Resources and Barriers Tangible

Activities Processes, tools, events, actions Products Curricula, materials Services Education, training Counseling Infrastructure Relationships, capacities

Outputs DIRECT results of program activities Speak to size and scope Data about activities Number of classes taught Participation rates Hours of services provided

Outcomes & Impacts Outcomes What is expected to result from activities Individual level Impacts Higher level changes that result from program activities

Problem Statements

Feb 11: Identifying Problems and Asking Researchable Questions RLF Chapters 3 & 4 Light, R., Singer, J., & Willet, J. (1999). What are your questions? In By design: planning research on higher education (pp ). Cambridge, MA: Harvard University Press. Kerlinger, F. (1986). Problems and hypotheses. In Foundations of behavioral research (pp ). New York: Harcourt Brace Jovanovich.