1. Program Evaluation In A Nutshell 1 Jonathan Brown, M.A.

2. Two Types of Evaluation At the most basic level there are two types of evaluation: Formative: primary purpose is to provide information to improve a program. Summative: concerned with providing information about program adoption, continuation, or expansion. 2

3. Two Types of Evaluation 3 Relative Emphasis Program Life Summative Evaluation Formative Evaluation

4. What Are You Assessing? The evaluation plan must consider the situation or context of the evaluation. The purpose of the evaluation. Improvement (formative evaluation). Accountability (summative evaluation). Knowledge generation. Hidden agendas. 4

5. What Are You Assessing? The evaluation plan must consider the situation or context of the evaluation. The program’s structure and context. The stage of program development. Administrative and political context. Conceptual and organizational structure of the program. 5

6. What Are You Assessing? The evaluation plan must consider the situation or context of the evaluation. The resources available for the evaluation. Personnel Equipment Facilities to support data collection, analyses, and reporting. Is specialised expertise needed? 6

7. What Are You Assessing? Typically, evaluation involves assessing one or more of the five following areas: 1.The need for a program. 2.The program’s design. 3.The program’s implementation or process. 4.The program’s impacts or outcomes. 5.The efficiency of a program. 7

8. Needs Assessment Purpose of a social program is to alleviate a social problem. Needs assessments assess: The nature, extent, and distribution of a social problem. How these features will impact the design of an intervention program. 8

9. Needs Assessment These assessments can be conducted when… Planning a new program. Restructuring an existing program. Determining if an existing program is responsive to the current needs of the target population. 9

10. Assessing Program Theory Program theory evaluation is an evaluation guided by a program’s explicit theory of how it causes intended outcomes. Program theory outlines the following: Target population Resources Activities Outcomes A program’s theory is represented by Theory of Change Models (TCMs) and Program Logic Models (PLMs). 10

11. Assessing Program Theory 11 Conditions before the project With no information about the program, it remains a mysterious black box. The evaluation will also not be able to attribute the final conditions to the specific aspects of the program. Conditions after the project The Program is a

12. Assessing Program Theory Assessing program theory can be done by applying SMART principles (Knowlton & Phillips, 2009). S pecific – the program is clear enough to implement and evaluate. M easurable – indicators can be qualified and/or quantified. A ction oriented – activities will provoke the desired change in targets. R ealistic – the program is plausible and feasible. T imed – duration of activities and intended outcomes are specified. 12

13. Process Evaluation Given that a valid theory on how to intervene on a diagnosed social problem has been done, the program must be implemented well to be successful. Two forms: Process evaluation Program monitoring 13

14. Process Evaluation Process evaluation looks at service utilization and program organization. Service Utilization The extent to which the intended targets receive services. Important when participation is voluntary, or participants must learn new procedures, or habits. Coverage Bias 14

15. Process Evaluation Process evaluation looks at service utilization and program organization. Program Organization How well the program is in terms of managing its efforts and using its resources to accomplish its activities. Service delivery: None or not enough intervention Wrong intervention Unstandardized intervention 15

16. Outcome Evaluation Also known as an impact assessment. Assesses the extent to which a program produces the outcomes it intends. But the desired changes could be caused by factors unrelated to the program. Interested in the changes produced by the program above and beyond any external elements. 16

17. Outcome Evaluation Program outcomes The states or conditions of the target that changed as a result of a program. 1.Observed characteristics of the target population or social conditions. 2.Occur to different degrees. 17

18. Outcome Evaluation Program outcomes Outcome level – status of an outcome at a point in time. Outcome change – difference between outcome levels at different points in time. Program effect – the unique portion of an outcome that can be attributed to only the program. 18

19. Outcome Evaluation 19 Outcome Variable Before Program During Program After Program Outcome status without program Outcome status with program Program effect Post-program outcome level Pre-program outcome level Outcome Change (Rossi, Lipsey, & Freeman, 2004)

20. Efficiency Assessment Due to limited resources, program accomplishments must be judged against program costs. Two types of efficiency assessments: Cost-benefit analysis Cost-effectiveness analysis Such assessments require placing a dollar amount on program activities and benefits. 20

21. Typical Evaluation Questions Needs Assessment What is the nature and magnitude of the problem? What are the characteristics of the population in need? What are the needs of the population? What services are needed? How much service is needed and over what period of time? What service delivery arrangements are needed to provide services to the population? 21

22. Typical Evaluation Questions Program Theory Assessment Is the program doing the right thing? How could the program better address the needs of participants? How should the program be organized? What are the best delivery systems for the services? Is the program specific enough to measure? 22

23. Typical Evaluation Questions Process Evaluation How many persons are receiving services? Are administrative and service objectives being met? Are the intended targets receiving services? Are the necessary program functions being performed adequately? Are resources being used effectively and efficiently? 23

24. Typical Evaluation Questions Outcome Evaluation Are the goals and outcomes being achieved? Do the services have adverse side effects? Are some participants affected more by the services than others? Is the problem the services are intended to address made better? 24

25. Typical Evaluation Questions Efficiency Assessment Are resources used efficiently? Is the cost reasonable in relation to the magnitude of the benefits? Would alternative approaches yield equivalent benefits at less cost? 25

26. QUESTIONS? 26 Jonathan Brown, M.A. jbrown@asc4solutions.com