1. Assessing Program Impact Chapter 8

2. Impact assessments answer… Does a program really work? Does a program produce desired effects over and above what would have occurred without the intervention? Are there important unintended effects?

3. Impact Assessments can be relevant at many points Policy formation Evaluating a pilot program Ongoing programs – may need to modify or refine high costs of programs – must continually evaluate their efficacy demonstrations of effectiveness are required to renew grants justify uses of public funds

4. Prerequisite conditions impact assessments build on earlier forms of evaluation the program’s objectives must be well articulated and have plausible, measurable outcomes intervention is well implemented and has been in place for sufficient time

5. Equivalence identical composition - same mixes in terms of their program-related and outcome-related characteristics identical predispositions - equally disposed toward the project and equally likely to attain any given outcome status identical experiences – over the time of observation

6. If two groups are “equivalent”, both are subject to the same degree of change induced by factors outside of the program. Any difference in outcome between them, therefore, should represent the effect of the program.

7. Two classes of Approaches: Experimental vs Quasi-Experimental Research Designs Randomized field experiment the “gold standard” research design Participants are randomly sorted into at least two groups – a control group and an intervention group Quasi-experiments Nonrandomized in which participants (given intervention) are compared with nonparticipants (the controls)

8. Randomized Field Experiments Principal advantage: it isolates the effect of the intervention being evaluated by ensuring that intervention and control groups are statistically equivalent except for the intervention received Best way to achieve equivalence!

9. Randomization is just not a “shot in the dark Allocating targets to intervention and control groups requires considerable care The evaluator must use an explicit chance-based procedure Be careful when using lists use statistical significance testing to judge whether a specific difference is likely to have occurred simply by chance

10. Data Collection Strategies Two strategies can improve the estimates of program effects Make multiple measurements of the outcome variable Collect data periodically during the course of an intervention

11. Units of Analysis the units on which the outcome measures are taken should be based on the nature of the intervention and the target units Be careful! Observational vs. experimental units Assumption of independence

12. Limitations of Randomized Experiments These research designs are not applicable to all program situations The program usually has to be stable and operationally mature for authentic results If the program changes during the course of the experiment, it is difficult to differentiate between what program version produced what effects. Difference between Experimental and Actual program delivery

13. Ethical Considerations ethical qualms about randomization – deprives control groups of positive benefits If program resources are scarce do you allot services by chance? or insist that the most needy targets receive priority? The next chapter that will look at alternative designs.

14. “Perfect” vs “Good Enough” “Perfect” vs “Good Enough” time and resource constraints intended use of the results feasibility of design ethical considerations credibility differences between experimental and actual intervention delivery high turnover in policy considerations integrity of a randomized experiment is easily threatened

15. Impact Assessments Overview All impact assessments are comparative – with a group receiving alternative services or no treatment. Ideally, the conditions being compared should be identical in all respects except for the intervention All assessment involves establishing control conditions The most valid results generally require more skills, more time to complete and more cost.

16. Beware!Rigorous Impact Assessments involve technical and managerial challenges and significant resources - sometimes has political pressures Is it justified by the circumstances?