When are Impact Evaluations (IE) Appropriate and Feasible? Michele Tarsilla, Ph.D. InterAction IE Workshop May 13, 2013

A Few Initial Remarks IE represent only one of the different types of international development evaluation IE definitions vary (e.g., OECD-DAC and Randomistas Movement) and clarity of specifications is often lacking Disagreement within the development community over the most adequate design or method to be used in IE IE sometimes are feasible and appropriate but their questions are not relevant

WHEN is IE appropriate? When there is a real need for original and new information on a certain intervention’s effects Examples: - the extent to which an untested hypothesis holds (e.g. in case of an innovative or pilot projects); -which combination of activities or dosage of intervention is the most effective/contributing to the attainment of the envisaged impacts; When the time elapsed between baseline and follow-up is sufficient (it does not always need to be 4-5 years) Not appropriate when essential goods/services are denied to the comparison group; Data Quality review ensured throughout data collection

WHEN are IE appropriate? When the right sampling frame is in place and the target population has been adequately identified (IE often misses the effects of intervention on the most marginalized, invisible, mobile population) When a communication strategy AND a dissemination strategy are in place (as IE strive to contribute to public knowledge, effects/results that will need to make sense for the general public - effect size vs. statistical significance) When the intervention being evaluated is linear and not too complex or multi-level (recursive logic models or multiple causal pathway for each individual)

When are IE Appropriate? When a good monitoring system is in place (monitoring data will allow opening the so-called black box – the how/why’s etc.) When selection bias and other threats to validity have been adequately addressed When triangulation is pursued in a systematic fashion in the course of data collection When no major deviation from the implementation strategy is envisaged; No major adjustments are expected to take place throughout implementation. When context and other environmental factors influencing the impact are qualified

When are IE Appropriate? Some exceptions to this is the Familias en Accion in Colombia) When there is sufficient time available for the findings (likely to be yielded by the evaluation) as to inform the decision that will need to be made When there are rival plausible explanation for the results observed

When are IE Feasible? When you have sufficient resources (money and technical expertise - better if independent); When you have created sufficient support from local and national authorities; When a good understanding of the evaluation rationale, timeline and practical implications among implementing partners is there (otherwise, the risk is that the control or comparison group will sabotage the activity) When you have a baseline or a baseline could be reconstructed (e.g., by exploiting medical records/data collected by other donors in the target areas of interest)

When are IE Feasible? When you have identified and tested for alternative explanations of the differences between the treatment and the comparison/control group When a clear theory of change is available and agreed upon and understood by the program/project and the evaluation team; When an evaluability assessment has been conducted When RFPs for evaluators and implementers clearly call for joint planning/data collection and collaboration on evaluation-related activities; When incentives for implementers and evaluators are aligned with each other Not feasible when the intervention is universal and no untreated comparison/control group could be identified Not feasible when randomization is not possible (e.g. infrastructure)

Thank you! Contact info: Michele Tarsilla