Impact of evaluations matters IDEAS Conference 2011, Amman “Evidence to Policy: Lessons Learnt from Influential Impact Evaluations” Presenter: Daniel Svoboda, Czech Republic

Background There are never-ending discussions about sophisticated evaluation methods, competencies and independence. In the fight for perfect results, the use of evaluation results is still underestimated.

Critical Gaps The decision makers often have no idea how (and why) to use evaluation results The results of development interventions are assessed by donors; and final beneficiaries have little chance to say what their real priorities are The evaluators do not accept any responsibility for the intervention to be evaluated There are problems in communicating evaluation findings

Challenge Focus on sustainable benefits must lead to shift from accountability to fund managers to accountability for real results - positive (or negative) impacts on people's lives. The same need for a changed paradigm concerns also the evaluations - their purpose is not to provide unchallenged findings but to contribute to benefits of the target groups. So far, it seems that influential evaluations are considered only for a specific category of evaluations…

Questions What is the real purpose of impact evaluations?  Is it to satisfy the clients by proving their attribution?  Is it to test the theory of change in order to use the discovered causalities next time?  Is it to discover whatever impact of any intervention?  Or is it to confirm or revise the intended sustainable benefits for the target groups by discovering the real needs, mechanisms, motivations and assumptions ?

Questions  Who are the real clients of impact evaluations?  Can any donor or independent evaluator understand the genuine changes in lives and minds of the target groups?  Can the benefits be imposed or evaluated from outside? The impacts are directly linked to the target groups - and these are the only actors who can confirm the needs and priorities, and the existence, relevance, extent and reach of the benefits; as well as the key actors who must be engaged in any development intervention and impact evaluation.

Methods Participatory approaches do not mean using the participants as objects for formal approvals or as objects of evaluators´ focus groups, questionnaires or counterfactual experiments. The beneficiaries must participate in setting the priorities of development policies, programs and projects, in setting the indicators of benefits that are to improve their lives, and they must also enforce their own evaluation questions, evaluation methods, and utilization of lessons learned.

Methods Similarly, country-led evaluation system does not mean using the local staff trained, paid and ruled by donors but full local understanding of the need for evaluations, and introduction of appropriate own national systems, structures and mechanisms for effective performance, and use of evaluations.

Methods The evaluator must recognize (or revise) the theory of change and must confront the needed, achieved or achievable results with intervention's design and approaches. If these key aspects are not covered by the evaluation, then probably it is not a real evaluation. And in the same way, if there is no theory what change the evaluation can or should bring and what the expected use of evaluation results is, there is no reason to consider it for evaluation and to carry it out.

Methods Evaluators must be aware of and responsible for the impacts of their work and must be able to evaluate them as well. Therefore they should apply the same approaches for evaluations as required for interventions´ planners. This includes appropriate evaluation logic model (theory of change) with all crucial assumptions and risks, and all development effectiveness criteria, starting with ownership and ending with accountability for results.

Impact of evaluations What is the purpose of any evaluation? Use of evaluation recommendations and lessons learned… Why the results should be applied? To contribute to the common goal of the intervention and its evaluation – to sustainable benefits for target groups…

Impact of evaluations Result-based Evaluations? Goal = sustainable benefits for the target groups (final beneficiaries, not donors as evaluation clients!) Outcome = changed situation or behavior = proper and timely use of the evaluation outputs (recommendations) to enhance developmental effectiveness of the intervention or of the ODA system in general Output = evaluation findings and recommendations (report) Activities = evaluation itself

Impact of evaluations What are the indicators of evaluation success or failure? What is the added value of the evaluation? What are the sources of verification at each result level? What are the assumptions and risks at each level? What are the key preconditions? What means and costs are needed? And is not it a typical “killing assumption” that the intended user is not ready to use the evaluation results?

Conclusions There is no sense to carry out non-influential evaluations, however great their technical quality can be. Evaluators must be fully aware of the impact of their work and must take their part of responsibility for it. All evaluations must have their specific theory of change, and this theory must clearly relate to the intervention to be evaluated. Only the evaluation results matter. It does not mean the evaluation activities or report but the contribution of evaluation to sustainable benefits for the target groups.

Example In 2008, the “Evaluation of EC aid delivery through Civil Society organizations” was completed. The overall recommendation calls upon the EC to drastically improve the overall use of civil society as a channel for aid delivery. This implies: (i) ensuring greater consistency between official EC policy objectives towards civil society and current practices in using the CSO channel; (ii) better identifying and tapping the full added value of CSOs in helping to achieve key EC development objectives in various contexts; (iii) improving the conditions for achieving sustainable impact with aid delivered through CSOs; (iv) removing the political and institutional barriers at the level of EC (HQ and Delegations) for an effective and efficient use of the CSO channel.

Example What is the response to this concrete evaluation? What is its impact? All recommendations were agreed and a political support to cooperation with CSOs was confirmed. There are already some achievements at “activity” level like the running structured dialogue with CSOs, member states and the parliament, or civil society help desk. On the other hand, at “outcome” level, the space for CSOs both from European and developing countries is getting radically reduced, applied financial mechanisms undermine CSOs´ role of initiative and exclude them from political consultations, the predictability of aid worsened since 2008 and the CSOs still cannot fully apply their roles and added values as agents of social and developmental changes.

Example What was then the reason for this concrete evaluation? The positive and irreplaceable role of CSOs was confirmed by evaluators and agreed by the client. But the situation of CSOs got worsened. Whose fault? Did the client expect different results? Did the evaluator make a mistake in formulating non-applicable recommendations? Did the evaluator underestimate the external assumptions or the effects of global crisis? Are there other critical constraints at the client's side? Is there any chance to reach a positive impact? Who knows? And who should know?

Impact of evaluations? Thank you for your attention! Daniel Svoboda: