POLICY IMPACT EVALUATION Biogas in Rwanda Willem Cornelissen – ERBS, Erasmus University Rotterdam

Outline The central questions of the CDI seminar Policy evaluation in relation to impact evaluation IOB’s role; its evaluation of renewable energy; IE biogas in Rwanda: results chain Coming back to the central questions

Central questions Who defines what is IE? Who defines what is to be evaluated? How is utilization of IE findings influenced by IE design and communication?  What influences IE design?  What influences communication of IE findings?  What can be done in design and communication to enhance utilization of IE findings?

4 What policy evaluation is all about Policy is about what a government wants to achieve – it is specific, measurable and time bound (the target) A strategy is how to go about achieving the particular policy objectives (the direction) Plans and programmes – who, when and resources required (the means) Review processes Institutions and capacities

Public Budget of the Netherlands Policy articles; these are –in principle- subject to evaluation (or parts thereof) Ref. to renewable energy: Chapter V (Foreign Affairs); policy article 6: “better protected and improved environment”, art 6.1 “Protection and sustainable use of the environment in its global context and national context of developing countries” In 2011 the art. 6.1 was amended to: “sustainable use of the environment in the world” (the national context was eliminated).

The budget law defines Governmental Regulation for Policy Evaluation research (RPE)  Periodic policy evaluation  Study of net policy effects Objective of RPE: improvement of the quality of policy information; more ‘value for money’  Justification of public expenditure  Effectiveness  Efficiency  Sustainability Purpose:  Contribute to the Minister’s accountability to the Parliament  Learning function OECD

IOB: Policy and Operations Evaluation Department Mandate: evaluations of policies and operations of the Min. of Foreign Affairs Its findings are public - through parliament (Renewable) energy: policy evaluation underpinned by eight Impact Evaluations Amongst these: biogas in Rwanda Motivation: Overall policy objective: developmental + environmental; strategy objective: biogas in Africa; programme: Dutch funded programme in Rwanda

Biogas results chain

Choice of methods Statistical requirements: robustness, representativeness (sample size), confidence levels; attributable results Pragmatic: time constraint, limited financial resources (excludes option for DiD and RCTs) Mixed methods IE: Quantitative: 600 hh (300 with; 300 without); cross-sectional with propensity score matching Qualitative: village research, stakeholder analysis, semi structured survey masons

Coming back to the central questions (1) Design Perception of accountability -through Parliament- to Dutch tax payer Design in part determined by budget law (RPE) and in part by OECD evaluation questions and by pragmatic constraints Communication Accountability: Ministry for Dev.Coop and Foreign Trade + Foreign Affairs towards Parliament Budget accountability Learning: Government of Rwanda Policy makers Min.Dev.Co. + Foreign Tr Programme and project implementers, practioners

Coming back to the central questions (2) Dissemination (not done yet) Publication (book) Newsletter and website Scientific publication Presentation in seminars Utilization Formalized feed-back within ministry, and with implementers But:…. Utilization depends on WHO communicates and the convenience of the moment HOW it relates to mainstream thinking and norms (the logic of appropriateness) The appeal for what is NOVEL The importance of personal gain (saving face; prestige; office politics)

Dilemma’s Elliot Stern stated that evaluators sharpen their tools & techniques to evaluate more and more rigorously and hence less and less policies. However, IE is needed to address the effectiveness question of the policy evaluation Policies are volatile... IE measures past policies, not current ones Impact evaluation is an artisanal skill…..overkill and undermining of quality