Evaluation Total ODA Impact Stefan Molund, SIDA Fourth meeting of the DAC Network on Development Evaluation, Paris, 30 – 31 March 2006

Executive summary The proposal ripe for go/no go decision The proposal ripe for go/no go decision Main alternatives: Main alternatives: 1.Introduce the proposal to interested partner countries 2.Put it aside for the time being Remaining questions about design and process best discussed with partner countries Remaining questions about design and process best discussed with partner countries

Consultation process Presentation of original proposal. Paris, November 2004 Presentation of original proposal. Paris, November 2004 Assessment of technical feasibility by team of development economists (Bigsten, Gunning, Tarp) Assessment of technical feasibility by team of development economists (Bigsten, Gunning, Tarp) Multidisciplinary seminar on evaluation of total ODA and alternative approaches to macro-evaluation. Stockholm, November 2005 Multidisciplinary seminar on evaluation of total ODA and alternative approaches to macro-evaluation. Stockholm, November 2005 Assessment of proposals by enlarged evaluation task force. Edinburgh, February, 2006 Assessment of proposals by enlarged evaluation task force. Edinburgh, February, 2006

Original proposal Focus on development impact Focus on development impact Broad coverage Broad coverage Long-term perspective Long-term perspective Concern with change mechanisms Concern with change mechanisms Country development starting point: tracing links from impacts backward to inputs rather than the other way around (as in most evaluations). Country development starting point: tracing links from impacts backward to inputs rather than the other way around (as in most evaluations). Focus on combined donor support (the distinctive feature of a total ODA evaluation) Focus on combined donor support (the distinctive feature of a total ODA evaluation) Impacts assessed in partner country perspective. Impacts assessed in partner country perspective.

Original proposal (contd.) Joint donor-partner country evaluation with partner country actively involved Joint donor-partner country evaluation with partner country actively involved Independent evaluation team Independent evaluation team Multidisciplinary evaluation team well grounded in country context Multidisciplinary evaluation team well grounded in country context

Key conclusions from consultations Original proposal too broad in scope Original proposal too broad in scope Evaluation technically feasible provided that statistical ‘gold standards’ for causal analysis are replaced by more realistic standards of process tracing, ‘plausible association’, inference to best explanation, etc. Assessing total ODA impact very different from assessing project impact on restricted target group by experimental or quasi-experimental methods. Eclectic multidisciplinary approach required. Evaluation technically feasible provided that statistical ‘gold standards’ for causal analysis are replaced by more realistic standards of process tracing, ‘plausible association’, inference to best explanation, etc. Assessing total ODA impact very different from assessing project impact on restricted target group by experimental or quasi-experimental methods. Eclectic multidisciplinary approach required.

Key conclusions from consultations (contd.) Partner country development strategies not suitable as primary evaluation criteria. Generally accepted measures of development more appropriate. (However, assessing usefulness of total ODA to partner country development efforts would still be very relevant.) Partner country development strategies not suitable as primary evaluation criteria. Generally accepted measures of development more appropriate. (However, assessing usefulness of total ODA to partner country development efforts would still be very relevant.) Inviting partner countries to join the evaluation is the logical next step. Inviting partner countries to join the evaluation is the logical next step.

Revised evaluation design Focus on development impact Focus on development impact Focus on impact on restricted set of development variables (New) Focus on impact on restricted set of development variables (New) Long-term perspective Long-term perspective Focus on change mechanisms Focus on change mechanisms Country development starting point Country development starting point Focus on combined donor support Focus on combined donor support Impacts assessed against generally accepted welfare criteria. (New) Impacts assessed against generally accepted welfare criteria. (New) The process requirements are the same as before. The revised proposal would serve as a framework for open discussions between the partners.