1. 1 The Cases of Chile and Colombia Presentation to a LAC Region Summer Seminar 9 August 2006 Keith Mackay Independent Evaluation Group (kmckay@worldbank.org)

2. 2 What Does “Success” Look Like? -- Why Countries Want an M&E System To support budget decision-making = performance-based budgeting To support budget decision-making = performance-based budgeting To support national and sectoral planning To support national and sectoral planning To design policies and programs To design policies and programs To assist sector ministries / agencies in their management To assist sector ministries / agencies in their management To strengthen accountability relationships To strengthen accountability relationships

3. 3 Chile’s M&E System -- Architecture Chile’s M&E System -- Architecture Designed, managed and used by Hacienda Designed, managed and used by Hacienda Developed incrementally, over past decade Developed incrementally, over past decade Performance indicators ( ∑ 1,600) for all government programs (1994) Performance indicators ( ∑ 1,600) for all government programs (1994) Government program evaluations ( ∑ 160) -- these are desk reviews (1996) Government program evaluations ( ∑ 160) -- these are desk reviews (1996) Rigorous impact evaluations ( ∑ 14) (2001) Rigorous impact evaluations ( ∑ 14) (2001) Comprehensive Spending Reviews -- desk reviews of all programs in a functional area (2002) Comprehensive Spending Reviews -- desk reviews of all programs in a functional area (2002)

4. 4 Chile’s M&E System -- Strengths (1) ‘Graduated’ approach to M&E ‘Graduated’ approach to M&E Evaluations conducted externally, in fully transparent process, and are highly credible Evaluations conducted externally, in fully transparent process, and are highly credible All M&E findings reported publicly and sent to Congress All M&E findings reported publicly and sent to Congress M&E system closely linked to the information needs of Hacienda, especially for budget process M&E system closely linked to the information needs of Hacienda, especially for budget process Performance information used to set performance targets for ministries -- these are largely met Performance information used to set performance targets for ministries -- these are largely met

5. 5 Chile’s M&E System -- Strengths (2) High utilization of M&E findings by Hacienda in the budget process and to impose management improvements on ministries / agencies (see Table) High utilization of M&E findings by Hacienda in the budget process and to impose management improvements on ministries / agencies (see Table) Utilization of government evaluations -- 2000 to 2004 Minor adjustment, modification, relocation of program Change in program design or manage- ment process Institutional relocation of program Substantial redesign of program or of internal management processes Program termination TOTAL 25%39%5%21%10%100%

6. 6 Chile’s M&E System -- Challenges Unevenness in quality of evaluations -- due to cost and time constraints Unevenness in quality of evaluations -- due to cost and time constraints Chile probably not spending enough on evaluations Chile probably not spending enough on evaluations Low utilization -- low ‘ownership’ -- of Hacienda’s evaluations by sector ministries Low utilization -- low ‘ownership’ -- of Hacienda’s evaluations by sector ministries

7. 7 Colombia’s M&E System (SINERGIA) -- Architecture SINERGIA is managed by the Department of National Planning (DNP), with strong support from the Presidencia SINERGIA is managed by the Department of National Planning (DNP), with strong support from the Presidencia On-line sub-system -- SIGOB -- for monitoring and reporting government progress vis-à-vis Presidential Goals On-line sub-system -- SIGOB -- for monitoring and reporting government progress vis-à-vis Presidential Goals Ambitious agenda of impact evaluations ( ∑ 15 underway) Ambitious agenda of impact evaluations ( ∑ 15 underway) DNP provides technical assistance to a few ministries/agencies to develop M&E, and to municipalities to pilot SIGOB and performance-based budgeting DNP provides technical assistance to a few ministries/agencies to develop M&E, and to municipalities to pilot SIGOB and performance-based budgeting

8. 8 Colombia’s M&E System -- Strengths Very high utilization of SIGOB by President for oversight of ministers and ministries -- via performance targets -- and for accountability, i.e. ‘social control’ Very high utilization of SIGOB by President for oversight of ministers and ministries -- via performance targets -- and for accountability, i.e. ‘social control’ Rigorous impact evaluations conducted externally, and have high credibility Rigorous impact evaluations conducted externally, and have high credibility Collaborative approach between DNP and sector ministries/ agencies, and with municipalities Collaborative approach between DNP and sector ministries/ agencies, and with municipalities Performance budget reports; efforts to further strengthen performance budgeting Performance budget reports; efforts to further strengthen performance budgeting Efforts to engage with civil society Efforts to engage with civil society

9. 9 Colombia’s M&E System -- Challenges Too high reliance on donor funding for SINERGIA --low level of government funding support Too high reliance on donor funding for SINERGIA -- low level of government funding support Insufficient reliance on M&E information to support national planning and budget decision-making -- this may now be changing Insufficient reliance on M&E information to support national planning and budget decision-making -- this may now be changing Weak coordination of M&E roles / functions within DNP and with central and sector ministries Weak coordination of M&E roles / functions within DNP and with central and sector ministries SIGOB data quality perceived as low SIGOB data quality perceived as low

10. 10 How to Define a Government M&E System as Being “Good Practice” Can be dangerous concept -- each country’s starting point and desired end-point are unique Can be dangerous concept -- each country’s starting point and desired end-point are unique What a “successful” M&E system is NOT: What a “successful” M&E system is NOT:  complex set of laws, decrees, regulations  # performance indicators collected  # rigorous impact evaluations conducted  # rapid evaluations conducted  = these simply reflect the architecture of the system or are measures of M&E effort

11. 11 How to Define a Government M&E System as Being “Good Practice” Chile and Colombia have good-practice M&E systems because Chile and Colombia have good-practice M&E systems because 1. the quality of their M&E work is generally reliable, and 2. in particular, the monitoring information and evaluation findings which they produce are used intensively High utilization reflects strong demand for M&E, and can be viewed as a predictor of an M&E system’s sustainability High utilization reflects strong demand for M&E, and can be viewed as a predictor of an M&E system’s sustainability

12. 12 Lessons (1) Lessons from Chile and Colombia are consistent with other countries’ experience Lessons from Chile and Colombia are consistent with other countries’ experience 1. Key role of powerful champion of M&E 2. Opportunistic development of M&E systems, via continuous review and modification; non-linear development of the systems 3. Centrally-driven, by capable ministry 4. Incentives are key -- cultural change to strengthen demand, achieve high utilization 5. Avoid competing systems – Planning, Finance

13. 13 Lessons (2) 6. Build reliable ministry data systems 7. Role of structural arrangements to ensure M&E objectivity and quality 8. Long-haul effort, requiring patience 9. Limitations of relying on laws, decrees, regulations 10. An M&E system can be built and operated are relatively low cost  Chile’s M&E system costs $0.75m p.a.  Colombia’s system costs about $2m p.a.

14. 14 Useful Resources (1) Chile: www.dipres.cl/fr_control.html Colombia: www.dnp.gov.co/paginas_detalle.aspx?idp=266www.dnp.gov.co/paginas_detalle.aspx?idp=266 Ernesto May et al. (eds.), Towards the Institutionalization of Monitoring and Evaluation Systems in Latin America and the Caribbean, World Bank/IADB, 2006. http://web.worldbank.org/WBSITE/EXTERNAL/COUNTRIES/LACEXT/0,,conte ntMDK:20893139~pagePK:146736~piPK:146830~theSitePK:258554,00.html (Disponible en Español) http://web.worldbank.org/WBSITE/EXTERNAL/COUNTRIES/LACEXT/0,,conte ntMDK:20893139~pagePK:146736~piPK:146830~theSitePK:258554,00.html Keith Mackay, Institutionalization of Monitoring and Evaluation Systems to Improve Public Sector Management, Independent Evaluation Group, World Bank, 2006. www.worldbank.org/ieg/ecd/institutionalizing_me.html (Disponible en Español) www.worldbank.org/ieg/ecd/institutionalizing_me.html

15. 15 Useful Resources (2) Ariel Zaltsman, Experience with Institutionalizing M&E Systems in Five Latin American Countries: Argentina, Chile, Colombia, Costa Rica and Uruguay, Independent Evaluation Group, World Bank, 2006. www.worldbank.org/ieg/ecd/experience_five_la.html Fernando Rojas et al, Chile: Análisis del Programa de Evaluación del Gasto Publico, World Bank, 2005. http://iris37.worldbank.org/domdoc/PRD/Other/PRDDContainer.nsf/WB_Vie wAttachments?ReadForm&ID=85256D2400766CC785257155005CB26B& http://iris37.worldbank.org/domdoc/PRD/Other/PRDDContainer.nsf/WB_Vie wAttachments?ReadForm&ID=85256D2400766CC785257155005CB26B& World Bank website on Building Government M&E Systems: www.worldbank.org/ieg/ecd/