The World Bank Gladys Lopez Acevedo

Questions Which kind of systems have countries, blocs of countries (OECD, EU, etc), or international organizations (World Bank, PNUD, IDB, etc) developed that would allow, facilitate and promote policy makers to share information on evaluation? Which federative countries have developed efficient systems for sharing successful local experiences? How do these systems function?

Background There are several systems implemented both by international organizations and by countries to share and promote evaluation Surprisingly there is very little systematically documented on these systems therefore hard to say if they are successful or not On a recent Bank publication “ Building Better Policies” we analyzed leading evaluation systems that had been in place for more than a decade visit

Classification on cross-comparison 4 CountryPurposeLeadershipOperationUsers ChileBudget/ControlDirectCentralizedSingle MexicoBudget/AccountabilityDelegatedDecentralizedMultiple AustraliaBudgetDirectDecentralizedSingle CanadaBudgetDirectDecentralizedSingle UK/NAO*AccountabilityDirectCentralizedMultiple UK/TreasuryBudget/ControlDirectDecentralizedSingle * Supreme Auditor

Chile: Main Features 5 System is highly centralized – closely linked to and based on a highly centralized, top-down budget process Budget office manages most details, and is main user of information – utilization fostered by close link to budget process Management control by hierarchical oversight Little buy-in from other (potential) stakeholders Some impact on allocations and program management Chile: Utilization of Government Evaluations—2000–05 Effect on program Minor adjustment of program, for example, improved processes or information systems Major adjustment of management processes, for example, changed targeting criteria, or new information systems Substantial redesign of program or organizational structure Institutional relocation of program Program termination Programs affected 24%38%25%5%8% Percentage of all evaluated programs. Source: Guzman 2007

Chile: How does it work? 6 6 Budget Office Budget Office Budget Office Programs Budget Office Define the evaluation plan Decide what and how to evaluate Coordinate and steer evaluation system Maintain quality control Define monitoring parameters Programs are evaluated directly by Budget Office Select and hire evaluators Supervise evaluation implementation Monitor indicators and targets Collection of evaluation results Dissemination of evaluation results Utilization of evaluation results Budget Office Congress

Australia( similar to Canada and USA ): Main Features 7 Mandate to evaluate each program every 3-5 years Portfolio Evaluation Plans to be prepared annually for the following three years Department of Finance: Steering & quality control Sector Departments: planning, implementation Evaluation results primarily used for budgetary decisions: allocations of funds for new policies and reallocation of savings (i.e. the discretionary part of annual budget process) System lasted from – now system is even more decentralized

Australia: How does it work? 8 Treasury Department of Prime Minister and Cabinet Department of Finance Parliament Citizens Inputs to PEPs Inputs to Portfolio Evaluation Plans (PEPs) Formal notification of PEPs Involvement in evaluations Publication of department’s evaluation reports Reporting of key Evaluation findings in department’s budget paper (prospective) and in their annual reports (retrospective) Source: Mackay 2011 Sector Departments (and outrider agencies)

How do institutional differences affect utilization and sustainability? 9 Centralization requires the right institutional structure – a centralized evaluation system design in a fragmented public sector will fail In a system with multiple stakeholders delegation to an impartial agency might be a viable option (AEVAL or CONEVAL type)– but beware of over-engineering and objective overload Who is to gain and who has to worry about buying into the system: The senior civil service, the legislature, ministries, service delivery units, the finance ministry, the head of government (PM or President)? Staying in control of overall steering and quality of outputs does not mean being in charge of all aspects of evaluation implementation – strategic delegation might be smart for buy-in and workload Is it possible to imagine a long-term sustainable, well utilized evaluation that does not have a stable link to budgetary decisions?

Questions Is there any consistent effort for building an international or national system for the classification of public policies? If so, what is this system? How does it classify public policies? How refined (how many categories does it consider) and well structured (on how many levels can the classification be conducted) is it?

Background Yes, most country systems have gone through a process of public policy classification according to the objective of the system ( budget, accountability, etc) For example, the USA through the PART system used a public policy classification compatible with their budget classification, similarly other countries are trying to do something similar ( Mexico, South Africa) There are also international agencies such as 3IE (Campbell Evaluation Guidelines) which have gone through a process of program/public intervention classification To certain extend the Bank is also following this classification in our new jobs knowledge platform

Example from evaluation guidelines SA

Thank You! Contact: