SYSTEM OF EVALUATION AND MANAGEMENT CONTROL RESULTS-BASED BUDGETING THE CHILEAN EXPERIENCE Heidi Berner H Head of Management Control Division Budget Office, Ministry of Finance, Chile October, 2008

12 EVALUATION AND MANAGEMENT CONTROL SYSTEM OBJECTIVE: Providing performance information and introducing practices to improve the quality of public expenditure Improving resource allocation Improving the use of resources Improving transparency

13 EVALUATION OF PROGRAMS AND INSTITUTIONS 1997: Evaluation of Public Programs (EPG) 2001: Impact Evaluation of Programs 2002: Agency Evaluation (Comprehensive Spending Review) 1993: PERFORMANCE INDICATORS 2001: PRESENTATION OF PROGRAMS TO BUDGET INSTITUTIONAL WAGE INCENTIVE MECHANISMS 1998: Management Improvement Program: - Basic Framework Program (2001) - Advance Framework Program (2003) 2003: Incentive to Physicians 2001: STRATEGIC DEFINITIONS 1997: COMPREHENSIVE MANAGEMENT REPORT (BGI) 3

14 PERFORMANCE INDICATORS COMPREHENSIVE MANAGEMENT REPORT (BGI) PRESENTATION OF PROGRAMS TO BUDGET 2007: Technical Assistance 2008: Reviewing Program Presentation EVALUATION OF PROGRAMS AND INSTITUTIONS 2006: Increasing coverage programs or agencies will be evaluated between : Creation of New Line: Evaluation of New Programs INSTITUTIONAL WAGE INCENTIVE MECHANISMS 2009: Management Improvement Program (PMG) – Framework Program of Quality 2007: Institutional Efficiency Goals 2008: PUBLIC MANAGEMENT MODERNIZATION FUND 4 STRATEGIC DEFINITIONS

15 PERFORMANCE AND BUDGETARY INFORMATION Not Direct relation (not mechanical) Use together with other categories of information - Political Priorities - Financial Restrictions Presentation to the Congress along with the Budget Law

16 PERFORMANCE INDICATORS Aimed at telling how a government organization is performing over time Agencies compete with their past record Ongoing, periodical information Measure performance in different: –Dimensions (effectiveness, efficiency, economy, service quality) –Delivery levels (process, output, outcome) Need reliable data collection methods Support from strategic plan, management information system (PMG) Started 1993 as a pilot, since 2001 is part of the budget preparation Incrementalist approach, starting on voluntary basis Budget-related (public agencies presents their indicators and its goals during budget preparation) Disclosure policy: Congress and general public

17 PERFORMANCE INDICATORS Coverage and Quality of Performance Indicators

18 EVALUATION OF PROGRAMS AND INSTITUTIONS Assess programs against their stated aims and expected results on their beneficiaries Requirements: relevance, independence, timeliness, transparency, public, reliable, efficient. Agreement with Congress and Financial Management Law and its Regulation (2003) Started 1997 based on logical framework methodology (EPG) Budget-related (the results of the evaluation are taking into account in the budget preparation) Programs selected with Congress Performed by independent evaluators through panels of experts or universities and consulting firms. Counterpart in ministries/agencies in charge of programs Impact evaluations (2001) Agency evaluations (2002) Delivered in 6 to 12 months Reported to Budget, Congress and the public Commitments to incorporate recommendations from the evaluation Follow up of commitments compliance

19 Number of Programs and Agencies Evaluated ( ): On progress. EVALUATION OF PROGRAMS AND INSTITUTIONS

110 ( ) Evaluation of Programs (EPG - Impact) EFFECTS % PROG. 1.- MINOR ADJUSTMENTS 2.- MODIFICATIONS IN THE DESIGN OF THE COMPONENTS AND PROCESSES 3.- MAJOR REFORMULATIONS 5.- PROGRAM CANCELLED OR COMPLETELY REFORMULATED 23% 37% 27% 7% TOTAL PROGRAMS (174)100% 6%4.- INSTITUTIONAL RELLOCATION EVALUATION OF PROGRAMS AND INSTITUTIONS

111 New Line of Evaluation: Evaluation of New Programs (EPN). To assure that Chile remains in the leading ranks of countries with systematic evaluation processes by updating evaluation procedures and processes to world frontier levels. 11 EVALUATION OF PROGRAMS AND INSTITUTIONS i)Designing of the evaluation at the beginning of each new program ii)Establishing control groups for the evaluation, based on randomized trials whenever is possible iii)Establishing an international advisory committee to periodically review and assess the process of evaluation iv)Implementing this new line gradually

112 PRESENTATION OF PROGRAMS TO BUDGET Started 2000 with preparation of 2001 Budget Standard format to submit public programs for funding (2005 – 2009) Ministries submit new programs and existing programs with importance budget increases Submission based on logical framework matrix (aims, goals, expected results, components, indicators, target population) Finance makes proposal on basis of quality, consistency with government priorities and strategies, President makes final decision Provides better base line for future impact evaluation 2008: Budget Direction gives technical assistance (new Department in the Management Control Division) 2009: Review of all programs presented to budget

113 COMPREHENSIVE MANAGEMENT REPORTS Started 1997, from Executive-Congress agreement Conceived as accountability mechanism Report performance against stated mandates, goals, commitments, resources Supports compliance with other management control mechanisms (performance indicators, program evaluation, management improvement programs) and financial management regulations Standard report format defined by Budget Office Distributed to Congress, publicly available Basis for gathering information for the evaluation stage of the budget cycle

114 The achievement of management objectives are associated to a monetary incentive to all employees in the public institution (Law , 1998). The percentages of incentives are: MANAGEMENT IMPROVEMENT PROGRAM (PMG) % of Incentive for Institutional Performance % of Achievement of Management Objectives 2007* ,7%6,3%7,0%7,6%90% - 100% 2,85%3,15%3,5%3,8%75% - 89% 0% < 75% * Before 2007 the % of incentives were 5% and 2,5%. The modification was done by Law , august 2007.

115 The PMG has four period of development clearly defined. In each of these periods the accomplishment of the management objectives committed was associated to: Performance indicators. Period 1998/2000. Management Systems established in a Basic Framework Program (2001). Management Systems established in a Advance Framework Program including the requirements of ISO 9001:2000 (2005) Framework Program of Quality – certifying the processes involved in the provision of good and services by ISO 9001:2000 standard MANAGEMENT IMPROVEMENT PROGRAM (PMG)

116 Basic Framework Program Advance Framework Program Framework Program of Quality Public Management Areas and Systems (2001) Public Management Areas and Systems defined in the Basic Framework Program certified by ISO 9001:2000 standard (2005) Management Quality System implemented and processes of provision of good and services certified by ISO 9001:2000 standard (2009) 16 MANAGEMENT IMPROVEMENT PROGRAM (PMG)

117 Basic Framework Program 5 areas and 11 management systems with 4 or 6 development stages. The evaluation of the accomplishment of each stage is done by the Ministries of Finance, Internal Affairs and Presidency with the help of an Expert Network (public institutions in charge) Advance Framework Program The advanced framework program incorporates the objectives of the Basic Framework Program and the requirements of ISO 9001:2000 standard, in order to certificate each PMG system. Four stages: preparation to certification, certification and maintenance of certification for two years. MANAGEMENT IMPROVEMENT PROGRAM (PMG)

118 EXAMPLE Objectives of Management Development Stages Areas Systems I II III IV V VI VII VIII IX X Priority Weight Training o high 11% Performance Evaluation o medium 9% Human Resources Hygiene, Security and Improved work environment o medium 10% Integral Customer Service system o high 15% Customer Service e-goverment o high 11% Internal audits o high 11% Territorial management o high 12% Management Planning and Control Planning/ Management Control low 5% Government Procurement o medium 10% Financial Management Financial Accounting o medium 10% Gender focus o low 5% o MANAGEMENT IMPROVEMENT PROGRAM (PMG)

119 Created by the Budget Law Budding fund finances the elaboration of proposals to solve public management problems in some agency or program. The proposals can be presented by public agencies or by universities, consulting firms or private consultants. In January 2008 was the first bidding process. 66 proposals were submitted and 17 were selected to be developed during the first semester of from public agencies and 10 from universities, consulting firms and private consultants. The budget 2009 considers resources to implement 5 of these 17 proposals. 19 PUBLIC MANAGEMENT MODERNIZATION FUND

END OF PRESENTATION / Sistema Evaluación y Control de Gestión