The “Tools of the Trade”: An Overview of Diagnostic and Assessment Instruments December 1, 2003 Francesca Recanatini, WBI

Outline of the Session A working framework to select among tools Conceptual design Empirical tools Implementation process Sampling and Field work Analysis and use of the data A few country-specific illustrations

Objective Which are the key elements for a governance assessment? Which empirical tools and approaches are already available? How can we select among them? How can such assessments be used for policy purpose? How to choose among governance tools?

Governance assessment: one or many approaches? The characteristics of a governance assessment are a function of the objective of the assessment

Key starting points 1. What is the purpose of the assessment? Research and analysis Awareness raising Policy and Action planning Capacity building Monitoring

Key starting points 2. What is the focus of the assessment? Governance as a whole Corruption Performance of a specific agency/sector Quality of a specific public service delivered

Suppose we have determined…. The final purpose of the assessment The focus of the assessment What next?

An example – Peru 2002 Issue: the government wanted to monitor progress in terms of Transparency of public administration activities Civil society “ participation ” and voice Quality of public services

Peru 2002, cont. Purpose of assessment: monitoring Focus of the assessment: Transparency Citizens ’ “ Participation ” and Voice Quality of public services What next?

Existing Empirical Tools BEEPS IGR Public Official surveys PET QSDS Score Cards Investment Climate Surveys EC Audits PER CFAA CPAR GAC Case Studies HIPC Exp. Tracking ROSC

Which are the key elements of a Governance Assessment? Four dimensions: Conceptual Empirical Process / Capacity Building Analytical and Policy

Conceptual dimension Clear definition of the variable we focus on and its manifestations Translation of the definition into observable and measurable components Selection of methodological approach Understanding of the links between governance and Performance outcomes Development outcomes

Linking the Tools to the Blueprint PER HIPC E.T. ROSC CPAR EC Audits CFAAIGR & GAC & Governance Cross- Country Ind. BEEPS & INVEST. CLIMATE SCORE CARDS QSDS Public Official Surveys PETs

Examples of diagnostic tools are widespread … and many overlap Public Expenditure Instruments Public Expenditure Reviews (PER) Fiscal ROSC Country Procurement Assm’t Reviews (CPAR) HIPC Expenditure Tracking Assessments European Commission (EC) Audits Governance Surveys (e.g., IGRs, Anti- Corruption, Public Officials, Household Incidence, Poverty, etc.) Country Financial Accountability Assessments (CFAA) Policy & Expenditure Analysis Budget Formulation & Expenditure Programming Monitoring & Reporting Budget Execution Oversight Illustrative

Policy framework Govt. program PRSP Sector strategies etc… Budget allocation Outturn Timely disbursements in accordance with budgeted allocations OutputsImpactOutcomes - Poor institutions of budget management - Political economy factors Nontransparent process - Poor reporting on execution - High level of aggregation - Discretion in allocation Weak accountability - poor service delivery Difficult to assess - Household behavior - Social institutions PUBLIC EXPENDITURE TRACKING AND SERVICE DELIVERY SURVEYS Unclear policy framework Weak management information systems - limited coverage - poor data quality - late and scattered reporting New Concerns

Conceptual dimension, cont. Finding answers may require single or multiple methods and data forms The methodological approach can be a combination of different methods (for example, qualitative, quantitative or mixed) To each method corresponds a set of empirical tools that we can use Data can also be qualitative and/or quantitative For more information on alternative methods

Examples of Existing Empirical Tools Qualitative method Budget use monitoring Video Observations Judicial Investigations Quantitative method Investment Climate Surveys QSDS Public official surveys PETs Mixed method Governance Diagnostic Surveys Score Card approach PER

Empirical dimension Focus on institutions vs. individuals Experiential vs. perception data One vs. many types of respondents Standard vs. customized empirical tools Definition of sample and field work details Open end vs. close end questions

Citizens Government Officials Enterprises Civil Society Private Sector The State Linking the Tools to the Respondents PET QSDS PER CFAA CPAR Score cardsGAC IGR BEEPS INV. CL.

Examples of variables measured USERS/HOUSEHOLDS Quality of specific public services Cost and time to obtain a service Information available on basic rights Quality of public agencies Experience with inappropriate procedures and behavior

Examples of variables measured ENTERPRISES Quality of specific services and procedures Cost and time to comply with permits and licenses Information available on basic rights Quality of public agencies Experience with inappropriate procedures and behavior

Examples of variables measured PUBLIC OFFICIALS Quality of the rules and procedures Transparency of budget and employment decisions Information available on procedures Quality of management Experience with inappropriate procedures and behavior within their office

Process / Capacity Building dimension To increase impact and sustainability: Consultative and participatory approach to discuss purpose, use and features of the assessment Engage local NGOs and academic institutions to adapt/revise tools Public dissemination of results Joint design of policy recommendations

Challenge: poor governance and corruption 1. Establishment of Steering Committee 2. Diagnostic surveys + analysis 3. Draft of the NAS 4. Public dissemination + discussion 5. Revision of the NAS 6. Implementation by Government 7. Monitoring and Evaluation of NAS WBI Technical Assistance Key Partnership: Government + Civil Society Country Implemented A few Illustrations Guatemala Highly fragmented civil society Joint effort to build consensus Surveys to be launched in Dec. Sierra Leone Strong commitment (civil society, state, donors) => surveys + report completed. Results use for draft strategy and reform projects Honduras CNA: report and strategy to newly elected gov (January 2001); integration of strategy in the government plan

Analytical and Policy dimension Distill key links between manifestations of governance and: Quality of services Growth Specific characteristics of public sector Results could be used as one input for policy purpose

Key dimensions for analysis 1.Identify both weak institutions (in need of reform) and strong institutions (example of good governance) 2.Unbundle corruption by type – administrative, capture of the state, bidding, theft of goods and public resources, purchase of licenses and regulations

Key dimensions… Cont. 3.Assess the cost of each type of corruption on different groups of stakeholders 4.Identify key determinants of good governance 5.Develop policy recommendations

In sum, how to select among instruments? PER? Score cards? BEEPS? IGR? PET? QSDS? CFAA? CPAR? GAC? Case study? …..? Pub. Officials? HIPC Ex.Tr.? EC Audits? ROSC?

Governance Assessment Analysis & use Empirical tools & sample Conceptual dimension Implementation process Use a working framework….

..and remember the key dimensions for a ‘tool comparison’ Conceptual framework Approach Objective Subjective Measuring precision Cost effectiveness

Key dimensions, cont. Comparability of the data Across countries Over time Across empirical tools Final user of the data Different agencies Different stakeholders Type of respondent

Key dimensions, cont. Effectiveness for judicial actions Ability to identify general challenges Ability to identify priorities for reform Quality of information on: Specific institutional dimension Specific subject

Peru 2002 Purpose of assessment: monitoring Final users: government and civil society Key feature: Comparability across time Ability to identify progresses Type of information needed: agency- specific Approach: objective, and based on citizen ’ s feedback

Peru 2002 Conceptual dimension Transparency in the management of resources Quality of basic health and education services Quality of complaint and feedback mechanisms Empirical Tool Score card/Questionnaire to households Focus on agency-specific information Objective, experiential data Close-end questions

Peru 2002 Process/Capacity building: Partnership between WBI and with National Statistical Office on methodological issues Data and results publicly available Analytical dimension Monitoring of indices ’ performance over time Link between indices of performance and measures of poverty

Peru 2002 – Decisions taken To develop the following yearly indicators: Index of transparency and civil society participation Index of quality of public services To focus on households/users only To promote a partnership between the National Statistical Agency and citizens

Vulnerabilities of Corruption Reporting: Complaint Mechanisms (as reported by public officials; various countries, )

Extent of corruption (Various countries, 2001)

The thin lines represent margins of error (or 95% confidence intervals) for each Corruption increases inequality

Bribes and quality of services (based on public officials’ response, a Latin American country, 2001)

Corruption Restricts Access to Medical Services: Discouraged Poor Users Due to Bribes (as reported by public service users, 2001) Note: The figure shows the percentage of discouraged users not using medical service because a bribe is too high.

Sierra Leone Roads Transport Authority Corruption imposes barriers to households to access basic social services

SouthNorthEastWestWHOLE Prov. AreaCOUNTRY Corruption in budget Overall corruption Corruption in public contracts Corruption in personnel (2) Accessibility for poor Audit Mechanisms Enforcement of rules Politicization Quality of rules Resources Transparency Citizen voice Meritocracy Governance and corruption indicators by province

EnforcementCitizenWage Auditof rulesResourcesTransparencyVoiceSatisfaction Presidential Affairs Ministry of Finance Quasi-independent institutions Ministry of Trade and Industry Ministry of Internal Affairs Ministry of Energy and Power and NPA Ministry of Foreign Affairs & International Cooperation Ministry of Development & Economic Planning Ministry of Social Welfare, Gender & Children's Affairs Ministry of Health & Sanitation Ministry of Information and Broadcasting Ministry of Education, Science and Technology Ministry of Youth and Sports Ministry of Labor and Industrial Relations Ministry of Agriculture and Food Security Ministry of Local Government & Municipal and district councils Ministry of Transport & Communications Ministry of Works and Public Maintenance Ministry of Lands, Housing & Country Planning Ministry of Mineral Resources House of Parliament Office of the Attorney General, Minister of Justice and Director of Public Prosecutions Traffic Police and Sierra Leone Road Transport Authority Supreme Courts Local Courts SALWACO/GVWC and SIERRATEL SALPOST Sierra Leone Ports Authority SL National Tourist Board and SL Standards Bureau Police and Prisons Post-conflict institutions WHOLE COUNTRY Governance and Corruption Indicators by Institution

WBI Governance on the Web About Governance Diagnostics and Statistical Capacity Building: Governance Diagnostic Surveys Country Sites: Worldwide Governance Indicators : The Business Environment and Enterprise Performance Survey (BEEPS) : The Business Environment and Enterprise Performance Survey (BEEPS II) 2002: Courses and Surveys: Governance Diagnostic Capacity Building: Step by Step Guide to Governance Diagnostic Empirical Tools Implementation: