1. INNOVATIONS IN GOVERNANCE MEASUREMENT Public Accountability Mechanisms (PAM) Initiative Francesca Recanatini & Stephanie E. Trapnell April 26, 2013 1

2. Outline  Introductions  Objectives and potential outputs  ISPMS, AGI, and behavioral dimensions of measurement  Public Accountability Mechanisms  De jure and De facto  “Big ideas” for today 2

3. Objectives of the Workshop  To discuss innovations in the measurement of “de facto” aspects of governance focusing on:  the gaps in existing data coverage,  the strengths and weaknesses of recent data collection methodologies, and  the sustainability of these efforts.  If possible, to identify a “core” set of governance areas and related indicators (both existing and potential) 3

4. Outputs 4  Menu of governance indicators and efforts  Draft note on methodologies  Annotated bibliography of sources  Technical expert group (TEG)

5. A proposal and a new way forward ISPMS, AGI, and behavioral impacts 5

6. The challenge  To measure performance of institutions, and in particular, public sector institutions  To identify areas for institutional reform  To evaluate the impact of change in policies at the country and local level 6

7. Complementary sets of indicators  Measurement of the performance of Public Sector Management (PSM) systems in the middle of the public sector results chain.  Measurement of the way in which the government is held to account through political and non- executive institutions and directly by the public. 7 Indicators of the Strength of Public Management Systems (ISPMS) Actionable Governance Indicators (AGIs)

8. Public sector results chain within a broader governance environment 8 Broader Governance Environment

9. Moving forward: Utility criteria + feasibility CriterionDefinition 1. Action-worthyWe know (or strongly believe) that they contribute to results 2. ActionableThey are amenable to government action and project interventions 3. BehavioralFocus on function, not form 4. ReplicableGenerated transparently and can be reproduced by others …and 2 “feasibility” criteria. CriterionDefinition 1. Country coverageAvailable for a minimum of 20 countries 2. Time seriesCollected repeatedly

10. Selected Examples  1. “Does the executive’s budget or any supporting budget documentation present expenditures for the budget year that are classified by administrative units?” (Open Budget Survey)  2. Does a legislative committee hold public hearings on the individual budgets of central government administrative units in which testimony from the public is heard? (Open Budget Survey)  3. Tax payers access to information on tax liabilities and administrative procedures (PEFA) 10

11. De jure vs De facto An illustration: Public Accountability Mechanisms 11

12. Public Accountability Mechanisms (PAM) de jure indicators Financial disclosure (interests, assets, income) 217 indicators 2008, 2012 Freedom of information 37 indicators 2010 Conflict of interest restrictions 124 indicators 2011 Immunity protections 56 indicators 2012 12

13. Outputs of de jure data 13  Library of laws  Analytical publications on the design of public accountability mechanisms  Data, including qualitative and quantitative datasets, country profiles, and descriptive statistics  Country reports on enabling governance environment ….But how do we know what happens in practice?

14. Freedom of information systems in practice: public sector management functions  1.1 Facilities  1.2 Data/Records  1.3 Human resources  1.4 Financial  1.5 Policy  2.1 Demystification*  2.2 Responsiveness  2.3 Appeals  2.4 Proactive Disclosure 14 1.0 Administrative Functions2.0 Disclosure Functions * clarity about government processes, rules, and decisions

15. Freedom of information systems: Immediate impacts/Intermediate outcomes 15  3.1 Public engagement: Extent to which the public understands, believes in, and engages with the freedom of information process.  3.2Government commitment: Extent to which the government supports the freedom of information regime, including efforts to establish participatory decision-making  3.3 Administrative culture: Extent to which the bureaucratic culture has shifted from principle of secrecy to openness.  3.4Operational efficiency: Extent of improvement in operations and decision-making within the organization.

16. How do you measure impacts in a governance “ecosystem” 16  Participation of civil society organizations and citizens  Accountability and enforcement by parliament, judiciary, police, supreme audit institution and the ombudsman  Political will that enables meaningful change, including protection from retribution, lack of clientilism, and an environment that maintains appropriate incentives for civil servants  Foundational support mechanisms found in public sector systems, such as freedom of information frameworks, conflict of interest rules, financial disclosure practices, and human resources and policy management.

17. Methodologies matter 17  Surveys: validation issues, response problems  Expert assessment: time-consuming, with degrees of subjectivity  Household surveys: costly, time-consuming, logistical difficulties  Economic/social indicators: collected by whom? National statistics agencies, multi-laterals, NGOs, donors….  Outcome mapping/process tracing: lengthy, involved, requires local capacity

18. So where do we start? 18 MethodologyIndicatorsConcepts

19. “Big ideas” for today 19  Actionability and action-worthiness  Change in behavior and not in function  Sustainable measurement practices  Fuzzy concepts and methodologies  Practical applications of data  Challenges in de facto measurement

20. Some emerging trade-offs (or complementarities?)  Global focus versus sub-national focus?  Existing data or new data?  Focused on public sector or broad governance issues?  Perception-based versus objective data? Or both?  Using a single tool or multiple tools?  Detailed focus on a specific case/country 20

21. Some emerging issues Areas of focus?  Tax administration  Human resources  Rule of law Looking forward  Clarify the focus of the effort and link to a clear research question  Integrate the issue of sustainability  Benchmarking  Understanding strengths of different types of respondents  Clear definition of the context  Dissemination and wide use of the data (and need for a strategy)  New set of tools for certain phenomena (high level corruption) 21

22. PAM website: www.agidata.org/pamwww.agidata.org/pam AGI data portal: www.agidata.orgwww.agidata.org Metrics & Accountability: http://go.worldbank.org/H1K725TJV0 http://go.worldbank.org/H1K725TJV0 Thank you! 22

23. 23 InputsOutputs Rules (Institutional Arrangements) Organizational Capacities Governance System Performance Outcomes The Missing Middle Resources invested in projects to deliver its outputs. Examples: Funding, contracts, materials Goods and services produced by the project. Examples: Surveys and Trainings conducted, Laws revised, Agencies established Desired state of well-being -- a set of conditions, experiences or behaviors – that is the goal for change or improvement. Examples: Maternal/infant mortality rates, Standardized test scores of K-8 students Actionable Governance Indicators (AGIs) focus on specific and narrowly-defined aspects of governance.

24. 24