Methodological and Analytical Issues Gaia Dallera 6 June,

1. Introduction 2. Defining Governance 3. Governance Data Sources for the WGI 4. Constructing the Aggregate WGI Measures 5. Using and Interpreting the WGI Data 6. Analytical Issues 7. Conclusions 2

 The WGI are a research project to develop cross-country indicators of governance.  We will focus on the methodology and key analytical issues relevant to the overall WGI project. 3

 “the traditions and institutions by which authority in a country is exercised…...This includes three areas: 1. The process by which governments are selected, monitored and replaced. 2. The capacity of the government to effectively formulate and implement sound policies. 3. The respect of citizens and the state for the institutions that govern economic and social interaction among them”. 4

No.DimensionIndicatorsCode 1.Selected, Monitored and Replaced Voice and AccountabilityVA 2.Selected, Monitored and Replaced Political Stability and Absence of Violence/Terrorism PV 3.Formulate and implement sound policies Government EffectivenessGE 4.Formulate and implement sound policies Regulatory QualityRQ 5.Respect of citizens and state for institutions Rule of LawRL 6.Respect of citizens and state for institutions Control of CorruptionCC 5

These indicators are based on several hundred variables optained from 31 different data sources capturing governance perception Availability of the underlying data from (with few exceptions) the individual data sources (transparency and replicability) The WGI data sources reflect the perceptions of a very diverse group of respondent: There are four categories: 1. “Surveys” of domestic firms and individuals with first hand knowledge of the governance situation 2. “Public sector data providers” 3. Commercial business information providers 4. Non-governamental organizations 6

 An important qualification is that the sources of data provide different coverage across countries (some cover a majority of countries and some cover small groups of countries).  Each individual variable are not comparable across countries 7

8

 Combines the six aggregate governance indicators using a statistical tool known as the “Unobserved Component Model” (UCM)  The underlying premise of this statistical approach is straightforward – each of the individual data sources provides an imperfect signal of some deeper underlying notion of governance that is difficult to observe directly  This means that we face a signal extraction problem:  How can we isolate an informative signal about the unobserved governance component common to each individual data source?  How can we combine the many data sources to get the best possible signal of governance in a country?  Therefore we construct a linear regression model in order to: Standardize the data from these diverse sources into comparable units; Construct an aggregate indicator of governance as a weighted average; Construct a margin of error that reflects the unavoidable imprecision in measuring governance. 9

 Regression model y kj =a k +b k (g j +e jk )  Different thing to note about this regression model is that the error term is considered as an independent variable.  “a” and “b” are parameters that reflect the fact that different sources use different units to measure governance.  The error term and precisely its variance captures two sources of uncertainty. 10

 We consider the estimate of governance as a weighted average of the re-scaled scores for each country. E [ g j I y j1,…., y jk ]= ∑ w k  The re-scaling puts the observed data from each source into the common units we have chosen for unobserved governance.  The larger the weights, the smaller the variance of the error term.  A crucial observation is that there is an unavoidable uncertainty around this estimate of governance. 11 y jk - a k b k k=1 K

 We report the aggregate WGI measures in two ways: Standard normal units of the governance indicator ranging from -2.5 to Percentile rank terms ranging from 0 (lowest) to 100 (highest). 12

13

14

 Size of the confidence intervals varies across countries.  Resulting confidence intervals are substantial relative to the units in which governance is measured. 15

 Many of the small differences in estimates of governance across countries are not likely to be statistically significant at reasonable confidence intervals, since the confidence intervals are likely to overlap.  Example of overalapping confidence intervals: IndicatorCountryCoefficientScale Control of CorruptionJamaica to 8.0 Control of CorruptionPeru to

 This is not to say that aggregate indicators cannot be used to make cross country comparisons.  There are in fact many pair-wise country comparisons that are statistically significant.  63% of the pairwaise comparisons the confidence intervals do not overlap 17

 For example: 2009 Control of Corruption indicator covers 211 countries; Total of 21,155 pair-wise comparisons. % of Comparisons Confidence IntervalsSignificance 63%90% confidence intervals do not overlap Signals statistically significant differences in the indicator 73%75% confidence intervals do not overlap Same as above 18

 UCM has three main advantages: It maintains some of the cardinal information in the underlying data. It provides a natural framework for weighting the re-scaled indicators by their relative precision. It naturally emphasizes the uncertainty associated with aggregate indicators. In fact it formalizes the signal extraction problem and by doing that it provides a rationale for a more inclusive approach to combining data from different types of sources. 19

 Unbalanced vs balanced samples: Refers to the fact that WGI use all available data sources for all countries as opposed to using only those data sources that cover all countries and in all time periods (permitting balanced comparison). 20

 This choice is based on the view that perception data has a particular value in the measurement of governance because: Agents (citizens and enterprise) base their action on their perceptions; In many areas of governance there are few alternatives to relying on perception data e.g. corruption ; The “jure” notion of laws often differs from the de-facto reality. 21

 Potential problems: 1. Interpretation of subjective data Perception data is imprecise; But all measures of governance are imprecise proxies for the broader concept; Therefore it underscores the importance of using empirical methods and taking seriously the extent of imprecision. 2. Systematic biases in perception data on governance introduced by: Different types of respondents may differ systematically in their perceptions of the same underlying reality; The ideological orientation of the organization providing the subjective assessments of governance. Possibility that different providers of governance perceptions data rely on each other’s assessments (correlated perception errors). BUT THERE IS LITTLE EVIDENCE OF SUCH BIASES!!!! 22

 WGI reports six dimensions of governance covering 200 countries since  Updated annually.  Based on hundreds of variables from many different data sources. 23

 Due to the inherently unobservable nature of the true level of governance in a country, any observed measure is only a proxy.  Consequence is that our estimates are subject to non-trivial margins of error. 24

 Don’t over interpret small differences in performance (across countries or over time).  Presence of errors does not mean that the WGI cannot be used to make meaningful comparisons.  Estimation of and emphasis on such margins of error is intended to enable users to make more sophisticated use of imperfect information. 25

 In fact it is possible to make meaningful comparisons cross-country and over time.  Almost 67% of all cross-country comparisons in 2009 result in highly significant differences at 90% confidence intervals.  More than one quarter of countries show a significant change in at least one of the six WGI measures during the decade. 26

THANK YOU 27