May 1, 2008 Marcus KurtzAndrew Schrank Ohio State UniversityUniversity of New Mexico

Aggregation: Promise and Perils Conceptualization: Logically Prior “Governance” and “Rule of Law” Systematic Measurement Error (questions of ‘validity’) Bias Incomplete Domain of Measurement Inconsistent conceptualization Consequences for Aggregation Consequences for Scholarship and Practical Use

Conceptualization makes it possible to assess bias (i.e., systematic measurement error) Conceptualization is crucial to theory formation (e.g., if, for example, both policy preferences and institutional characteristics are included in a definition, then assessment of the causal linkages between them is impossible) Concepts should not be empirically derived from the data

Governance “[T]he traditions and institutions by which authority in a country is exercised. This includes the process by which governments are selected, monitored, and replaced; the capacity of the government to effectively formulate and implement sound policies; and the respect of citizens and the state for the institutions that govern social and economic interactions among them [italics added].” Kaufman and Kraay (2008,4) Implications Simultaneous measurement of formal and informal institutions Assumes “sound policy” is consensual knowledge and indistinct from institutions Policy formulation and implementation are conceptually tied Assumes that citizen “respect” for institutions is conceptually the part of the definition of an institution

Definitions of “Rule of Law” Citizen respect for the state and its rules (1999) Quality of contract enforcement, the police, and the courts and the likelihood of crime and violence (2005) Extent to which agents have confidence in and abide by the rules of society and the above (2006). Norms of limited governance (2007) Does shifting conceptualization have consequences? The Italian Example Problems of multiple understandings of the term Evaluating measurement hinges on conceptualization.

Questions: What factors into RL? Crime, courts, cops and property rights. Crime: Expert and survey data tend to privilege property crime Property rights: Public officials use expropriation to fight crime and corruption (i.e., Korea; RICO) Courts and cops: Indicators ignore other law enforcement agencies (i.e., regulatory authorities) Sources: Whose opinions matter? Most sources are businesses or their advisers. Why is this crucial? Are businessperson’s opinions representative?

Questions: What factors into RL? Crime, courts, cops and property rights. Crime: Expert and survey data tend to privilege property crime Property rights: Public officials use expropriation to fight crime and corruption (i.e., Korea; RICO) Courts and cops: Indicators ignore other law enforcement agencies (i.e., regulatory authorities) Sources: Whose opinions matter? Most sources are businesses or their advisers. Why is this crucial? Are businessperson’s opinions representative? Business elites differ from others in evaluating the Rule of Law. QuestionResponsesBusiness Trust in the judiciary 1 = much trust; 2= some trust; 3= little trust; 4 = no trust Odds ratio = 1.15 (p <.010) Trust in the police Odds ratio = 1.23 (p <.001) Source: Latinobarometer data used in WGI rule of law indicator. Ordered logit models; country dummies suppressed. NO

Incomplete Domain Measure property-related issues (crime/expropriation); little emphasis on the enforcement of other laws likely to be important for “the respect of citizens and the state for the institutions that govern social and economic interactions among them” Health, Safety, Environmental, Labor, Civil Rights, and Development regulations barely (if at all) covered. Unlikely to correlate strongly and positively with property rights protection. Inconsistent Conceptualization How can inclusion of the above be squared with the “norms of limited governance” (a policy choice) that are included in the definition of rule of law?

Aggregation rewards correlation Correlation due to similar random errors (example: two input sources rely on the same [unbiased, but noisy] information to rate countries) Error estimates potentially too small, still likely smaller than most individual data sources. Correlation due to similar biases If “independent” data sources agree because they involve similar sampling biases, halo effects, respondent policy preferences, or incomplete sampling of the conceptual domain, inter alia, then: A. We cannot be sure what we are measuring (institutions, policy, prior performance, some unknown combination, etc.). B. Aggregation only compounds the problem: (1) Similarly-biased sources assigned additional weight relative to unbiased ones. (2) Aggregation no longer reduces overall error – instead it at best trades reliability for validity. At worst, both are worsened.

Consider a hypothetical strong, consistent and positive relationship between measures of institutions (e.g., rule of law, government effectiveness) and growth. Without clarity on the conceptual underpinnings we cannot know whether this is a consequence of 1. The actual positive institutional effect of RL/GE; 2. Herd behavior on the part of investors who invest based on their purchase of the same sorts of (biased) data that go into the RL or GE measure [i.e., self-fulfilling prophecy]; 3. Policy choices that have been conceptually conflated into definition (e.g., “norms of limited government”); or 4. Correlates of institutions (e.g., wealth, human capital, etc.)