Conceptualizing and Measuring Institutions: A View from Political Science December 15, 2007 Marcus KurtzAndrew Schrank Ohio State UniversityUniversity of New Mexico

Outline Measuring governance: objective v. subjective indicators. Limits to subjective indicators: --Conceptualization: a moving and blurry target. --Operationalization: pro-business bias and systematic error. --Measurement: non-independence and random error. Objective alternatives?

Objective v. Subjective Measures of Governance Objective measures of concrete outputs Tax ratios (Organski & Kugler 1980) Tax structures (Kling 1968; Krasner 1985) Public service provision (Migdal 1988; Putnam 1992) Subjective (perceptions-based) measures of institutions Bureaucratic quality from ICRG (commercial risk analysts) Property rights indices from Heritage Foundation (NGO staff) Bribery data from the IMD (firm surveys) Corruption perceptions from TI (aggregated assessments) Aggregate perceptions-based measures: the state of the art World Governance Indicators (World Bank)

Objective v. Subjective Measures of Governance Objective measures of concrete output Tax ratios (Organski & Kugler 1980) Tax structures (Kling 1968; Krasner 1985) Public service provision (Migdal 1988; Putnam 1992) Subjective (perceptions-based) measures of institutions Bureaucratic quality from ICRG (expert assessments) Property rights indices from Heritage Foundation (NGO staff) Bribery data from the IMD (firm surveys) Corruption perceptions from TI (aggregated assessment) Aggregate perceptions-based measures: the state of the art World Governance Indicators (World Bank) Our focus: WGI rule of law (RL)

Conceptualization: a moving target? What is the rule of law? Citizen respect for the state and its rules (WGI 1999) Quality of contract enforcement, the police, and the courts and the likelihood of crime and violence (WGI 2005) Extent to which agents have confidence in and abide by the rules of society and the above (WGI 2006). Norms of limited governance (Kaufmann et al. 2007) Does shifting conceptualization have consequences? The Italian example Problems of multiple understandings of the term Evaluating measurement hinges on conceptualization.

Measurement: a pro-business bias? Questions: What factors into RL? Crime, courts, cops and property rights. Crime: Expert and survey data tend to privilege property crime Property rights: Public officials use expropriation to fight crime and corruption (i.e., Korea; RICO) Courts and cops: Indicators ignore other law enforcement agencies (i.e., regulatory authorities) Sources: Whose opinions matter? Most sources are businesses or their advisers. Why is this crucial? Are businesspeople’s opinions representative?

Measurement: a pro-business bias? Questions: What factors into RL? Crime, courts, cops and property rights. Crime: Expert and survey data tend to privilege property crime Property rights: Public officials use expropriation to fight crime and corruption (i.e., Korea; RICO) Courts and cops: Indicators ignore other law enforcement agencies (i.e., regulatory authorities) Sources: Whose opinions matter? Most sources are businesses or their advisers. Why is this crucial? Are businesspeople’s opinions representative? Businesspeople differ from others in evaluating the Rule of Law. QuestionResponsesBusiness Trust in the judiciary 1 = much trust; 2= some trust; 3= little trust; 4 = no trust Odds ratio = 1.15 (p <.010) Trust in the police Odds ratio = 1.23 (p <.001) Source: Latinobarometer data used in WGI rule of law indicator. Ordered logit models; country dummies suppressed. NO

Measurement: Non-Independence and Noise Worldwide Governance Indicators provide estimates of uncertainty What is the reliability of the rule of law (RL) measure?

Measurement: Non-Independence and Noise Worldwide Governance Indicators provide estimates of uncertainty What is the reliability of the rule of law (RL) measure? Rule of Law, 1996 KKM SEs Rule of Law, *KKM SEs Rule of Law, *KKM SEs Correlation (median)0.818 (median)0.716 (median) Confidence Interval [0.876 – 0.934][0.752 – 0.867][0.607 –0.788] All Countries Below median income Correlation (median)0.538 (median)0.398 (median) Confidence Interval [0.581 – 0.821][0.343 – 0.701][0.148 –0.600] What are the consequences of noise for growth models?

Conclusion: bringing objective reality back in? Subjective indicators and the problem of causal mechanisms. Analogy with self-reported health data. An objective alternative would: --be issue specific; --look at measurable outputs; --include data on inputs; --differentiate rule of the game from their application. Example: Labor law enforcement in Latin America.

Conclusion: bringing objective reality back in? Subjective indicators and the problem of causal mechanisms. Analogy with self-reported health data. An objective alternative would: --be issue specific; --look at measurable outputs; --include data on inputs; --differentiate rule of the game from their application. Example: Labor law enforcement in Latin America. Slope as ROI; residuals as quality of enforcement Explaining residual informality by the intensity of enforcement: