“Dirty Pool” Revisited: When Less is More Robert S. Erikson, Pablo M. Pinto, Kelly T. Rader Department of Political Science Download paper at:

Do Democracies Trade more with other Democracies? How the literature approaches this question: Convention: –Measure units as dyad-year e.g.: Sierra Leone-Ecuador 1997 –Measure democracy as democracy score for the least democratic dyad member –Relate democracy to trade levels while controlling for other stuff Main issues: –Fixed effects? –Lagged dependent variable on RHS? –Dyads as unit not an issue.

Do Democracies Trade more with other Democracies? Methodological issues discussed in 2001 symposium in IO 90,000+ dyad-years, but only up to 115 countries over 30 years We present a cautionary tale about using dyads as units of analysis Large data set with a massive N can create overconfidence in hypothesis testing

Symposium on Research Design and Methods in IR: –Editors' Introduction (pp )Editors' Introduction Peter Gourevitch, David A. Lake –Dirty Pool (pp )Dirty Pool Donald P. Green, Soo Yeon Kim, David H. Yoon –Clear and Clean: The Fixed Effects of the Liberal Peace (pp )Clear and Clean: The Fixed Effects of the Liberal Peace John R. Oneal, Bruce Russett –Throwing out the Baby with the Bath Water: A Comment on Green, Kim, and Yoon (pp )Throwing out the Baby with the Bath Water: A Comment on Green, Kim, and Yoon Nathaniel Beck, Jonathan N. Katz –Proper Nouns and Methodological Propriety: Pooling Dyads in International Relations Data (pp )Proper Nouns and Methodological Propriety: Pooling Dyads in International Relations Data Gary King

Our contribution Randomization tests on dyads to infer the correct p-values Switch to nation-years as units of analysis: proportion of trade with democracies as a function of regime Difference-in-difference analysis of change in trading partners after a regime shock Cross-sectional analysis of trade by regime type

GKY (current dollars)

GKY sample (constant dollars)

Expanded Trade Data

Randomization Tests! Simulate the state of the world if the null hypothesis were true Scramble the key independent variable to break the relationship between it and the outcome Estimate coefficients and significance tests using preferred model, knowing the “answer” is zero Repeat at least 1000 times See if the actual estimated effect is rare compared to distribution of spurious effects Sample estimate of Fisher’s exact test (1935)

Randomization Tests! Drop the country labels on the floor and scramble them. Pick them up randomly and reinsert. Albania might become “Bolivia” for example, in all dyads involving true Albania. Rescramble 1000 times. With random country labels and the equation rerun, the data present a distribution around the null outcome (presumably zero, but with complications). T Where in the distribution of observations does the observed value from the actual equation fall? That position in the cumulative distribution yields the p- value with the randomization test.

Advantages of Randomization Tests Unlike parametric tests… –Don’t rely on distributional assumptions about  ij –Don’t rely on distributional assumptions about the test statistic –Standard errors and significance tests determined empirically –Especially useful when theoretical standard errors are hard to derive Like the democratic trade models! But, do assume exchangeability of errors

Figure 1: Randomization tests results

Figure 1 (cont): Randomization tests results

Country-year Models

Figure 2: Dem. trade and regime (residuals) Residual proportion of trade with other democracies by residual regime Data: nation-year; controls: nation and year dummies. Y = X + ε std. err =.044 p =.000 N= 6459 Adj. r 2 =0.011

Figure 3: Democratic shocks Democratic trade at t n relative to democratic shock at t 0

Figure 4: Democratic shocks Residualized democratic trade at t n relative to democratic shock at t 0

Figure 5: Anti-democratic shocks Democratic trade at t n relative to anti-democratic shock at t 0

Figure 6: Anti-democratic shock Residualized democratic trade at t n relative to anti-democratic shock at t 0

Figure 7: Effect of regime shocks Change in democratic trade as a function of democratic shocks 2.6 = mean change in democratic trade per year x 6 years

Figure 8: Country analysis Residual democratic trade over mean regime score (year adjusted) Data: countries Y = 1.10 X + ε st. err =.185 p =.000 N= 154 adj. r 2 =0.183

Conclusions Dyads are inappropriate units of analysis for democratic trade hypothesis: –Randomization tests show hypothesis testing in traditional regression results are too optimistic (p-values up to 41 trillion times larger!) Randomization tests are useful for testing claims, especially when the appropriate parametric test is difficult to figure out Move to nation-year as unit of analysis –We find no regime effect Less is more : –Event and difference in differences analyses of shocks seem to confirm democratic trade hypothesis