1. Varieties of Democracy Data
Incorporating Measurement Uncertainty

2. V-Dem’s online graphs display uncertainty
Russia:

3. What does this uncertainty mean?
How does it affect estimates when you use the data?
How can you incorporate the uncertainty into your estimates?

4. A subtle distinction
We treat measurement error as a property of raw coder scores.
The measurement model purges the data of measurement error as much as possible, given certain assumptions.
What is left is measurement uncertainty: a distribution of possible true scores.

5. Example of measurement error
Coders disagree.
The difference between the “true”
score and each coder’s score is
that coder’s measurement error.

6. We can estimate the “true” score
To recap from the data release workshop,

7. We assume that coders/raters perceive a continuous underlying reality.

8. However, raters who perceive the same reality. . .

9. . . .but with different ordinal thresholds. . .

10. . . .can express their perceptions differently.

11. The result:

12. The model estimates difficulty thresholds, assuming
Global mean thresholds are between -2 and 2 (uniformly distributed)
The mean country thresholds are allowed to vary around the global thresholds, with a standard deviation of 0.2
Coder thresholds are allowed to vary around their country’s thresholds, with a standard deviation of 0.2
What this looks like:

13. An example for v2svinlaut: International autonomy.
Black: posteriors of global mean thresholds

14. An example for v2svinlaut: International autonomy.
Gold: 20 posteriors for all country thresholds

15. An example for v2svinlaut: International autonomy.
Blue: posteriors of coder thresholds for Denmark
Red: posteriors of coder thresholds for Venezuela

16. So
The measurement model purges the data of measurement error that can be attributed to
Random coder error
Questions having different thresholds
Countries having different thresholds on the same question
Coders for the same country having different thresholds
And some other assumptions
We therefore have good estimates of the true scores, but the estimates are distributions of values, not point estimates
The point estimates we report are the means of these distributions.
They should be correct on average, but they don’t contain all the information we have.
Large samples from these distributions are available at V-Dem’s archive at CurateND.

17. Does this guarantee that our data is unbiased?
No. If coders for a given country share the same bias, the MM won’t detect it.
However,
Lateral coding should make shared biases less likely.
Validation testing so far (e.g., on corruption indicators) has turned up no significant and sizable bias associated with coder characteristics
Age
Gender
Highest degree
Country of origin or residence
Country of education
Government employee
Support for free market
Support for electoral democracy

18. How does measurement uncertainty affect analyses using the data?

19. Here are point estimates for Venezuela, 1900-2012, on two variables.
This relationship implies that there is no uncertainty about what these values are.

20. But if we look at 900 estimates of the true values, we get a more complete sense of what the relationship is.

21. Today you’ll learn how to estimate relationships many times for different draws from these distributions and then combine those estimates into summary estimates.