1. Measurement

2. What is measurement? “the assignment of a value on a variable to a unit of measurement in accordance with an operational definition” (Kleinnijenhuis 1999: 83 in Pennings et. al. 1999)

3. Components of measurement –A value: a categorization or number (e.g. whether or not a country is a democracy or not; or level of democratization) –A variable: a characteristic that can vary in value among observations –A unit of measurement: an observation/case (e.g. a country in a particular year) –An operational definition: a definition of a concept that facilitates the assignment of a value to an observation (e.g. what countries would be classified as democracies)

4. From concept to operational definition Reduce abstraction to facilitate measurement E.g. Political party positions / electoral appeals –Concept: the salience of themes to political parties –Operational definition: the percentage of a manifesto devoted to each of a set of defined themes (including “free enterprise”, “welfare state expansion”, “expansion of military capabilities”)

5. Levels of measurement Nominal –Categories with no rankings (e.g. types of democracies / types of political parties / ethnic groups) - MECE Ordinal –Some observations score higher/lower than others, but we do not know how much more or less (e.g. Left-Right position of parties / intensity of attachment to ethnic group) Interval –The differences between the values of the observations are meaningful Ratio –There is a meaningful zero point (e.g. numbers of decisions taken by referenda/ cross border trade)

6. Guidelines for constructing measures Reveal assumptions implicit in operationalising a concept: often nothing intrinsic in a concept that implies a particular level of measurement Maximise information: Wherever sensible, aim for as high a level of measurement as possible Address trade-off between comparability and descriptive richness

7. Lijphart’s (1977) typology of democratic systems Structure of society Elite behaviour HomogeneousPlural CoalescentDepoliticized: Austria (1966-) Consociational: Belgium, Netherlands, Switzerland, Austria (1945-66) AdversarialCentripetal: Finland, Denmark, UK, USA, Norway, Sweden, Germany Centrifugal: France, Italy, Canada

8. Evaluating measurement Validity and reliability Valid: the measure does measure the concept it purports to measure? –Face validity –Correlational validity (internal validity) –Predictive validity (external validity)

9. An example of a correlational validity test Compared key-informants judgements on “what were the controversial issues” in a particular decision situation with documentation –Key informants identified five controversial issues in decision making on a legislative proposal in the EU (the tobacco directive) –To what extent and in what way were these related to the controversial issues identified through analysis of documentation? For details see: http://www.dhv-speyer.de/tkoenig/DEU_manuscript_Oct2005.htm

10. E.g. of correlational validity test cont. Related to experts’ issues? Issues from documents Actors with positions on point of contention. Average (range) Number of Council meetings at which point was raised. Average (range) Related87.9 (4-17)3.0 (1-6) Unrelated234.8 (2-9)1.7 (1-4) 31 controversial issues on the basis of documentation 5 issues from the informants How many of the 31 were related to the 5? What distinguished document-sourced issues that were related to informants’ issues from those that were not?

11. Reliability Reliable: repeated measures on the same cases give the same results –Intra observer reliability –Inter observer reliability

12. Another example of testing validity and reliability The enactment of election pledges in a coalition system of government: The Netherlands (Thomson, R. 2001. The programme to policy linkage: The fulfilment of election pledges on socio-economic policy in the Netherlands, 1986-1998. European Journal of Political Research 40: 171-197)

14. Face validity Do the election pledges appear to be substantively important? Do the patterns of fulfilment match with qualitative descriptions of the main policy events in the period?

15. Face validiity

16. Reliability tests Inter-coder reliability –Did other readers of the same election manifestos identify the same statements as pledges? Inter-coder reliability –Did subject-area specialists identify the same pledges as unfilfilled or fulfilled?

17. An inter-coder reliability test a / (a+b+c) = 199/226 =.88 Statement identified as pledge by Thomson? YesNoTotal identified by 2 nd coder Statement identified as pledge by 2 nd coder? Yes199 (a)13 (b)212 No14 (c) Total identified by Thomson 213

18. Measurement error Aim at unbiased and efficient measures Bias: systematic error –We either overestimate or underestimate the true value Inefficiency: Non-systematic error –On average, we estimate the true value correctly, but with considerable variation

19. The ideal is unbiased and efficient estimates True value Measure/ estimate Unbiased, but inefficient Unbiased and efficient Biased

20. The impact of bias on inference Bias / systematic error –Consistent over or underestimation of the true value –E.g. consistent overestimation of annual income Leads to bias in description If it affects all cases equally, does not lead to bias in estimating causal effects

21. The impact of inefficiency on inference Hampers description, although descriptions are correct on average Hampers explanation, but the impact differs depending on whether inefficiency pertains to the IV or DV Inefficiently measured DV – estimates of causal effects will be difficult, but correct on average (unbiased) Inefficiently measured IV – estimates of causal effects will be difficult and incorrect on average (i.e. biased)

22. Why inefficiently measured DVs do not bias estimates of causal effects IV: unemployment DV: violence

23. Why inefficiently measured IVs bias estimates of causal effects IV: unemployment DV: violence