C OMPARATIVE P OLITICS T HE C OMPARATIVE M ANIFESTO P ROJECT Giorgio Sirtori Prof. Marco Giuliani

CMP’ S OBJECTIVE To measure policy positions of all relevant parties competing in any democratic election in post WWII period for OECD and EU members, Central and Eastern Europe and (in the future) Latin America and South-East Asia

M ANIFESTO ( ELECTION PROGRAM ) → the text published by a political party in order to compete for votes in national election Why manifesto? Parties’ only authoritative policy statements Indicators of the parties’ policy preferences at a given point in time

T WO - STEP PROCESS 1. Unitising − cutting text in quasi-sentence The coding unit is a quasi-sentence, that contains exactly one statement e.g.: ″We need to address our close ties with our neighbours (107) / as well as the unique challenges facing small business owner in this of economic hardship (402)″

. 2. Coding − find the right code for a quasi- sentence Attribute to each coding-unit one, and only one, category CMP developed a category system composed of 56 categories, grouped in 7 policy areas, designed to be comparable between parties, countries, elections and across time

. Manifesto data can be used, and have been used, to provide valid and reliable measurements of party policy position e.g.: Left-right scale estimates from Mapping policy preferences (Budge et al. 2001)

CMP LEFT - RIGHT SCALE

CMP LEFT - RIGHT MEASURE Left-right sore = proportion(right - left) % (R – L)/(R + L) % Scale ranges from -100 to +100

CMP left-right scale positions for British parties ( )

C OMPARATIVE P OLITICS S HORTCOMINGS OF LOCATING P OLITICAL P ARTIES USING P ARTY M ANIFESTO D ATA Irina Dámátár Prof. Marco Giuliani

E STIMATING LEFT - RIGHT POSITIONS OF POLITICAL PARTIES The aim of the Comparative Manifestos Project is to place parties on the left-right political spectrum, parting from the information provided within their electoral programmes Each party’s position is measured as the difference (in percentages) between the right-associated text mentions and left-associated ones; the final scale ranges from -100 to 100 The identity of left and right manifesto coding categories was determined using a series of within-country exploratory factor analysis of a wide range of coding categories

T HE COMPONENTS OF THE CMP LEFT - RIGHT SCALE

S HORTCOMINGS No indication of the uncertainty associated with any CMP estimate - when placing a policy in a certain point on the left-right dimension, no associated error is indicated CMP left-right measure consists of pre-defined and fixed scale components - the constituent elements of the left-right scale are defined in the same way for all countries, at all time periods but the meaning of left and right can vary in time and space (e.g.: market regulation is a left issue in the CMP dataset, while in Germany is a valance issue) - CMP measure may not include relevant variables, that can explain much of the parties’ variation, while considering unrelated ones

. The CMP scale combines party positions with party specific measures of the relative salience of the left-right scale, producing a measure that is affected by non-right-left issues mentioned in the party manifesto - positions of the parties are in part determined by coding categories that are not in the CMP scale The scores which CMP interpret as positional data may indicate in fact the willingness of a party to move in accordance to changing political circumstances rather than their actual position - hence, the measurement made by CMP and the score attribute to a certain party lacks precision

. CMP approach identifies position on the basis of saliency scores; but this can be seen more as a movement of position towards the electorate rather than a party’s true position - to eliminate this shortcoming, Franzmann & Kaiser stress upon the distinction between position issues valence issues matters upon which the electorate’s matters evaluated either negatively opinions are clearly divided or positively by all voters - only position issues can structure the policy space -hence, they classify items based on their position or valance character

. Using factor analysis to compute party positions leaded to highly difficult interpretations when the number of variables out weighted the number of cases (low measure of sampling adequacy) Addressing this shortcoming by focusing on the economic “super-issue” as equivalent to the left-right scale disregards the fact that left and right are not perceived only in economic terms Nevertheless, in its second stage CMP followed a two step approach: issues were coded as left or right, and the resulting composition was examined through factor analysis -> to see whether these issues did indeed fit in the assumed dimension - factor analysis is also problematic because aims at a reduction of dimensions

E XPLAINING DIFFERENCES BETWEEN CMP SCALE POSITIONS AND E XPERT S URVEYS JUDGEMENTS Computing the position of a party using an expert survey estimate may lead to different results than the ones of the CMP Why? Although there is no pattern to explain these differences, there are a number of hypothesis that can be considered: I-st possibility: party policies change between the time of the election for which CMP scored a party manifesto, and the time when the expert survey was conducted

. II-nd possibility: one of the two measurements contains significant error III-rd possibility: measuring different quantities - party manifestos may have policy preferences not expressed in their election programmes and which hence cannot be captured in the scores provides by CMP - no single pre-defined scale will accurately characterize the left right dimension of politics in all countries - “new” dimensions of politics, such as immigration and environment, whose omission from the measurement can cause inaccuracies (e.g.: until 1980s, in Germany, environmental protection was a valance issue or at least was not ideologically structuring the policy space)

P LEASE NOTE THAT NO SHEEP WERE HARMED IN THE MAKING OF THIS PRESENTATION

C OMPARATIVE POLITICS E XPERT S URVEYS P ARTY P OLICY IN M ODERN D EMOCRACIES Kenneth Benoit Michael Laver Silvia Merisio Prof. Marco Giuliani

A NALYSING P ARTY C OMPETITION Why voters choose to support specific political parties? Why parties do what they do once the election is over? Public needs and wants Policy adopted by political parties

N EED TO ESTIMATE PARTY POLICY POSITIONS Problem of gathering reliable information Election Manifestos How parties behave (e. g. Roll call votes) Surveying party politicians or party supporters Not systematic analysis Extremely expensive and difficult across countries Expert surveys

E XPERT SURVEYS “Expert” most comprehensive population of country specialists These experts are asked to locate party policy positions, in the party systems of which they have expert knowledge On a set of predefined policy dimensions

P OLICY DIMENSIONS Economic Policy Social Policy European integration General policy (environment, immigration, etc.) Specific dimensions for Post-communist countries (e.g. privatization, religion, etc.) Subjective sympathies/closeness to party General Left-Right dimension All dimensions are evaluated on a scale ranging from 1 to 20

E XPERT SURVEYS They provide information on party positions Wide range of countries Common format Expert surveys are a systematic way to summarize the judgments of the consensus of experts on the matters at issue

R ISKS The same concept/ dimension can be interpreted in different ways How to assess the content of validity Importance of broadly equivalent dimensions e.g. economic policy Importance of broadly equivalent dimensions e.g. economic policy Reasonableness of numbers and comparison of the result with other method

T HE SURVEY Reference to a previous work by Laver and Hunt (1992) Survey conducted in countries Core dimensions on economic and social policies Benoit and Laver (2006) Survey conducted in countries (all European countries from the former Soviet Bloc) 38 policy dimensions (including specific dimensions for each country) Real continuity between the two works

T HE SURVEY (2) Inclusion of countries classified as non-fully democracies by Freedom House; e.g. Belarus, Russia, Albania, Moldova, Ukraine Translation in 24 languages; Web or paper format.

S URVEY DETAILS FROM E ASTERN E UROPE COUNTRIES

P OLICY DIMENSIONS FROM THE EXPERT SURVEYS ( NUMBERS PROVIDE THE TOTAL EXPERT RESPONSES ON EACH DIMENSION )

S URVEY DETAILS FROM ITALY

S URVEY DETAILS FROM LITHUANIA

C ONCLUSIONS High response rate in all countries Wide sample of countries which had never been investigated before Flexible model Possibility to use expert surveys for different aims and topics

C OMPARATIVE P OLITICS SHORTCOMINGS OF EXPERT SURVEYS Adelaida Foitik Prof. Marco Giuliani

EXPERT SURVEYS – ADVANTAGES -> POPULARITY  economical way of measuring party positions  provide information on party policy positions in a common and standardised format  can be administered at any time, unlike manifestoes-tied to electoral calendars  as long as experts are willing to respond to surveys, the expert survey methodology may probe topics that do not surface in manifestos or other data sources e.g. internal dissent within a party  quick and easy compared to other methods(content analysis of party electoral programs or legislative behavioural studies)  they reflect the judgements of experts=> weight and legitimacy BUT how valid is the information?? ANALYZE: what, how & when

THE VALIDITY OF EXPERT JUDGMENTS Framing survey questions: What is the basis of the judgments that experts offer? Do experts answer questions in the way they were intended? -> Budge (2000): 4types of problems what ‘party’ is being judged by the expert? - Is it the party in the electorate, the party in government or the party organization? what criteria do experts bring to bear when they judge party positions? - what do abstract labels like ‘left’ and ‘right’ mean to the expert? do experts judge the intentions of parties or their behavior? what is the time frame for the judgments that we ask experts to make? ->Fundamental question: how do experts interpret the questions in expert surveys and how do they link substantive knowledge about parties to those questions?

REMARK Wildly varying considerations when judging party positions  misleading expert judgments  Solution: expert survey design- good questionnaire will attempt to: avoid ambiguous terms such as ‘party’ and ‘left-right’ give them a more circumscribed meaning indicating precisely what judgment should concern specify time frames explicitly to limit variation on this dimension

C ONT ’ D e.g.: experts may be asked to judge the position of the party leadership on issue X during the past year e.g. in evaluating the left-right position of parties, experts may be asked to describe what ‘left-right’ means in a particular country or what criterion they used to define this dimension HOWEVER, better expert survey design might not alleviate all of the validity concerns raised by Budge: problems like telescoping (recalling facts for the wrong time period before/after they happened or afterwards) are bound to plague such judgments, even when we provide an explicit time frame Even the most carefully crafted question may still leave an interpretative space for experts that could distort their judgments.

I. THE 1999 EXPERT SURVEY OF NATIONAL PARTY POSITIONS ON EU INTEGRATION ->evaluated by Budge, developed by Ray (1999), objective: to assess the positions of national political parties vis-à-vis European integration, run , 15 EU Member States NOTE: in order to minimize response variation due to differential scale interpretations by experts: the question specified the object that was to be evaluated (the party leadership) the time frame for the evaluation (1999) response options were explicitly labeled => the question was designed to put the experts in a common frame of mind so that they would be judging the same object, on the same dimension, at the same point in time

CONCLUSION OF THIS SURVEY => encouraging findings- the correlations among experts are on average very high, both within particular countries and across the whole set -> substantial convergence in the judgment criteria that experts use: remarkable consistency in responses, agreement in expert placements of political parties with other measures. => considerable confidence in the expert data

II. BENOIT AND LAVER (2006) - instead of using experts’ answers to locate parties within a given policy space, use them to directly map the ideal points of the respondents Focus: the Italian case-rely on two expert surveys conducted with the same methodology in a short time span (2003,2006)=> possible to check for the robustness of our results - locate each party on the L-R dimension- all aspects of party policy - policy preferences of parties- 4 substantive policy dimension (economic policy, social policy, the decentralization of decision making and environmental policy ) - “sympathy scale” asked experts to place all parties on a scale indicating their own closeness to each party’s-> to test for any possible respondent bias by checking whether expert placements of parties on substantive dimensions were correlated with their personal sympathy for a party’s policies

EXPERT PREFERENCES AND BIAS: AN ANALYSIS - party positions estimated on the basis of survey data are not always reliable - respondents tend to place the parties they like closer to where they locate themselves( = rationalization problem) =>This is a point well recognized in BL: “the classic problem of a sample bias is not a concern [in expert surveys], even if the experts we consult hold strong political preferences, as long as these preferences do not interfere with their expert knowledge” RESULTS: => on a pool of 158 observations (i.e., the number of parties surveyed), 20 biased parties (12.7%) were found. NOTE: almost two-third (65%) of the biased parties are extreme-right ones. HOWEVER: not every extreme-right party is biased (just the 59%) AND not every biased-party is indeed an extreme-right one (one-third of the total)

REMARKS  the consequent level of bias (1.2 out of 10parties), may have nothing to do with the “true” characteristics of the different communities of political experts because information drawn from individuals is quite vulnerable to selection bias  the selection bias is often due to low response rates. Instead of trying to increase the rate of response in the hope of decreasing the level of bias (if a selection bias problem is indeed present), we should attempt to give a more circumscribed policy- meaning to the questions administered, avoiding as far as possible ambiguous terms such as, “Left-Right”

CONCLUSION OF THIS SURVEY a rationalization problem is indeed present: this bias is relevant because it can jeopardize the usefulness of expert scores for empirical analysis the ambiguity of a label as Left-Right locations => expert judgments become unreliable evidence of bias in the expert placements of parties especially against right-wing parties (not necessarily extreme-right ones) the Left-Right position of a party is less important than the policy preferences of experts (and their variance) to explain the probability of a party being biased HOWEVER, the bias is less pronounced when we pass from a label as “Left- Right” to less abstract policy dimensions

CONCLUDING REMARKS Reliance on expert judgments is an attractive option for measuring complex phenomena such as party positions about policies: they are comparatively straightforward to conduct, but we need to make sure about their validity. As suggested by Budge’s concerns, those who use expert surveys should be cautious. We have investigated the shortcomings of the expert survey with particular attention to 2 specific cases: while in the first one the results appear to be encouraging revealing the remarkable agreement among experts about the placement of parties, in the second one there is evidence of bias in the expert placements of parties along the Left-Right dimension Validity assessment is an exercise that should be a central part of expert survey

THANK YOU FOR YOUR KIND ATTENTION