5th ESRA conference July 15-19, 2013, Ljubljana Quality of Cross-Country Data: Assessing through Interviewer Impressions 5th ESRA conference July 15-19, 2013, Ljubljana Dr Tinatin Zurabishvili, CRRC-Georgia tina@crrccenters.org Dr Heghine Manasyan, CRRC-Armenia heghine@crrccenters.org

The Caucasus Challenges of conducting surveys in an environment where the respondents’ trust towards surveys is low (Hayes et al., 2006), and Conditions during face-to-face interviews are rarely perfect (relatives present, etc.). Importance of paradata, including interviewer assessments, as an additional resource to judge about the quality of the data (e.g., Couper 2013). Limitations – interviewer assessments can themselves be biased (West 2013). www.crrccenters.org

The question: To what extent, and in which direction, are the interviewer assessments biased, and to what extent do they help us determine problems in the survey process, namely, help us judge about the reliability of the answers provided by the CB respondents? We expect that the interviewers’ assessments, although representing judgments, still help to “filter out” the respondents that are not sincere, not involved, providing less reliable answers. www.crrccenters.org

Caucasus Barometer Project Annual representative surveys of the adult (18+) population of Armenia, Azerbaijan, and Georgia, conducted by CRRC since 2004 (current methodology employed since 2007). Circa 6500 respondents interviewed annually. Dataset and documentation - in open access for all interested researchers and public. Website: http://www.crrccenters.org/caucasusbarometer/overview/ Online data analysis tool: http://www.crrc.ge/oda/ www.crrccenters.org

The 2012 survey Fieldwork dates: October 26 – November 29, 2012. Data collection mode: face-to-face interviews (PAPI). Sample size: 6715 completed interviews (2382 in Armenia, 1829 in Azerbaijan, and 2502 in Georgia). Sampling: 2-stage cluster sampling. Results are representative for adult (18+) population of Armenia, Azerbaijan and Georgia, except territories affected by military conflict. Country-level confidence intervals: +/-1.4 for Armenia, +/-2.1 for Azerbaijan, and +/-1.9 for Georgia. The results are not weighted (West 2013). www.crrccenters.org

Securing quality: limited number of interviews per interviewer Average number of interviews per interviewer by country (2012): Armenia Azerbaijan Georgia 37.2 40.6 21.9 Limiting the number of interviews as one of the ways to minimize interviewer bias. www.crrccenters.org

CB 2012 Interviewer Assessment Form How often was the respondent distracted during the interview? [W2] How often did you feel that the respondent lacked knowledge about the questions you asked? [W3] How often did you feel the respondent was reluctant to answer the questions? [W4] On a scale of 1 to 5, how would you rate the respondent’s level of intelligence? [W5] www.crrccenters.org

Overall: (%, by country, not weighted) www.crrccenters.org

(%, by country, not weighted) * Sum of answers “very intelligent” and “intelligent”. www.crrccenters.org

Interviewers’ ‘Attitude Index’ POSITIVE ATTITUDE: W3 (options 1 and 2) AND W4 (options 1 and 2) AND W5 (options 4 and 5). NEGATIVE ATTITUDE W3 (options 4 and 5) AND W4 (options 4 and 5) AND W5 (options 1 and 2). INDEX: Positive / Indifferent / Negative Indifferent and Negative indexes do not work – we get very small numbers and there is no variation. www.crrccenters.org

Spread of Positive Attitudes: Armenia Azerbaijan Georgia 31% 18% 40% Indifferent and Negative indexes do not work – we get very small numbers and there is no variation. www.crrccenters.org

CB 2012 sensitive questions analyzed for this presentation (1) P3: How strongly do you agree or disagree with the opinion that, overall, TV journalists in /country/ are serving interests of people like you? 5-point scale, Item NR*: 11% in Armenia, 8% in Azerbaijan, 15% in Georgia (average – 11%). *NR includes both “Don’t know” and “Refuse to answer” options. www.crrccenters.org

CB 2012 sensitive questions analyzed for this presentation (2) P4_13: How much do you trust or distrust /country’s/ local government? 5-point scale, Item NR: 6% in Armenia, 3% in Azerbaijan, 15% in Georgia (average – 8%). www.crrccenters.org

CB 2012 sensitive questions analyzed for this presentation (3) P15: Which of the following statements do you agree with: “People should participate in protest actions against the government …” or “People should not participate in protest actions against the government …”? 5-point scale, Item NR: 11% in Armenia, 20% in Azerbaijan, 26% in Georgia (average – 19%). www.crrccenters.org

CB 2012 sensitive questions analyzed for this presentation (4) P18: Would you say that the most recent /type of election/ election was conducted completely fairly, to some extent fairly, or not at all fairly? 3-point scale, Item NR: 21% in Armenia, 40% in Azerbaijan, 13% in Georgia (average – 25%). www.crrccenters.org

CB 2012 sensitive questions analyzed for this presentation (5) P19_7: … please tell me, in your opinion, how important it is for a good citizen to be critical towards the government? 10-point scale, Item NR: 11% in Armenia, 17% in Azerbaijan, 18% in Georgia (average – 15%). www.crrccenters.org

Findings a. The interviewers tend to like more the respondents who provide socially desirable answers, and they tend not to like those who answer “Don’t know.” www.crrccenters.org

Findings b. Comparison of the CB data by interviewer assessments leads to the most significant differences for the respondents answering “Don’t know,” and very rarely – in case of other answer option. A typical example: www.crrccenters.org

… how important it is for a good citizen to be critical towards the government? (%) Armenia Azerbaijan Georgia Positive att. Non-positive att. Not important 10 11 36 28 4 7 5 12 13 2 22 21 26 20 15 9 19 Important 43 37 8 41 29 DK 3 14 17 www.crrccenters.org

Next steps Add more questions to the Interviewer assessment form in CB2013. www.crrccenters.org

Thank you http://www.crrccenters.org/ tina@crrccenters.org