1. 5th ESRA conference July 15-19, 2013, Ljubljana
Quality of Cross-Country Data: Assessing through Interviewer Impressions
5th ESRA conference July 15-19, 2013, Ljubljana
Dr Tinatin Zurabishvili, CRRC-Georgia
Dr Heghine Manasyan, CRRC-Armenia

2. The Caucasus Challenges of conducting surveys in an environment where
the respondents’ trust towards surveys is low (Hayes et al., 2006), and
Conditions during face-to-face interviews are rarely perfect (relatives present, etc.).
Importance of paradata, including interviewer assessments, as an additional resource to judge about the quality of the data (e.g., Couper 2013).
Limitations – interviewer assessments can themselves be biased (West 2013).

3. The question:
To what extent, and in which direction, are the interviewer assessments biased, and to what extent do they help us determine problems in the survey process, namely, help us judge about the reliability of the answers provided by the CB respondents?
We expect that the interviewers’ assessments, although representing judgments, still help to “filter out” the respondents that are not sincere, not involved, providing less reliable answers.

4. Caucasus Barometer Project
Annual representative surveys of the adult (18+) population of Armenia, Azerbaijan, and Georgia, conducted by CRRC since 2004 (current methodology employed since 2007).
Circa 6500 respondents interviewed annually.
Dataset and documentation - in open access for all interested researchers and public.
Website:
Online data analysis tool:

5. The 2012 survey Fieldwork dates: October 26 – November 29, 2012.
Data collection mode: face-to-face interviews (PAPI).
Sample size: 6715 completed interviews (2382 in Armenia, 1829 in Azerbaijan, and 2502 in Georgia).
Sampling: 2-stage cluster sampling.
Results are representative for adult (18+) population of Armenia, Azerbaijan and Georgia, except territories affected by military conflict.
Country-level confidence intervals: +/-1.4 for Armenia, +/-2.1 for Azerbaijan, and +/-1.9 for Georgia.
The results are not weighted (West 2013).

6. Securing quality: limited number of interviews per interviewer
Average number of interviews per interviewer by country (2012):
Armenia
Azerbaijan
Georgia
37.2
40.6
21.9
Limiting the number of interviews as one of the ways to minimize interviewer bias.

7. CB 2012 Interviewer Assessment Form
How often was the respondent distracted during the interview? [W2]
How often did you feel that the respondent lacked knowledge about the questions you asked? [W3]
How often did you feel the respondent was reluctant to answer the questions? [W4]
On a scale of 1 to 5, how would you rate the respondent’s level of intelligence? [W5]

8. Overall: (%, by country, not weighted)

9. (%, by country, not weighted)
* Sum of answers “very intelligent” and “intelligent”.

10. Interviewers’ ‘Attitude Index’
POSITIVE ATTITUDE:
W3 (options 1 and 2) AND
W4 (options 1 and 2) AND
W5 (options 4 and 5).
NEGATIVE ATTITUDE
W3 (options 4 and 5) AND
W4 (options 4 and 5) AND
W5 (options 1 and 2).
INDEX:
Positive / Indifferent / Negative
Indifferent and Negative indexes do not work – we get very small numbers and there is no variation.

11. Spread of Positive Attitudes:
Armenia
Azerbaijan
Georgia
31%
18%
40%
Indifferent and Negative indexes do not work – we get very small numbers and there is no variation.

12. CB 2012 sensitive questions analyzed for this presentation (1)
P3: How strongly do you agree or disagree with the opinion that, overall, TV journalists in /country/ are serving interests of people like you?
5-point scale,
Item NR*: 11% in Armenia, 8% in Azerbaijan, 15% in Georgia (average – 11%).
*NR includes both “Don’t know” and “Refuse to answer” options.

13. CB 2012 sensitive questions analyzed for this presentation (2)
P4_13: How much do you trust or distrust /country’s/ local government?
5-point scale,
Item NR: 6% in Armenia, 3% in Azerbaijan, 15% in Georgia (average – 8%).

14. CB 2012 sensitive questions analyzed for this presentation (3)
P15: Which of the following statements do you agree with: “People should participate in protest actions against the government …” or “People should not participate in protest actions against the government …”?
5-point scale,
Item NR: 11% in Armenia, 20% in Azerbaijan, 26% in Georgia (average – 19%).

15. CB 2012 sensitive questions analyzed for this presentation (4)
P18: Would you say that the most recent /type of election/ election was conducted completely fairly, to some extent fairly, or not at all fairly?
3-point scale,
Item NR: 21% in Armenia, 40% in Azerbaijan, 13% in Georgia (average – 25%).

16. CB 2012 sensitive questions analyzed for this presentation (5)
P19_7: … please tell me, in your opinion, how important it is for a good citizen to be critical towards the government?
10-point scale,
Item NR: 11% in Armenia, 17% in Azerbaijan, 18% in Georgia (average – 15%).

17. Findings
a. The interviewers tend to like more the respondents who provide socially desirable answers, and they tend not to like those who answer “Don’t know.”

18. Findings
b. Comparison of the CB data by interviewer assessments leads to the most significant differences for the respondents answering “Don’t know,” and very rarely – in case of other answer option. A typical example:

19. … how important it is for a good citizen to be critical towards the government? (%)
Armenia
Azerbaijan
Georgia
Positive att.
Non-positive att.
Not important 10 11 36 28 4 7 5 12 13 2 22 21 26 20 15 9 19
Important 43 37 8 41 29 DK 3 14 17

20. Next steps
Add more questions to the Interviewer assessment form in CB2013.

21. Thank you http://www.crrccenters.org/ tina@crrccenters.org