1. The Effect of Interviewer And Personal Visits on Response Consistency
Jonathan Katz
QDET2 Conference
Miami, FL
November 10th, 2016
Disclaimer: Any views expressed are those of the author and not necessarily those of the U.S. Census Bureau.

2. Literature Reliability
Optimal goal is to design questions where responses will be replicated with the same true values
Unfeasible
“parallel measures”
Pre-testing
Re-interviews (Groves, 1989)

3. Literature continued…
Interviewer Variability
Had different findings depending on whether interviewers were included as a random effect (Dijkstra,1983) (O’Muircheartaigh & Campanelli, 1998)
Hard-To-Reach Respondents
Mixed research if increasing effort for hard-to-reach respondents impact data quality
Peytchev, Peytcheva, & Groves (2010) found abortion experiences to be underreported
Hox, de Leeuw, & Chang (2012) found “data quality” not to be affected between “eager” and “reluctant” respondents
Social/Political Attitudinal Items
Olson (2006) discussed that length of marriage item may have had less measurement bias (compared to time since divorce) because it’s a more “salient” event

4. Motivation
Measure how interviewers and number of contact attempts each contribute to reliability of responses to attitudinal items ( using European Social Survey Round 7 Data )
Journal papers have focused upon
the relationship between reliability and number of contact attempts (or recruiting hard-to-reach survey non-respondents)
the relationship between interviewers and reliability

5. Data Used for Study European Social Survey (ESS) Round 7
40,185 cases
22 countries participated
Supplementary questionnaire administered face-to-face following main questionnaire
(Country response rates ranged from 30 % to 80 %)
Final Data Set
N = 28,415 cases
1,819 interviewers
21 countries

6. Items of Interest
Six attitude items on an 11 point scale (0-10) administered both in main questionnaire and supplementary questionnaire
3 social attitude items
3 political attitude items

7. Social Attitude Items
Racial Attitude
And how important do you think being white should be in deciding whether someone born, brought up and living outside [country] should be able to come and live here?
Extremely unimportant – – Extremely important
Language Attitude
Please tell me how important you think being able to speak [country’s official language(s)] should be in deciding whether someone born, brought up and living outside [country] should be able to come and live here.
Country Way of Life Attitude
Now, how important do you think being committed to the way of life in [country] should be in deciding whether someone born, brought up and living outside [country] should be able to come and live here?

8. Political Attitude Items
Role in Political Issues Attitude
How able do you think you are to take an active role in a group involved with political issues?
Completely Unable – – Completely Able
Participation in Politics Attitude
How confident are you in your own ability to participate in politics?
Completely Unconfident – – Completely Confident
Ease in Politics Attitude
How easy do you personally find it to take part in politics?
Extremely Difficult – – Extremely Easy

9. Methods Response Score Differences for Histograms
Multilevel Logistic Regression Models
Dependent Variable
Response Score Difference of ‘0’ or ‘1’  Reliability
Response Score Difference of ‘>1’  Low reliability
Fixed Effects
Respondent- age, education, gender
Interviewer- age, gender
Number of Personal Visits to Completion
Random Effects
Country of Interview (Level 1) and Interviewer ID (Level 2)

10. Hypotheses
Interviewers and the number of contact attempts will both significantly affect reliability in items from main questionnaire to supplementary questionnaire
Variability between interviewers will be significantly different than zero
An increase of contact attempts to interview completion (higher “burden”) would be associated with less reliability across responses to identical items
Association between personal visits and reliability would be more pronounced for social attitude items compared to political attitude items

11. Social Attitude Response Score Difference Histograms

12. Political Attitude Response Score Difference Histograms

13. Percentage of Personal Visits to Interview Completion Histogram
Mean: 2.85
Median: 2

14. Multilevel Logistic Regression Model of Social Attitudes Items
Characteristic
Racial Attitude
Language Attitude
Country Way of Life Attitude
Estimate (Standard Error)
Odds Ratio
Fixed Effect
Number of Contact Attempts
0.04 (0.01)***
1.04
0.01
(0.01)*
1.01
0.00
(0.01)
1.00
Variance Components
Random Effect
Interviewer
0.24 (0.02)*** --
0.16 (0.02)***
0.17 (0.02)***
Country
0.14 (0.05)***
0.06 (0.02)***
***p<0.01
**p<0.05
*p<0.10

15. Multilevel Logistic Regression Model of Political Attitudes Items
Characteristic
Role in Political Issues Attitude
Participation in Politics Attitude
Ease in Politics Attitude
Estimate (Standard Error)
Odds Ratio
Fixed Effect
Number of Contact Attempts
0.01
(0.01)
1.01
0.00
1.00
Variance Components
Random Effect
Interviewer
0.08 (0.01)*** --
0.07 (0.01)***
0.10 (0.01)***
Country
0.11
(0.05)**
0.14
0.10
(0.04)**
***p<0.01
**p<0.05
*p<0.10

16. Testing the Variability Between Interviewers Equal to 0
Category
Item
Chi-Square Statistic
Social Attitudes
Racial Attitude
305.68***
Language Attitude
187.35***
Country Way of Life Attitude
228.82***
Political Attitudes
Role in Political Issues Attitude
98.22***
Participation in Politics Attitude
56.42***
Ease in Politics Attitude
65.87***
***p<0.01
**p<0.05
*p<0.10

17. Findings for Hypotheses
Interviewers
Significant between-interviewer variability at p<0.01 for all six items
Contact Attempts
Statistically significant at p<0.10 in two of the social attitude items in multilevel logistic regression models
However, both these regression coefficients were positive (opposite direction than hypothesized!)
None of the “contact attempts” regression coefficients for political attitude items were significant

18. Future Research
Goal is to examine if there is significant differences between social and political attitude items in terms of variability across interviewers.
In other words, is variability between interviewers more significant for social attitude items compared to political attitude items?

19. Limitations ESS does not use an interpenetrated design.
Income item was not added in the model
Interviewer race and respondent race were unavailable
End point labels on low end of scale for political attitude items had different wording
Did “burden” play a role?

20. jonathan.m.katz@census.gov or (301) 763-5956
Questions?
or (301)
I would like to thank Andy Peytchev and Luke Larsen for their consulting on this research.

21. Appendix

22. Estimate (Standard Error)
Racial Attitude
Characteristic
Value
Estimate (Standard Error)
Odds Ratio
Number of Contact Attempts
(0.0088)***
1.042
Gender
Female
(0.0278)*
1.051
Interviewer Gender
(0.0379)***
1.156
Education
Lower secondary
(0.0555)
0.982
Lower tier upper secondary
(0.0570)
1.065
Upper Tier Upper Secondary
(0.0567)***
1.226
Advanced Vocational, sub-degree
(0.0576)***
1.251
Lower Tertiary Education, BA Level
(0.0638)***
1.518
Higher Tertiary Education, >= MA level
(0.0625)***
1.791
Respondent Age
- -
(0.0008)***
0.987
Interviewer Age
(0.0015)**
1.003
Variance Components
Interviewer
(0.0198)*** --
Country
(0.0503)***
***p<0.01
**p<0.05
*p<0.10

23. Estimate (Standard Error)
Language Attitude
Characteristic
Value
Estimate (Standard Error)
Odds Ratio
Number of Contact Attempts
(0.0077)*
1.013
Gender
Female
(0.0257)*
0.955
Interviewer Gender
(0.0336)*
1.060
Education
Lower secondary
(0.0534)
0.979
Lower tier upper secondary
(0.0544)*
1.094
Upper Tier Upper Secondary
(0.0539)
1.058
Advanced Vocational, sub-degree
(0.0545)*
1.111
Lower Tertiary Education, BA Level
(0.0589)***
1.167
Higher Tertiary Education, >= MA level
(0.0573)***
1.218
Respondent Age
- -
(0.0007)***
0.997
Interviewer Age
(0.0013)***
1.004
Variance Components
Interviewer
(0.0155)*** --
Country
(0.0219)***
***p<0.01
**p<0.05
*p<0.10

24. Country Way of Life Attitude
Characteristic
Value
Estimate (Standard Error)
Odds Ratio
Number of Contact Attempts
(0.0076)
1.002
Gender
Female
(0.0254)
0.985
Interviewer Gender
(0.0338)
1.045
Education
Lower secondary
(0.0529)
1.023
Lower tier upper secondary
(0.0537)
1.082
Upper Tier Upper Secondary
(0.0535)***
1.186
Advanced Vocational, sub-degree
(0.0539)**
1.125
Lower Tertiary Education, BA Level
(0.0584)***
1.312
Higher Tertiary Education, >= MA level
(0.0566)***
1.258
Respondent Age
- -
(0.0007)***
Interviewer Age
(0.0013)**
Variance Components
Interviewer
(0.0161)*** --
Country
(0.0146)***
***p<0.01
**p<0.05
*p<0.10

25. Ease in Politics Attitude
Characteristic
Value
Estimate (Standard Error)
Odds Ratio
Number of Contact Attempts
(0.0073)
0.997
Gender
Female
(0.0247)***
1.167
Interviewer Gender
(0.0306)***
1.126
Education
Lower secondary
(0.0531)***
0.829
Lower tier upper secondary
(0.0534)***
0.814
Upper Tier Upper Secondary
(0.0533)***
0.787
Advanced Vocational, sub-degree
(0.0535)***
0.752
Lower Tertiary Education, BA Level
(0.0573)***
0.678
Higher Tertiary Education, >= MA level
(0.0556)***
0.633
Respondent Age
- -
(0.0007)**
1.001
Interviewer Age
(0.0012)
Variance Components
Interviewer
(0.0126)*** --
Country
(0.0426)**
***p<0.01
**p<0.05
*p<0.10

26. Participation in Politics Attitude
Characteristic
Value
Estimate (Standard Error)
Odds Ratio
Number of Contact Attempts
(0.0073)
0.996
Gender
Female
(0.0247)***
1.230
Interviewer Gender
(0.0296)***
1.088
Education
Lower secondary
(0.0531)***
0.734
Lower tier upper secondary
(0.0535)***
0.723
Upper Tier Upper Secondary
(0.0535)***
0.684
Advanced Vocational, sub-degree
(0.0536)***
0.660
Lower Tertiary Education, BA Level
(0.0573)***
0.573
Higher Tertiary Education, >= MA level
(0.0556)***
0.522
Respondent Age
- -
(0.0007)**
1.002
Interviewer Age
(0.0012)**
1.003
Variance Components
Interviewer
(0.0117)***
Country
(0.0544)**
***p<0.01
**p<0.05
*p<0.10

27. Role in Political Issues Attitude
Characteristic
Value
Estimate (Standard Error)
Odds Ratio
Number of Contact Attempts
(0.0073)
1.006
Gender
Female
(0.0247)***
1.230
Interviewer Gender
(0.0299)**
1.088
Education
Lower secondary
(0.0533)***
0.799
Lower tier upper secondary
(0.0536)***
0.780
Upper Tier Upper Secondary
(0.0535)***
0.738
Advanced Vocational, sub-degree
(0.0537)***
0.691
Lower Tertiary Education, BA Level
(0.0574)***
0.579
Higher Tertiary Education, >= MA level
(0.0557)***
0.551
Respondent Age
- -
(0.0007)***
1.004
Interviewer Age
(0.0012)
1.001
Variance Components
Interviewer
( )*** --
Country
( )**
***p<0.01
**p<0.05
*p<0.10