1. New Insights on the Cognitive Processing of AD and IS Questions
Jan Karem Höhne
Stanford University
Timo Lenzner
Gesis – Leibniz Institute
for the Social Sciences
2nd Place at the “PAPOR Student Paper Competition (2016)”
San Francisco, CA (USA) – 15 to 16 December 2016

2. Research Question What are possible REASONS for differences?
Do Agree/Disagree (AD) questions require higher cognitive effort than Item-Specific (IS) questions?
Research question since Fowler (1995).
So far, empirical evidence is missing.
What are possible REASONS for differences? 2

3. Survey Practice Since Likert (1932) AD questions have become popular.
Eurobarometer, ANES, ISSP …
European Sociological Review, Political Analysis, Public Opinion Quarterly …
Reasons for popularity¹:
Measuring different contents with the same scale.
Streamlining questionnaires. 3
¹ Saris et al. (2010) 3

4. Critique on Survey Practice
Fowler (1995) identifies several drawbacks of AD questions:
Insufficient anchoring,
Complex cognitive processing,
Small discriminatory power,
Prone to response bias.
“IS questions represent a simpler, more informative, and more direct method” (Fowler, 1995, p. 57). 4

5. Concept of “Asking Manner”
AD questions do not change “asking manner”.
Fostering a superficial response process.
IS questions change “asking manner” permanently.
Require constant reconsideration of the dimension of interest.
Encourage an active and intensive response process.
Accordingly, IS questions seem to be more demanding than AD questions.
Höhne et al. (2017) 5

6. Eye Tracking
Saccade
Fixation
During reading the eyes make a series of quick movements → saccades¹
They are accompanied by moments when the eye movements pause → fixations¹
Immediacy Assumption
Interpretations at all levels of processing are not deferred.²
Fixation time corresponds to the duration of central processing.²
Eye-Mind
Assumption
¹ Rayner & Pollastek (2006)
² Just & Carpenter (1980) 6

7. Research Hypotheses
Respondents fixate more often and longer on the stems when answering IS than when A/D questions (H1a).
Respondents fixate more often and longer on the response categories when answering IS than when A/D questions (H1b).
Respondents read more response categories in the IS than in the A/D question format (H2).
Respondents show more re-fixations between the response categories when answering IS than when A/D questions (H3). 7

8. Research Design Split-Ballot Experiment AD IS
Both groups received 3 questions dealing with political interest and 5-point fully-labeled response scales.
Agree/Disagree:
I am very interested in politics.
Item-Specific:
How interested would you say you are in politics? AD IS
N = 44
N = 40 8

9. Sample Size: N = 84 Sex: 53% female Age: Mean = 37.5 (SD = 14.6)
Education: 20% lower secondary school
12% intermediate secondary school
68% at least college preparatory secondary school
Survey Experience: 93% have participated in a web survey once before
Chi-square tests reveal no significant differences between the groups with respect to age, gender, education, computer usage, Internet usage, and survey experience. 9

10. Eye Tracking Parameter
Results
Eye Tracking Parameter
Question
Part AD
Questions IS
Fixation Count
Question Stem
0.22
(0.01)
(0.02)
Response Categories
0.16
0.29
Fixation Time
0.04
(0.00)
0.05
0.10
Effect
Size
η2 = 0.00
η2 = 0.23*
η2 = 0.16*
*p < The table reports estimated marginal means after controlling for the fixation rate and reading rate, respectively. To control for length differences of question stems and response categories between the two question formats, we divided the two eye-tracking parameters by the # of characters (Ferreira & Clifton, 1986). 10

11. Results Eye Tracking Parameter AD Questions IS Effect Size
# of categories read
3.10
(0.12)
3.06
(0.13)
η2 = 0.00
# of categories re-fixated
0.02
(0.01)
0.07
η2 = 0.14*
*p < The table reports response categories read (on average) and estimated marginal means for # of categories re-fixated after controlling for re-fixation rate. To control for length differences of cate-gories between the question formats, we divided the # of categories re-fixated by the # of characters (Ferreira & Clifton, 1986). 11

12. Results
The 3 heat maps above correspond to the AD group and the 3 heat maps below correspond to the IS group. Moreover, the heat maps include the mouse clicks of respondents. 12

13. Limitations The experiment only investigates cognitive processing.
No evidence regarding data quality.
The experiment only investigates single questions.
No evidence regarding other presentation modes. 13

14. Conclusion
No differences between AD and IS question stems → “question comprehension”.
Processing of IS response categories is more conscientious → higher fixation count/time and more re-fixations.
No differences between the number of AD and IS response categories read → rating scales.
A continuous manner of asking questions negatively affects diligence in processing → concept of “asking manner”.
Differences between cognitive complexity and cognitive effort expended.
In line with previous research we encourage researchers to give preference to IS over AD questions. 14

15. Thank you for your attention!

16. Literature
Ferreira & Clifton (1986). The Independence of Syntactic Processing. Journal of Memory and Language, 25,
Fowler (1995). Improving Survey Questions. Sage, Thousand Oaks.
Höhne, Schlosser, & Krebs (2017). Investigating Cognitive Effort and Response Quality of Question Formats using Paradata. Field Methods, 29(4).
Just & Carpenter (1980). A Theory of Reading: From Eye Fixations to Comprehension. Psychological Review, 87, 329–354.
Rayner & Pollastek (2006). Eye-Movement Control in Reading. In: Traxler & Gerns-bacher (Eds.), Handbook of Psycholinguistic (Vol. 2, pp ). Elsevier, Amsterdam.
Saris, Revilla, Krosnick, & Shaeffer (2010). Comparing Questions with Agree/Disagree Response Options to Questions with Item-Specific Response Options. Survey Re-search Method, 4, 61–79.