1. Gregor Čehovin, Dr. Vasja Vehovar
Implications of disposition codes for monitoring breakoffs in web surveys
Gregor Čehovin, Dr. Vasja Vehovar
General Online Research (GOR 17)
Berlin, March 2017

2. Presentation structure
PART 1: CONCEPTUAL BACKGROUND
Defining breakoffs
Unit progress level & breakoffs
Breakoff indicators
Breakoffs in relation to AAPOR codes & unit nonresponse
PART 2: EMPIRICAL STUDY
Breakoffs & questionnaire length
Data source & preparation
Data overview
Summary of preliminary findings: Modelling QBR & IBR
PART 3: CONCLUSION
Concluding remarks
Limitations

3. PART 1: CONCEPTUAL BACKGROUND
Defining breakoffs
Unit progress level & breakoffs
Breakoff indicators
Breakoffs in relation to AAPOR codes & unit nonresponse

4. Defining breakoffs Overall aim of the study:
Define different types of breakoffs & develop corresponding indicators.
Analyze the specific relation between breakoffs & web questionnaire length.
2 key characteristics.: „… eligible unit starts answering the web questionnaire, but then leaves the survey prematurely, before concluding…“ (Callegaro et al. 2015).
Introduction breakoff: Breakoffs that occur at the introduction page or at the first page with questions, but before the respondent provided any answers.
Questionnaire breakoffs: Breakoffs that occur later in the questionnaire, i.e. at some point after the first page.
Reporting only total breakoff: Less methodological value (fundamentally different factors of introduction & questionnaire breakoff).

5. Unit progress level & breakoffs
Progression through the questionnaire (before possibly breaking off), regardless of responding to items (i.e. item nonresponse). Formally:
Q: all pages (items) in a questionnaire.
q: value assigned to unit, which moved through a page and moved to next one.
Respondents can prematurely quit after concluding the page q, where q=0…(Q-1).
Introduction breakoff: progress level q=0. Left without processing any items.
Questionnaire breakoff: progress level 0<q<Q. Breakoff after passing through the first page (q=1) with questions & before the last page (q=Q).
Total (overall) breakoff: progress level q<Q.
Concluded questionnaire: progress level q=Q. Provided a response or item nonresponse on all pages with questions, q=1…Q.

6. Breakoff indicators
Based on the previous definitions, we can specify the corresponding rates:

7. Breakoffs and relation to AAPOR codes

8. Breakoffs in relation to unit nonresponse
Due to specifics, breakoffs need separate approach from other NR causes.
Integrating them with unit nonresponse without further structuring can hinder efforts of breakoff prevention (Peytchev, 2009).

9. PART 2: EMPIRICAL STUDY Breakoffs & questionnaire length
Data source & preparation
Data overview & means comparison
Summary of preliminary findings: Modelling QBR & IBR

10. Central issue of empirical study: Breakoffs & questionnaire length
Time (seconds) needed to fill in the questionnaire. Depending on:
Number of questions, number of pages…
Question complexity, routing and randomization, questionnaire layout…
Device used & other circumstances.
Lengthy questionnaires usually contribute to lower data quality & higher breakoffs (e.g. Krosnick & Presser, 2010; Peytchev, 2009; Vicente & Reis, 2010).
Doubling questionnaire length increases breakoff rate by roughly ½, e.g.:
21%  34% (Ganassali, 2008);
17%  24% (Deutskens, de Ruyter, Wetzels, & Oosterveld, 2004);
32%  43%  53% (Galesic, 2006).
Sometimes unclear (e.g. Burdein, 2014; Crawford et al., 2001)  address the specific relation between breakoffs and web questionnaire length.

11. Study data source & preparation
Initially 40,000 surveys created between 2009 and 2014.
Surveys performed in Slovenia by public and private sector institutions & individuals (DIY research studies).
Anonymized survey-level paradata: Number of pages, questions and variables (question items), the number of respondents & breakoff rates.
Final dataset with N=7,676 surveys & approx. 1,250,000 responses:
At least 10 respondents finished the survey.
Questionnaire consisting of at least 2 pages.
Questionnaire consisting of variables & variables per page.
Excluded entries that are not real questionnaires (popular voting, quizzes… )

12. Study data overview (N=7676)
IBR represents 79% of all breakoffs for the average survey.
The opposite (QBR>IBR) possible in surveys with high legitimacy & a good invitation, followed by a lengthy/difficult questionnaire (e.g. Petrovčič et al. 2103).

13. Comparison of mean values
Variables: QBR from 8% (2–10 variables) to 23% (101–500 variables).
Pages: QBR from 7% (2 pages) to 20% (21–150 pages).
The same yielded almost no differences in mean IBR, which is to be expected.

14. Summary of prelim. findings: Modelling QBR
QBR on average p.p. for each additional variable.
QBR on average p.p. for each additional page.
invitations not significant.
Overall 10% of the variability in QBR (adj. R2=0.101).

15. Summary of prelim. findings: Modelling IBR
IBR on average p.p. if invitations are used.
Effect of pages & variables strongly diminished compared to modelling QBR.
Modelling IBR explains considerably less variability (modelling IBR adj. R2=0.043 vs. QBR adj. R2=0.101).

16. PART 3: CONCLUSION
Concluding remarks & considerations
Limitations

17. Concluding remarks
Defining IBR & QBR separately: Greater practical value vs. only TBR.
Study implications (in addition to impact of questionnaire length on QBR):
Survey design: Breakoff prevention at introduction rather than later (IBR>QBR).
IBR depends on more external factors (from sponsor to engagement).
QBR does not necessarily mean that a survey is unusable (at least one question answered). The researcher can here define minimum quality criteria.
Considerations for the future development of disposition codes:
Terminologically separate breakoffs from the context of the completion status.
Additional codes to define possible breakoffs in “(1.1) Complete” and “(1.2) Partial or break-off with sufficient information”.
Increase precision of “(2.122) Breakoff, after interview started” in terms of unit progress level. E.g. at least after the introduction before completing any pages with questions (q=0) or after completing & submitting the first page (q=1).
Specify IBR and QBR consistently in all relevant categories to avoid ambiguity.

18. Limitations
Surveys from mostly DIY research studies performed in Slovenia by public and private sector institutions and individuals.
Survey in the commercial and marketing sector have a wider spectrum of breakoff rates. We can expect them to be reported at lower up to comparable levels.
Analysis is performed on a single level and assumes that individual questionnaires have equal importance or weight in the analysis.
Breakoffs are only a specific indicator of data quality. It is possible for a longer but less burdensome questionnaire to outperform a shorter but cognitively more demanding one in terms of data quality.

19. Thank you! Questions?
1KA. EnKlikAnketa – OneClickSurvey.
AAPOR. (2016). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.
Burdein, I. (2014). Shorter Isn’t Always Better. CASRO Journal 2014.
Callegaro, M., Lozar Manfreda, K., & Vehovar, V. (2015). Web Survey Methodology. London: SAGE.
Crawford, S. D., Couper, M. P., & Lamias, M. J. (2001). Web surveys. Perception of burden. Social Science Computer Review, 19(2), 146–62.
Deutskens, E., de Ruyter, K., Wetzels, M., & Oosterveld, P. (2004). Response rate and response quality of Internet-based surveys: An experimental study. Marketing Letters, 15(1), 21–36.
Galesic, M. (2006). Dropouts on the web: Effects of interest and burden experienced during an online survey. Journal of Official Statistics, 22(2), 313–328.
Ganassali, S. (2008). The influence of the design of web survey questionnaires on the quality of responses. Survey Research Methods, 2(1), 21–32.
Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. In P. Marsden & J. D. Wright (Eds.), Handbook of survey research (2nd ed., pp. 263–313). Howard House, UK: Emerald Group Publishing.
Petrovčič, A., Lozar Manfreda, K., & Petrič, G. (2013). The effect of invitation characteristics and response reluctance on non-response in web forum surveys. Presented at the European survey research association (ESRA)
Peytchev, A. (2009). Survey Breakoff. Public Opinion Quarterly, 73(1), 74 –97.
Vicente, P., & Reis, E. (2010). Using questionnaire design to fight nonresponse bias in web surveys. Social Science Computer Review, 28(2), 251–267.