Improving Survey Response Rates: The Effect of Embedded Questions in Web Survey Email Invitations Nick Inchausti, SurveyMonkey Mingnan Liu, Facebook

Response Rates Response rates are a critical indicator of data quality for surveys, and we as survey researchers are always trying to get to 100% response For mixed-mode or web surveys, email invitations are widely used as the first point of contact to reach respondents We wanted to find out what happens if you begin data collection in the initial survey invitation itself?

Literature Review Email personalization sometimes boosts response rates (Heerwegh, 2005), but sometimes has no effect (Porter & Whitcomb, 2003) A white background and simple header had higher response rates than other conditions (Whitcomb & Porter, 2004) Mentioning the purpose of an email (requesting for survey participation) and the sponsor of the survey in the email subject line also had an impact on survey participation (Porter & Whitcomb, 2005) Several other factors, including the length of the email, placement of the URL, and the estimated time of the survey, have also been explored in survey experiments in order to improve survey participation and response rate (Kaplowitz, Lupi, Couper, & Thorp, 2012; Keusch, 2012; Trouteaud, 2004).

Research Question New product feature: ability to embed survey question within an email Research Question: Do survey invitations that include an embedded question have higher response rates than standard invitations? Click rate Completion rate Data quality check: comparison of responses to the first question

Embedded Question Screenshot: embedded survey invitation email Screenshot: standard survey invitation email

Experiment Design Sample frame: group of SurveyMonkey customers who had agreed to participate in research projects and provided their email addresses Data collected July 27 – August 8, 2016 Initial email invitation to complete a survey, follow-up reminder 4 days later Random assignment into 1/2 conditions: standard and embedded first question 4333 valid emails for embedded condition 4347 valid emails for standard condition 13-question survey, identical in each condition Experience with SurveyMonkey Satisfaction with the survey platform Interest in additional features

Results – Improved email click rate Higher click rate for embedded survey than the standard survey (32.0% vs. 26.2%) Statistically significant p<.001 This means that respondents in the embedded condition were more likely to click on the embedded question and start the survey than the respondents in the standard condition to click on the “Begin survey” button.

Results – Improved survey completion rate Higher completion rate for the embedded survey than the standard survey (29.1% vs. 24.4%) Statistically significant p<.001 Completion rate = # complete # sent

Results – Small negative effect on survey drop-out rate   Valid email Clicked Completed % Completed/Clicked Embedded 4333 1388 1261 90.8% Standard 4347 1141 1059 92.8% The proportion of respondents who completed the survey divided by the number who clicked into the survey was slightly higher for those in the standard condition… Which means that the embedded version had a slightly higher drop-out rate… But this difference is not statistically significant (p=.07).

Results – No effect on data quality Do we get different responses when we ask a question embedded in an email vs. in the survey itself? No. The ratio of the two NPS scores between the embedded and standard email invites was 0.98, suggesting the two responses to the first question were almost identical for the two conditions.

Summary of Results Using an emailed survey invitation with the first question embedded: Improves the email click rate Improves the survey completion rate Has only a small negative effect on the survey drop-out rate Has no effect on data quality, in terms of responses to the first (embedded) question

Discussion Overall, successful test for adding a new feature. Additional advantage: even if respondents drop out of the survey, their answers to the first question will be recorded in the embedded condition. In the standard condition, if respondents drop out before completing the first page, all data will be lost.      Future research opportunities: Embedding more than one question (or the whole survey?) in the email itself Experimenting with different survey lengths and question types Will this work with a different population of respondents?

Thank you Contact us at: Nicki@surveymonkey.com Mingnanliu@fb.com