Association for Institutional Research Annual Forum May 2014 Amber D. Lambert, Ph.D. Angie L. Miller, Ph.D. Center for Postsecondary Research, School of Education, Indiana University Living with Smartphones: Does Completion Device Affect Survey Responses?

Literature Review In higher education, surveys are used frequently for collecting information to demonstrate effectiveness (Kuh & Ikenberry, 2009) Example purposes: curriculum improvement, internal evaluation, accreditation, outcomes assessment, strategic planning Student surveys are most prominent, but surveys are also used to gather information from other stakeholders, including faculty, staff, and alumni (Cabrera et al., 2005; Kuh & Ewell, 2010)

Literature Review Although online surveying is more efficient and convenient, survey response rates have actually been falling (Atrostic et al., 2001; Baruch, 1999; Porter, 2004) Transition to web-based surveys over the past decade has generated much research on the new mode of delivery (Dillman, 2007) Initial concerns over sampling bias and coverage (Couper, 2000)

Literature Review As internet access grows, now research focuses on how mode impacts responses themselves: Content of one’s response (Descombe, 2006) Humanizing aspects of interface (Tourangeau et al., 2003) Technology to make survey dynamic (rather than static) (Norman et al., 2001)

Literature Review Also research on web-based surveys and: Page breaks and scrolling (Peytchev, 2009) Effectiveness of progress bars on completion (Villar et al., 2013) Browser compatibility and response placement (Kaye & Johnson, 1999) Color contrast and placement of emphasis (Tourangeau, 2004)

Literature Review Research on web-based surveys now must shift away from laptops and desktops to smartphones and tablets Mobile devices offer internet access virtually anywhere, but touch screen functioning, truncated viewing area, and smaller keyboards can place additional burdens on survey respondents (Buskirk & Andrus, 2012; Peytchev & Hill, 2010)

Literature Review Recent research comparing survey patterns between PC, tablet, and smartphone users found: Young people more likely to use smartphones, young and employed people more likely to use tablets (de Bruijne & Wijnant, 2014) Mobile phone respondents have lower completion rates, shorter open-ended answers, and are younger; no gender or education differences (Mavletova, 2013)

Research Questions Goals of this study: A) explore patterns in responses to a multi-institution alumni survey, looking at how type of completion device is impacted by various demographic variables, including age, income, gender, and employment status B) examine relationships between type of device and other survey-taking characteristics, including breaking off, backing up, time duration, item nonresponse, and open-ended text box completion

Method: Participants Data from the 2012 and 2013 administrations of the Strategic National Arts Alumni Project (SNAAP) Participants were 58,768 alumni from 109 different arts high schools, arts colleges, or arts programs within larger universities Sample consisted of 2% high school level, 76% undergraduate level, and 22% graduate level alumni 41% male, 59% female,.2% transgender Majority (85%) reported ethnicity as Caucasian Average institutional response rate: 18%

What is SNAAP? On-line annual survey designed to assess and improve various aspects of arts-school education Investigates the educational experiences and career paths of arts graduates nationally Questionnaire topics include: Formal education and degrees Institutional experience and satisfaction Postgraduate resources for artists Career Arts engagement Income and debt Demographics

Method: Metadata Measures Completion device: tracked through data collection platform- PC (42%), Mac (43%), Smartphone (9%), and Tablet (5%) (with an “other”.4% not traceable) Breakoff: did respondents reach the end of the survey and hit the “submit” button? Backup status: did the respondent go back (using the browser) to previously completed pages?

Method: Metadata Measures Time duration: how long (in minutes) did respondents spend with the survey open in their browser? Item nonresponse: did respondents answer complex “matrix layout” items without leaving any missing? Open-ended text boxes (10 total throughout survey): did respondents write ANY response? If so, how long was their response?

Method: Metadata Measures Example of “matrix layout” question sets:

Method: Demographic Measures Demographic information collected for: Continuous variables of age (write-in number box) and income (used midpoints of response ranges) Categorical variable of gender Male, female, transgender Categorical variable of current employment status Full-time (35 hours or more per week) Part-time only Unemployed and looking for work In school full time Caring for family full time Retired Other

Analyses Series of 16 chi-squared analyses was done for each of the categorical metadata and demographic variables For gender, current employment status, completion status, backup status, item nonresponse status (for two sets of matrix layouts), and open-ended response status (for 10 open-ended questions) Series of ANOVAs and Mann-Whitney tests for continuous metadata and demographic variables For age, income, duration, and length of open-ended responses

Results: Demographic Variables Smartphone users are significantly younger, while tablet users have significantly higher income AgeIncome PCMean N Std. Deviation MacMean N Std. Deviation Smart PhoneMean N Std. Deviation TabletMean N Std. Deviation OtherMean N Std. Deviation TotalMean N Std. Deviation F Sign 0.000

Results: Demographic Variables Women are more likely to use tablets and smartphones; retired people less likely to use smartphones but more likely to use tablets PCMacSmart PhoneTabletTotal Count% % % % % Gender Male % % % % % Female % % % % % Transgender 24.1%35.2%13.4%7.3%790.2% Employment Status Full-time (35 hours or more per week) % % % % % Part-time only % % % % % Unemployed and looking for work %7383.4%1284.1%762.9% % In school full time %7283.4%1344.3%642.5% % Caring for family full time %2771.3%832.7%783.0%6981.4% Retired % %902.9% % % Other % %1825.8%1696.5% %

Results: Metadata Variables Smartphone users took a significantly longer amount time (selecting only for those who completed the survey) MedianN Std. Deviation PC Mac Smart Phone Tablet Other Total F Sign 0.000

Results: Metadata Variables Smartphone users were far more likely to break off, but all device users were equally likely to back up to previous pages PCMacSmart PhoneTabletTotal Count% % % % % Completion Status Complete % % % % % Partial complete % % % % % Backup Status Respondent did not back up in survey % % % % % Respondent backed up in survey % %2184.0%1524.9% %

Results: Metadata Variables Smartphone users were more likely to provide complete responses to complicated layout questions (i.e. lower item nonresponse) PCMacSmart PhoneTabletTotal Count% % % % % Response to Complicated Question 1 Did not respond to all items % %1665.3%2409.3% % Did respond to all items % % % % % Response to Complicated Question 2 Did not respond to all items % %1655.3%2198.5% % Did respond to all items % % % % %

Results: Metadata Variables Overall, smartphone and tablet users were less likely to write responses to open-ended questions PCMacSmart PhoneTabletTotal Count% % % % % Question 1 Respondent did NOT write something % % % % % Respondent did write something % % % % % Question 2 Respondent did NOT write something % % % % % Respondent did write something % % % % % Question 3 Respondent did NOT write something % % % % % Respondent did write something % % % % % Question 4 Respondent did NOT write something % % % % % Respondent did write something % % % % % Question 5 Respondent did NOT write something % % % % % Respondent did write something % %2357.5%1937.4% %

Results: Metadata Variables PCMacSmart PhoneTabletTotal Count% % % % % Question 6 Respondent did NOT write something % % % % % Respondent did write something % % % % % Question 7 Respondent did NOT write something % % % % % Respondent did write something % % % % % Question 8 Respondent did NOT write something % % % % % Respondent did write something % % % % % Question 9 Respondent did NOT write something % % % % % Respondent did write something % % % % % Question 10 Respondent did NOT write something % % % % % Respondent did write something % %1825.8%1806.9% %

Results: Metadata Variables And of those who did write open-ended text responses, smartphone and tablet users overall wrote significantly shorter responses Question 1Question 2Question 3Question 4Question 5 PCMedian N Std. Dev MacMedian N Std. Dev Smart PhoneMedian N Std. Dev TabletMedian N Std. Dev OtherMedian N Std. Dev TotalMedian N Std. Dev F Sign 0.000

Results: Metadata Variables Question 6Question 7Question 8Question 9Question 10 PCMedian N Std. Dev MacMedian N Std. Dev Smart PhoneMedian N Std. Dev TabletMedian N Std. Dev OtherMedian N Std. Dev TotalMedian N Std. Dev F Sign 0.000

Discussion Many patterns of results are consistent with previous literature Differences on type of device used based on age and employment status Potential generational effects? Unlike previous studies, also found differences for gender and income Tablets as “luxury” items? Are women more compliant to survey requests, regardless of device?

Discussion Types of devices do seem to affect (in some ways) respondents’ survey-taking behaviors Also mirrors previous literature that taking surveys on smartphones and tablets can increase respondent burden Smartphone users are far more likely to abandon the survey, and those who do finish require more time to complete it Smartphone and tablet users were less likely to answer open-ended questions, and when they did answer them their responses were much shorter

Discussion Interestingly, smartphone users were more likely to fully complete complex layout item sets Counterintuitive at first glance because these questions may require more scrolling (vertically and horizontally) on a truncated screen, so one would expect fewer complete responses Could be that those who persevered to these points in the survey on a smartphone (about 1/3 and ½ of the way through) are the more dedicated and conscientious survey takers

Conclusions Limitations of study: sample may not be completely representative of all survey takers (only arts alumni, lower response rates, and selective participation) When designing web-based surveys, need to take into account that respondents may use smartphones and tablets May need to rely less heavily on long layouts and open- ended questions Future research continuing to look at device type is necessary as technology rapidly becomes available to larger populations Convenience of “anytime, anywhere” internet access may have negative impact on data quality

Questions or Comments? Contact Information: Amber D. Lambert Angie L. Miller Strategic National Arts Alumni Project (SNAAP) (812) *Reference list available upon request or in full paper