1. AIR 2000 On-line vs. Paper-and-Pencil Surveying of Students’ Willingness to Re-enroll: A Case Study Phil Handwerk, Cristi Carson, and Karen Blackwell University of North Carolina at Greensboro AIR Forum May 2000

2. AIR 2000 Intro & Purpose Technology has made surveying on-line convenient, efficient, and affordable Questions regarding use of on-line surveying: Do on-line respondents differ from traditional, paper-and-pencil respondents? What are the pros/cons of surveying on-line? How can on-line surveys be improved?

3. AIR 2000 Methodology 14 closed-ended questions concerning: student’s evaluation of campus environment survey preferences Random sample of 3,000 undergraduate students 1,500 Paper-and-Pencil Survey 2 mailings with survey 1,500 On-line Survey Initial e-mail and postcard 2 e-mail reminders Analyzed results (answers and demographics) by Survey Method (Chi-sq) 2 follow-up Focus Groups Those who would choose UNCG again Those who would not choose UNCG again or were not sure

4. AIR 2000 Findings: Do on-line respondents differ from traditional, paper-and-pencil respondents? Both on-line & paper-and-pencil respondents are representative of population Response Rates Differed Significantly (p<0.0014) 26.2% on-line 33.3% paper-and-pencil

5. AIR 2000 Findings: Do on-line respondents differ from traditional, paper-and-pencil respondents? No significant demographic differences: Sex Race Class Housing (on or off campus)

6. AIR 2000 Findings: Do on-line respondents differ from traditional, paper-and-pencil respondents? *Significant difference (p<0.0014) occurred across Survey Method

7. AIR 2000 No significant attitudinal differences: Whether or not would choose UNCG again Across three item scales identified through factor analysis Findings: Do on-line respondents differ from traditional, paper-and-pencil respondents?

8. AIR 2000 Findings: Do on-line respondents differ from traditional, paper-and-pencil respondents? *Significant difference (p<0.01) occurred across Survey Method

9. AIR 2000 Findings: What are the pros/cons of surveying on-line? *Significant difference (p<0.0014) occurred across Survey Method

10. AIR 2000 Findings: What are the pros/cons of surveying on-line? IR perspective: Pro: Respondents do data entry Pro: Avoid most printing and mailing costs Pro: Monitor field in real time Con: What if student can’t get to a computer? Student perspective: Pro: Convenient Pro: Don’t have to keep track of survey & remember to mail Con: Getting to a computer

11. AIR 2000 “What is the maximum length of time you are willing to spend on a survey, such as this, regarding UNCG?” Mean: 13 min., Mode: 10 min., Median: 10 min. No significant differences across survey method completed No significant differences across preferred method of future surveys Findings: How can on-line surveys be improved?

12. AIR 2000 Findings: How can on-line surveys be improved? Incentives (from focus groups) $100 lottery for survey completion Students didn’t notice/would have completed anyway Students suggested: Movie tickets Gift certificates (shopping, restaurants) E-mailing Had used Student’s university account Few check/redirect university account Get preferred e-mail when collecting address & phone

13. AIR 2000 Conclusions Does a higher response rate for paper imply: Greater comfort with format? Lack of access to technology? On-line respondents do not appear to differ in profile (demographics & attitudinal) compared to paper-and-pencil respondents. Perhaps higher % of <18-24 yr olds on-line is a sign of growing enthusiasm/comfort with on-line surveys? Why higher % and favorability in on-line comments?

14. AIR 2000 Conclusions Most students either prefer on-line surveys (for several reasons) or don’t have a preference on format. Regardless of format, most students only willing to spend 10 minutes on a university- focussed survey. Monetary incentive not necessarily a motivating factor. University e-mail accounts may not be best way of contacting students.

15. AIR 2000 Implications Surveying on-line can be an efficient and reliable means by which to poll students. As familiarity with technology continues to expand, perhaps an increased comfort and willingness to take on-line surveys may arise. Surveys, regardless of format, need to be brief. Creativity with incentives may pay-off. Search for alternative ways of approaching students to complete an on-line survey other than using their University e-mail accounts.