1. 1 © 2008 Arbitron Inc. Arbitron Diary Service Offering Respondents a Choice of Survey Mode: Use Patterns of an Internet Response Option in a Mail Survey Robin Gentry DC AAPOR Web Survey Methods Workshop September 10, 2009

2. 2 © 2007 Arbitron Inc. Overview The Arbitron Diary Methodology eDiary Pilot Test The Revised eDiary Methodology Return Rate Effect Who Chooses the eDiary? Mode Effects? Where Do We Go from Here?

3. 3 © 2007 Arbitron Inc. The Arbitron Diary Methodology Conducted in approximately 280 radio markets 48 one-week survey periods per year » Combined into four quarterly ratings releases Recruited by RDD phone survey with an ABS based CPO supplement Everyone 12 or older in household asked to participate Respondent households mailed a 7-day radio listening diary for each household member

4. 4 © 2007 Arbitron Inc. The Paper Diary

5. 5 © 2007 Arbitron Inc. The eDiary Pilot Test Purpose: Feasibility test of 3 web versions of diary Conducted in Fall 2003 Recruited past diary survey households by phone for a web version of survey after confirming computer ownership/Internet usage Collected e-mail addresses for up to three household members » 493 households (with 763 unique e-mail addresses)

6. 6 © 2007 Arbitron Inc. Test Methodology E-mail sent within 24 hours explaining the survey process E-mail direct link to eDiary the day before the start of the survey week Follow-up e-mail sent two days later reminding respondents to complete the survey » Additional follow-ups sent if no activity on account Thank-you e-mail sent the day after the end of the survey week reminding households to submit their completed survey Simulates the mail-based contacts with households receiving paper diaries

7. 7 © 2007 Arbitron Inc. Three Versions of Web Survey Version 1: Paper Diary Clone » Designed to model paper diary with standard day parts and open time frames

8. 8 © 2007 Arbitron Inc.

9. 9 © 2007 Arbitron Inc. Three Versions of Web Survey Version 1: Paper Diary Clone » Designed to model paper diary with standard day parts and open time frames Version 2: Medium Assist Diary » Eliminates day parts and adds auto-completion of previous stations and drop-down boxes

10. 10 © 2007 Arbitron Inc.

11. 11 © 2007 Arbitron Inc. Three Versions of Web Survey Version 1: Paper Diary Clone » Designed to model paper diary with standard day parts and open time frames Version 2: Medium Assist Diary » Eliminates day parts and adds auto-completion of previous stations and drop-down boxes Version 3: Full Assist/Build-a-Diary » Allows the user to build the diary one line at a time. Offers lots of assistance including look-up of stations in the local area.

12. 12 © 2007 Arbitron Inc.

13. 13 © 2007 Arbitron Inc. Results of Test Diary completion rate of 38% (n=290) » Similar across all 3 versions Listening Estimates » Test not designed to produce projectable audience estimates » Given this, overall listening levels were similar to paper diary » All 3 web-page versions resulted in similar reported quarter hours

14. 14 © 2007 Arbitron Inc. Conclusions of Test All three versions had similar usability The recruitment methodology could be improved » E-mail addresses are difficult to collect and verify over the phone. Of 763 email addresses collected, over 100 error messages » Respondent must proxy report for other household members » Spam blockers and virus checkers cause trouble » People don’t always read their e-mail in a timely fashion

15. 15 © 2007 Arbitron Inc. Revised eDiary Methodology Give respondents a choice of mode: paper or eDiary » Send paper diaries to all household members » Include an insert with instructions for using the eDiary » Also, mention the option during recruitment and follow-up calls » Each household member can choose their preferred mode Chose the medium assist version (#2) » Similar to the paper survey but takes advantage of some positive aspects of the web Those registered for the eDiary get personalized reminders if they provide an e-mail address at registration » In addition to the phone/mail reminders delivered to the household

16. 16 © 2007 Arbitron Inc. eDiary Insert

17. 17 © 2007 Arbitron Inc. Larger Scale Test Conducted in Winter 2005 Households screened for computer ownership/web usage 173 eDiaries returned out of 2507 total returns (6.9%) No significant differences in listening behavior overall or by day of week or format Decision made to implement the eDiary as part of the standard methodology

18. 18 © 2007 Arbitron Inc. Unexpected Return Rate Decline Conducted live test in Winter 2007 » Approximately 30,000 households (63,000 individuals) » Half received eDiary mentions/insert, half received usual paper diary only Expected no impact on return rates Instead, return rate was 4.2 points lower in group offered the eDiary option (56.4% vs. 60.6%) One possible explanation—the eDiary insert » Despite emphasizing the choice of modes in phone calls, respondents may have thought they had to do the survey online » Quickly developed and tested a revised insert

19. 19 © 2007 Arbitron Inc. Revised eDiary Insert

20. 20 © 2007 Arbitron Inc. Some Improvement Second Test in Winter 2007 » Approximately 25,000 households (55,000 individuals) » Again, half received eDiary mentions/insert, half received paper diary only Smaller return decline—3.4 points vs. 4.2 points in original test Other explanations » Some respondents complete the diary but forget to submit it » Callback study suggests some had technical difficulties or confusion about registering for eDiary » These likely account for about 1 point of return decline

21. 21 © 2007 Arbitron Inc. Is the Choice the Problem? Similar findings for the American Community Survey* » Tested a choice between mail/Internet response in 2000/2001 » Return rate was 5.8 points lower in group offered mode choice ACS cited three explanations for response declines » Some respondents attempted the web survey but quit before completion and didn’t return paper instead » Offering a choice caused respondents to put the task aside and disrupted the response process » Notices about Internet reporting may have caused respondents with privacy concerns Similar findings in Grigorian and Hoffer (2008) and Smythe, Dillman, Christian and O’Neill (in press) *Griffin, D.H., D.P. Fisher and M.T. Morgan. “Testing an Internet Response Option for the American Community Survey.” Presented at the annual conference of the American Association of Public Opinion Research, May 17-20, 2001 in Montreal, Quebec, Canada

22. 22 © 2007 Arbitron Inc. Who Chooses the eDiary? 3.8% of respondents offered the choice of modes registered for the eDiary 4.5% of usable returned diaries were eDiaries Once registered for eDiary, respondent was likely to return a survey » 90% of those registered entered at least one listening episode » 73% returned a usable eDiary » An additional 8% returned a paper diary instead

23. 23 © 2007 Arbitron Inc. Age by Mode Choice

24. 24 © 2007 Arbitron Inc. Sex by Mode Choice

25. 25 © 2007 Arbitron Inc. Race/Ethnicity by Mode Choice

26. 26 © 2007 Arbitron Inc. Mode Effects? We intentionally used a web design that closely mimicked the paper diary. » Took advantage of some error checking features of the web (e.g. time verification, checking for complete entries) » Didn’t take full advantage of all possible web enhancements (e.g. look-up of local stations) because it might introduce mode effects No significant difference in amount of listening by mode » Average Quarter Hours of Listening: 73.4 for eDiaries and 73.9 for Paper Diaries Difference of 7.5 minutes over the entire week No difference in listening amounts or radio formats when test methodology (eDiary option) compared to control (no eDiary option)

27. 27 © 2007 Arbitron Inc. Testing Conclusions Don’t assume that adding a mode choice will necessarily improve response rates » In fact, it may produce the opposite result Designing an instrument which mimics the paper version may reduce mode effects, but may be less user friendly Given the choice, relatively few respondents choose the web option If your target population is age 25-54, a web option may be a good compliment to your paper survey

28. 28 © 2007 Arbitron Inc. Where Do We Go from Here? Mode Sequencing » Offer the Web option to those who don’t respond to the initial request Collect e-mail addresses from eConsenters » Likely to be web-savvy and less likely to have concerns about giving an e-mail address » Allows us to send a link to survey URL De-emphasize the mode choice » Still offer the Web option but emphasize less in materials and calls » “Prefer the Web? Internet eDiary option now available at www.arbitrondiary.com www.arbitrondiary.com –Remove the insert –Choice info printed only on letter and the diary itself

29. 29 © 2007 Arbitron Inc. Questions? robin.gentry@arbitron.com (410) 312-8389