1. January 2012 Choosing the Optimal Survey Mode: A Comparison of Web, Phone, Mail or In-Person Surveys AEA Coffee Break National Research Center, Inc. 3005 30th St. Boulder, CO 80301 t: 303.444.7863 f: 303.444.1145 e. michelle@n-r-c.com Presented by Michelle Kobayashi

2. Common Survey Modes © 2011 National Research Center, Inc. 2 MailPhoneWeb In-Person SAQ Interview 1-1 interviews Group administered SAQs SAQ administered via Web (often through link to clouds) Participants answer questions on landline or cell phone SAQ mailed to/from sample MailPhone In- Person Web

3. Comparison Criteria Representation Response Rates Inclusion of Low Literacy Respondents Inclusion of Non- English Speakers Decreasing respondent Burden Accuracy Mitigation of Social Desirability Bias Mitigation of Interviewer Bias Ability to use screening questions, skip patterns, visual aids Logistics CostSpeed © 2010 National Research Center, Inc. 3

4. Group Administered SAQ © 2010 National Research Center, Inc. 4 High response Fast turn- around Low burdenLow cost Greater Candor Eliminates Interviewer Biase Consider using if… Participants are captive Participants gather at specific place

5. Mail Surveys © 2011 National Research Center, Inc. 5 Lower Cost Moderate Response Greater Candor No Interviewer Bias Consider using if… Larger sample needed Cost is important Bias is an issue You have time

6. Phone Surveys © 2010 National Research Center, Inc. 6 Fast Allows Screening/skip patterns Mitigates Literacy Issues Useful in Multi-lingual Populations

7. © 2011 National Research Center, Inc. 7 Issues with Phone Data Collection Plummeting response ratesPlummeting response rates " Response rates for Pew Research polls typically range from 5% to 20%; these response rates are comparable to those for other major opinion polls.” Due to...Due to... –Decreasing coverage of landlines –Screening –Annoyance –Other http://pewresearch.org/pubs/2120/global- attitudes-polling-survey-methods?src=prc- newsletter

8. Consider using phone if… Population has strong connection to sponsor.Population has strong connection to sponsor. Population has a high rate of illiteracy or are Non-English speakers.Population has a high rate of illiteracy or are Non-English speakers. Cell phones can be afforded.Cell phones can be afforded. Primary audiences trust phone surveys most.Primary audiences trust phone surveys most. © National Research Center, Inc. 2011 8

9. Web Surveys © 2011 National Research Center, Inc. 9 FastInexpensive Screening questions and skip patterns can be used Greater Candor Eliminates Interviewer Bias Consider using if… Participants are captive Participants have strong connection to sponsor Participants can access computer/email An inclusive sampling frame exists

10. Typical Survey Costs ( Large Population Surveys) Mail 400 completes $20K 1,000 completes $27K Phone 400 completes $27K+ 1,000 completes $34K+ Web $8-10K © 2010 National Research Center, Inc. 10

11. Surveying Using Multiple Modes Increases representation and response ratesIncreases representation and response rates Analysis/interpretation can be a challengeAnalysis/interpretation can be a challenge Multiple modes of invite – but like modes of responseMultiple modes of invite – but like modes of response © 2010 National Research Center, Inc. 11

12. © National Research Center, Inc. 2011 12 Selecting a survey method Comparison of Survey Administration Methods IssuePhoneMail Personal interview Web survey In-Person Group Administered SAQ Expensemoderatelessmostleastless Speed of administrationfastestmoderateslowestfastest Providing a high response ratelowmoderatehighestlowesthighest Obtaining candid responsesmoderatebestworstbest Eliminating interviewer biasmoderatebestworstbest Getting at in-depth topicsmoderate bestmoderate Permitting the use of visual aidsworstbetterbestmoderatebest Enforcing question orderbestworstbestworst Including respondents of lower socio-economic status moderateworstmoderateworstmoderate Accessing respondents from specific geographic locations worstbest worstvaries

13. © 2011 National Research Center, Inc. 13 Thank you! National Research Center, Inc. 3005 30th St. Boulder, CO 80301 t: 303.444.7863 f: 303.444.1145 e. michelle@n-r-c.com Presented by Michelle Kobayashi