January 2012 Choosing the Optimal Survey Mode: A Comparison of Web, Phone, Mail or In-Person Surveys AEA Coffee Break National Research Center, Inc th St. Boulder, CO t: f: e. Presented by Michelle Kobayashi

Common Survey Modes © 2011 National Research Center, Inc. 2 MailPhoneWeb In-Person SAQ Interview 1-1 interviews Group administered SAQs SAQ administered via Web (often through link to clouds) Participants answer questions on landline or cell phone SAQ mailed to/from sample MailPhone In- Person Web

Comparison Criteria Representation Response Rates Inclusion of Low Literacy Respondents Inclusion of Non- English Speakers Decreasing respondent Burden Accuracy Mitigation of Social Desirability Bias Mitigation of Interviewer Bias Ability to use screening questions, skip patterns, visual aids Logistics CostSpeed © 2010 National Research Center, Inc. 3

Group Administered SAQ © 2010 National Research Center, Inc. 4 High response Fast turn- around Low burdenLow cost Greater Candor Eliminates Interviewer Biase Consider using if… Participants are captive Participants gather at specific place

Mail Surveys © 2011 National Research Center, Inc. 5 Lower Cost Moderate Response Greater Candor No Interviewer Bias Consider using if… Larger sample needed Cost is important Bias is an issue You have time

Phone Surveys © 2010 National Research Center, Inc. 6 Fast Allows Screening/skip patterns Mitigates Literacy Issues Useful in Multi-lingual Populations

© 2011 National Research Center, Inc. 7 Issues with Phone Data Collection Plummeting response ratesPlummeting response rates " Response rates for Pew Research polls typically range from 5% to 20%; these response rates are comparable to those for other major opinion polls.” Due to...Due to... –Decreasing coverage of landlines –Screening –Annoyance –Other attitudes-polling-survey-methods?src=prc- newsletter

Consider using phone if… Population has strong connection to sponsor.Population has strong connection to sponsor. Population has a high rate of illiteracy or are Non-English speakers.Population has a high rate of illiteracy or are Non-English speakers. Cell phones can be afforded.Cell phones can be afforded. Primary audiences trust phone surveys most.Primary audiences trust phone surveys most. © National Research Center, Inc

Web Surveys © 2011 National Research Center, Inc. 9 FastInexpensive Screening questions and skip patterns can be used Greater Candor Eliminates Interviewer Bias Consider using if… Participants are captive Participants have strong connection to sponsor Participants can access computer/ An inclusive sampling frame exists

Typical Survey Costs ( Large Population Surveys) Mail 400 completes $20K 1,000 completes $27K Phone 400 completes $27K+ 1,000 completes $34K+ Web $8-10K © 2010 National Research Center, Inc. 10

Surveying Using Multiple Modes Increases representation and response ratesIncreases representation and response rates Analysis/interpretation can be a challengeAnalysis/interpretation can be a challenge Multiple modes of invite – but like modes of responseMultiple modes of invite – but like modes of response © 2010 National Research Center, Inc. 11

© National Research Center, Inc Selecting a survey method Comparison of Survey Administration Methods IssuePhon Personal interview Web survey In-Person Group Administered SAQ Expensemoderatelessmostleastless Speed of administrationfastestmoderateslowestfastest Providing a high response ratelowmoderatehighestlowesthighest Obtaining candid responsesmoderatebestworstbest Eliminating interviewer biasmoderatebestworstbest Getting at in-depth topicsmoderate bestmoderate Permitting the use of visual aidsworstbetterbestmoderatebest Enforcing question orderbestworstbestworst Including respondents of lower socio-economic status moderateworstmoderateworstmoderate Accessing respondents from specific geographic locations worstbest worstvaries

© 2011 National Research Center, Inc. 13 Thank you! National Research Center, Inc th St. Boulder, CO t: f: e. Presented by Michelle Kobayashi