EMR 6500: Survey Research Dr. Chris L. S. Coryn Kristin A. Hobson Spring 2013

Agenda Implementation procedures When more than one survey mode is needed

Implementation Procedures

Mail Survey Implementation

When high quality implementation procedures are used, response rates of 50%-70% are not uncommon – In my own work, I have had response rates as high as 90% Implementation must receive considerable thought and planning – It should not occur after questionnaire design, but simultaneously

Mail Survey Implementation General guidelines 1.To the extent possible, personalize all contacts to respondents (even when names are unavailable) 2.Send a token of appreciation with the survey request 3.Use multiple contacts, each with a different look and appeal 4.Carefully and strategically time all contacts 5.Select all mail-out dates with the characteristics of the population in mind

Mail Survey Implementation General guidelines 6.Place information in the mailing exactly where it needs to be used 7.Take steps to ensure that mailings will not be mistaken for junk mail or marketing materials 8.Evaluate the size and weight of mailing materials on mailing costs 9.Assemble the mailings in a way that maximizes the appealing aspects of each element when the package is opened

Mail Survey Implementation General guidelines 10.Ensure that all addresses in the sample comply with current postal regulations 11.Assign an individual ID number to each sample member 12.Establish procedures for dealing with undeliverable mail 13.Establish procedures for dealing with returned incentives 14.Establish procedures for dealing with respondent inquiries 15.Evaluate early returns for problems that can be addressed mid-stride

Prenotice Letter Sent to respondents a few days prior to the questionnaire It notes that a questionnaire is for an important survey that will arrive in a few days and that the person’s response will be greatly appreciated This letter can increase response rates by 3%-6% – No recent research comparing postcards to letters (I prefer the former)

Questionnaire Mailing Includes a detailed cover letter explaining why a response is important The questionnaire A prepaid postage envelope A token incentive if one is to be provided

Thank You Postcard Sent a few days to a week after the questionnaire The mailing expresses appreciation for responding and indicates that if the completed questionnaire has not yet been mailed it is hoped that it will be returned soon

Replacement Questionnaire Sent to respondents 2 to 4 weeks after the previous questionnaire mailing It indicates that the person’s completed questionnaire has not yet been received and urges the recipient to respond

Final Contact Sent using a different mode of delivery 2 to 4 weeks after the previous mailing The different mode of contact distinguishes each type of final contact from regular mail delivery Special contacts have been shown to increase response

Web Survey Implementation

Major concerns associated with web surveys – Mode of initial contact – Extent to which personalized elements can be used – Whether and how incentives are delivered – Timing of contacts

Web Survey Implementation General guidelines 1.To the extent possible, personalize all contacts to respondents 2.Send a token of appreciation with the survey request 3.Use multiple contacts and vary the message across them 4.Carefully and strategically time all contacts with the population in mind 5.Consider contacting respondents by another mode when possible

Web Survey Implementation General guidelines 6.Keep contacts short and to the point 7.Take steps to ensure that s are not flagged as spam 8.Carefully select the sender name and address the subject line text for communications 9.Provide clear instructions for how to access the survey 10.Make obvious connections between the opening screen and other implementation features

Web Survey Implementation General guidelines 11.Assign each sample member a unique ID number 12.Know and respect the capabilities and limits of the web server(s) 13.Establish a procedure for dealing with bounced s 14.Establish procedures for dealing with returned incentives 15.Establish procedures for dealing with respondent inquiries 16.Implement a system for monitoring progress and evaluating early completes

When More than one Survey Mode is Needed

Why Consider a Mixed-Mode Survey Design Lower costs Improve timeliness Reduce coverage error Deliver incentives

Four Types of Mixed-Mode Surveys

Type I Use one mode to contact respondents and to encourage response by a different mode

Type II Use a second mode to collect responses from the same respondents for specific questions within a questionnaire

Type III Use alternative modes for different respondents in the same survey period

Type IV Use a different mode to survey the same respondents in a later data collection period

Mixed-Mode Surveys and their Implications TypeMotivationLimitations I.Use one mode to contact respondents and to encourage response by a different mode Improve response rates Reduce coverage and nonresponse error Increased implementation costs II.Use a second mode to collect responses from the same respondents for specific questions within a questionnaire Reduce measurement error Reduce social desirability bias for sensitive question Increased design costs Increased nonresponse if respondent must respond by other mode at a later time III.Use alternative modes for different respondents in a the same survey period Improve response rates Reduce coverage and nonresponse error Reduce survey costs Increased design costs Measurement error from differences that may be confounded with differences among subgroups IV.Use a different mode to survey the same respondents in a latter data collection period Different modes become available to survey respondents Reduce survey costs Increased design costs Measurement error from mode differences that impact the ability to measure change over time

Why Different Survey Modes Sometimes Produce Different Answers to Survey Questions

Presence versus Absence of an Interviewer Locus of control Normative question order effects Social desirability Acquiescence

Aural versus Visual Communication Effects Primacy/recency effects