Self-Administered Surveys: Mail Survey Methods ChihChien Chen Lauren Teffeau Week 10

Dillman, 1991 Why MAIL surveys? Lower costs Perception of simple procedures Large scale general public Time People with hearing problem

Dillman Cont’d Sampling Error Noncoverage Error – incomplete sampling frame Nonresponse Error – people don’t answer Measurement Error – instrument deficiencies Sources of Survey Error:

Reducing Nonresponse Error Why is increasing response rate important in mail survey design? Hard to make the comparison between respondents and nonrespondents. What and How do they differ?

Dillman Cont’d Reducing Nonresponse Error…  Response Rate is indicator How to improve Response Rates? Subjective assessments: follow-up mailings, prior notice, incentives, return postage, sponsorship, personalization, interest in survey topic Quantitative assessments via meta-analysis: number of contacts (initial and follow-up); salience of topic; government sponsorship; employee, school or army population; incentive on first contact – Negative effects: market research, sponsorship, general population & length

Understanding Mail Survey Response Behavior: A Meta-Analysis (Yammarino, Skinner & Childers, 1991) To enhance understanding of mail survey response behavior by presenting a more comprehensive meta-analysis investigation of factors that affect mail survey response rates.

Yammarino et. al. Cont’d Feature 2: combined meta-analysis procedures Features 3: more factors through a larger number of study outcomes Features 4: investigation of two moderator variables Feature 1: incorporate several aspects of a properly conducted meta-analysis

Yammarino et. al. Cont’d Key results –a cover letter that includes appeals and a survey of less than four pages works for all target population –a reply/return envelop and monetary incentives are situation specific and need to consider other moderators –a preliminary notification subject to time period. In the study: ’76-87 –a stamped/metered return postage varied with time. In the study: ’40-70 differed for samples of groups –a follow-ups/repeated contacts differed from samples of groups) –Examine more moderators in subgroup characteristics of respondents, additional details about samples, way of survey delivered and situation (time & location) in which surveys are administered/completed

Dillman Cont’d Total Design Method – What is it? Recommended procedures that should be used holistically versus a la carte Based on Social Exchange theory Results in high response rates What are some limitations? Lack of empirical evidence Ignores the effect of financial incentives Over-reliance on social exchange? No tailorization for different populations

Dillman Cont’d Updating Total Design Method: Only a perspective on how to stimulate responses, not definitive… Need for updating; hence, ‘Tailored’ Design Method, 2000 Tailored b/c computers make it easier to customize surveys to specific populations Attempts to create respondent trust, perceptions of increased rewards, and reduced social costs to respondents Increased emphasis on visual design

Dillman Cont’d How to reduce Measurement Error? In mail surveys, measurement error usually stems from: Item non-response Poor open-ended responses Low education of respondents Complex skip patterns Question order Importance of pre-testing Fostering response conditions: cover letter, follow-ups, incentives etc. Know your population Limit skips and craft clear questions (wording and visual placement on survey)

Dillman Cont’d Less chance of interviewer subversion Less social desirability pressure How do Mail surveys impact respondents’ answers? What special considerations must researchers make for Mail surveys? Slower and potentially more costly data collection Cannot forget self-administered environment: - Clarity of questions and instructions paramount to lessen measurement error - Issue of who is answering survey (may not be intended respondent; salience of topic may influence returns)

Dillman Cont’d –Self-administered lessens order effects –Mail respondents less extreme answers –Mail respondents less social desirability –Pacing of mail surveys – more cognition time vs off-the-top-of-the-head answers What are some advantages of Mail over Telephone surveys? What about Context effects? Respondents don’t have to answer surveys linearly (Schwarz & Hippler – norm of evenhandedness) If in doubt, use split-ballot to control for potential effects. Use priming to your advantage if possible

Dillman Cont’d How to reduce Noncoverage Error? –Important especially with mail because of reliance on household addresses, which are usually out-of-date –Area of further research and development of practices Have accurate sampling frame. What if you don’t? Find a different way to do your research OR create your sampling frame = much more cost and more lead time.

Maximizing Response Rates You get a national grant to administer a mail survey. What steps would you follow to get the highest response rate possible if expense was not an issue? You find out that you will only get a fraction of the funding you need to administer a survey for your dissertation. What steps would you follow to get the highest response rate relative to your budget?

Web Surveys are Self-Administered Too! What is similar between web and mail surveying techniques? What what do you need to account for if doing a web instead of mail survey? What are the advantages of mixed methods?