ESRC Research Methods Festival, Oxford, July 2006 The Hunt for the Last Respondent How to decrease response rates and increase bias Ineke Stoop

ESRC Research Methods Festival, Oxford, July 2006 Enhance response rates Minimize nonresponse bias Enhance nonresponse rates Maximize nonresponse bias

ESRC Research Methods Festival, Oxford, July 2006 Design study Field experiences How to decrease response rates? Keeping the customer satisfied How to spruce up response rates? The sophisticated question How to maximize nonresponse bias? Sources International nonresponse literature European Social Survey Dutch experimental study Mainly face-to-face surveys

ESRC Research Methods Festival, Oxford, July 2006 How to decrease response rates?

ESRC Research Methods Festival, Oxford, July 2006 Preparation Short fieldwork period Timely results Fieldwork contracting Dont specify: Survey organization knows best Stick to national and organisational traditions

ESRC Research Methods Festival, Oxford, July 2006 Interviewers Foot soldiers of survey research Economize on interviewer training and instruction Experienced interviewers know best how to act Turnover rate is high and CAPI questionnaires provide instructions anyway Economize on interviewer remuneration Not a real job No close monitoring fieldwork Interviewers prefer organizing their own schedules Too much rules and control will lessen motivation

ESRC Research Methods Festival, Oxford, July 2006 Information and incentives No advance letters and brochures Nobody reads them Advance letters just scare off target respondents No toll-free telephone number In order to prevent respondent refusals No incentive required A survey is expensive enough as it is

ESRC Research Methods Festival, Oxford, July 2006 Contacting sample persons Limit the number of contact attempts Face-to-face calls during office hours only Interviewers have a family life too Evening visits too dangerous Dont send interviewers to problematic inner city areas Dangerous and low response rate anyway Follow-up on appointments only when you need more respondents Stop contact attempts when you have enough respondents

ESRC Research Methods Festival, Oxford, July 2006 Contactability ESS R1

ESRC Research Methods Festival, Oxford, July 2006 At home, interviewer calls, contact rate

ESRC Research Methods Festival, Oxford, July 2006 Obtaining cooperation Use a questionnaire efficiently and include as many questions as you can (better safe than sorry) Matrix questions Small print (seems shorter) Do not spend too much time and effort on difficult respondents Elderly people, language problems, rarely at home No bargaining about timing interviews No refusal conversion (a refusal is a refusal) Stick to a single interview mode

ESRC Research Methods Festival, Oxford, July 2006 ESS: share of re-approached refusals that were converted

ESRC Research Methods Festival, Oxford, July 2006 Bad practices What happens if you do not take the previous recommendations seriously

ESRC Research Methods Festival, Oxford, July 2006 Response rates ESS R1 and R2

ESRC Research Methods Festival, Oxford, July 2006 Dutch experiment F2F survey + long drop-offs every household member 67% response rate Follow-up survey lite among sample of persistent refusals +70% responding refusals

ESRC Research Methods Festival, Oxford, July 2006 Why did refusers cooperate? High quality interviewers (and telling them they are the best), motivation, remuneration Extensive briefing Trust (money for incentives) and support (newsletter) interviewers Perceived importance (newspaper article, involvement management) Wide range of incentives F2F, PAPI, CATI and internet Commitment sponsor and fieldwork organisation

ESRC Research Methods Festival, Oxford, July 2006 Compare easy, hard to get, initial refusals and final refusals Differences small Initial refusals more mr. and mrs Average than easy respondents Hard to reach respondents different Final refusals: less participation, less religious, fewer PCs, more popular culture

ESRC Research Methods Festival, Oxford, July 2006 What if you cant get away with low response rates

ESRC Research Methods Festival, Oxford, July 2006 Do not spend an afwul lot of money

ESRC Research Methods Festival, Oxford, July 2006 Focus on promising groups Rural areas Families with small children No apartment dwellers People who can be reached on the phone

ESRC Research Methods Festival, Oxford, July 2006 Better: redefine the denominator Lower upper age limit (older people take too long) Raise lower age limit (parental permission or never at home) Exclude non-native speakers Exclude mobile phone only Exclude ex-directory telephone numbers Exclude non-internet users Exclude unused sample units from the net sample Include only those who have agreed on the telephone to participate in a f2f survey Members of access panels (pre-recruirment panels) Only those panel members that (almost always) cooperate

ESRC Research Methods Festival, Oxford, July 2006 Select conscientious access panel members: 70% or 80% guaranteed

ESRC Research Methods Festival, Oxford, July 2006 Still better Make nonresponse impossible Substitute nonrespondents by family members or neighbours Quota sampling Volunteer panels

ESRC Research Methods Festival, Oxford, July 2006 Added benefit maximize bias if topic of the survey is related to response mechanism or selection mechanism

ESRC Research Methods Festival, Oxford, July 2006 Different types of success Do a poor job Low response rates Focus on easy respondents and exclude different groups Adequate response rates, large nonresponse bias Pre-select willing panel members Vey high response rates, low nonresponse bias, poor survey quality

ESRC Research Methods Festival, Oxford, July 2006 Are high response rates better? Maximum bias smaller Difficult respondents similar to final nonrespondents? Hard to reach respondents similar to noncontacts? Converted refusals similar to final refusals?

ESRC Research Methods Festival, Oxford, July 2006 Assumptions valid? High response surveys not necessarily better than low response surveys Possibly because efforts are directed at easy nonrespondents (situational refusals) And survey-related or topic related refusal is left alone Low response rate especially harmful if nonparticipation is caused by single factor related to survey Political interest (Groves, Presser, Dipko, 2004)

ESRC Research Methods Festival, Oxford, July 2006 Increased fieldwork efforts Groves (1989) Devoting limited resources to increasing response rates with little or no impact on survey error is not money well spent Krosnick (1999) Prevailing wisdom that higher response rates are necessary for sample representiveness is being challenged Curtin et al. (2000) Large differences in response rates have only minor effects Keeter et al. (2000) Similar results Standard and Rigorous survey, significant differences between demographics Merkle and Edelman (2002) Response rates in exit pools not related to error Teitler et al. (2003) Diminishing returns at higher effort levels Loosveldt et al. (2003) ESS: converted refusals do differ in some countries (with high refusal conversion rate)

ESRC Research Methods Festival, Oxford, July 2006 What to do? If obtaining high response rates is difficult If high response rates not necessarily imply high quality If interviewing difficult respondents does not always give better insight in final nonrespondents If enhancing response rates means getting more of the same If you want to minimize bias

ESRC Research Methods Festival, Oxford, July 2006 Aim for high response rates Close monitoring High contact rate in all areas among all groups Obtain cooperation through multiple means Respect respondents Provide information Internal and external incentives Motivated interviewers Organisational support Mixed mode interviewing

ESRC Research Methods Festival, Oxford, July 2006 But maybe do not bother about the final few % Collect and code Sample information Observational data Fieldwork data (contact and cooperation) Information on respondent and nonrespondents related to the topic of the survey Other sources Central Question procedure Survey among nonrespondents

ESRC Research Methods Festival, Oxford, July 2006 Go for bias Surveys are very expensive: Not enhance response rates indiscriminately Not high response rates combined with non-probability sampling and high coverage errors Not high response rates within specific groups Collect and use auxiliary information to assess bias and to adjust for bias

ESRC Research Methods Festival, Oxford, July 2006 Treat data collection with a scientific, controlled approach as used in sampling and data analysis

ESRC Research Methods Festival, Oxford, July 2006