Nonresponse adjustment in the European Social Survey (ESS) Ineke Stoop (SCP-Netherlands) Jaak Billiet (Uni Leuven-Belgium) Achim Koch (Gesis-Germany)

Q2010, Helsinki, May 2010 European Social Survey Content To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions Methodology To advance and consolidate improved methods of cross- national survey measurement in Europe and beyond Input harmonisation

Q2010, Helsinki, May 2010 Basic facts Biennial survey 30+ countries interviews/country Fieldwork organisation/ country Probability sampling Design weights No substitution Face-to-face Fieldwork monitoring Guidelines interviewer training Target noncontact rate: 3% Minimum number of calls Timing of calls (evening, weekend) Target response rate: 70% Guidelines response enhancement Incentives/Brochure Refusal conversion Contact form Interviewer observations Call records Standard response rate calculation

Q2010, Helsinki, May 2010 Factors causing variance in nonresponse bias Survey climate Population characteristics Difficulty to obtain response House effects Fieldwork staff Fieldwork strategy Efforts to obtain high response rates Sampling frames Realised response rates Nonresponse composition noncontact/refusal/not able (language) Reasons for refusal Presence of auxiliary data to adjust for nonresponse bias

Q2010, Helsinki, May 2010 Response rates ESS

Q2010, Helsinki, May 2010 Assessing and correcting for bias in a cross- national study Ideal: same way everywhere Practice: availability, accessibility auxiliary data differs Sampling frames differs Usually no access to population register Quality population statistics Quality paradata Ease of response (wave data) Only approaches that COULD be used in every country?

Q2010, Helsinki, May 2010 Adjusting for nonresponse bias in ESS Post-stratification Aggregated info on sex, age, education Cooperative and reluctant respondents Reasons for refusal, Number of contacts, Interviewer judgment Observational data Interviewer observation for sample and contacts Doorstep questions refusals Very short questionnaire f2f Follow-up survey respondents and nonrespondents Short/very short questionnaire mail/telephone

Q2010, Helsinki, May 2010 Adjusting for nonresponse bias in ESS Post-stratification Aggregated info on sex, age, education Cooperative and reluctant respondents Reasons for refusal, Number of contacts, Interviewer judgment Observational data Interviewer observation for sample and contacts Doorstep questions refusals Very short questionnaire f2f Follow-up survey respondents and nonrespondents Short/very short questionnaire mail/telephone

Q2010, Helsinki, May 2010 Bias: top and bottom of 45 items Av. ASbiasItem Satisfaction with national government Most of the time people try to be helpful/are mostly looking out for themselves? Important that people are treated equally and have equal opportunities Left-right scale Important to be loyal to friends, devote to people close Borrow money to make ends meet (difficult/easy) Newspaper reading, politics/current affairs average weekday Immigrants make country a worse/better place to live Immigration bad/good for country’s economy Newspaper reading, total time average weekday Household size Interest in politics

Q2010, Helsinki, May 2010 Poststratification in ESS PositiveNegative Uniform strategy for all countriesQuality and availability variables differ Adjust all nonresponse (refusal + noncontact) Also sampling errors At least an idea of direction of nonresponse bias Only minor differences in regression parameters found Correlation PS variables and nonresponse?

Q2010, Helsinki, May 2010 Adjusting for nonresponse bias in ESS Post-stratification Aggregated info on sex, age, education Cooperative and reluctant respondents Reasons for refusal, Number of contacts, Interviewer judgment Observational data Interviewer observation for sample and contacts Doorstep questions refusals Very short questionnaire f2f Follow-up survey respondents and nonrespondents Short/very short questionnaire mail/telephone

Q2010, Helsinki, May 2010 Definition reluctant respondents/converted refusals Initial cooperation rate? Refusal conversion strategy None Easy All Large differences in strategies and results across countries If there are few: high initial, good job or poor start?

Q2010, Helsinki, May 2010 Converted refusals in ESS 2

Q2010, Helsinki, May 2010 Evaluation refusal conversion approach ESS Positive Enhances response rates Conveys message that every respondent is important Negative Privacy regulations Quality call records Different initial response rates Different efforts across countries Different success rate refusal conversion Aimed at different groups (all/easy) Few countries only

Q2010, Helsinki, May 2010 Adjusting for nonresponse bias in ESS Post-stratification Aggregated info on sex, age, education Cooperative and reluctant respondents Reasons for refusal, Number of contacts, Interviewer judgment Observational data Interviewer observation for sample and contacts Doorstep questions refusals Very short questionnaire f2f Follow-up survey respondents and nonrespondents Short/very short questionnaire mail/telephone

Q2010, Helsinki, May 2010 Type and quality of data Characteristics neighbourhood and dwelling Good data in about half of the ESS countries Better in later rounds Estimation of age and sex Good data in only a few of the countries Is refuser target person? Conflicting instructions contact form

Q2010, Helsinki, May 2010 Content Variables measured Type of dwelling (apartment/detached/...) Physical state of building/dwellings in neighbourhood Litter or rubbish in immediate area Vandalism, graffiti or damage to property Poorer condition correlates with lower education

Q2010, Helsinki, May 2010 Some results Living in apartment increases likelihood to refuse and not to be contacted (exception AT, why???) More likely to refuse and noncontact when physical condition is worse (exception AT, why???) Litter and vandalism in environment: weak effects (exception AT) Interaction between dwelling type (apartment) and physical condition in ES and SK (more likely not to participate and noncontact if apartment in bad state)

Q2010, Helsinki, May 2010 Evaluation interviewer observation approach ESS Positive All sample units used (cooperative, refusals, non contacted) Moderate correlation moderately with target variables as social status (education) Negative Missing data Poor quality of coding Neighbourhood scores comparable across countries? Neighbourhood data proxy for household/ individual data

Q2010, Helsinki, May 2010 Adjusting for nonresponse bias in ESS Post-stratification Aggregated info on sex, age, education Cooperative and reluctant respondents Reasons for refusal, Number of contacts, Interviewer judgment Observational data Interviewer observation for sample and contacts Doorstep questions refusals Very short questionnaire f2f Follow-up survey respondents and nonrespondents Short/very short questionnaire mail/telephone

Q2010, Helsinki, May 2010 Data and method NRS sample design in 4 countries ESS 3 Target groupTimingModeIncentivesLengthResponse rate R/NR % Sample size BEInitial refusalsParallelPAPI doorstep NoVery short CHRespondentsAfterMail/web/ CATI 10 Swiss francs Very short & short Nonrespondents NORespondentsAfterMail/web/ CATI Noshort Nonrespondents PLRespondentsAfterMailNotepadVery short & short Nonrespondents

Q2010, Helsinki, May 2010 Basic questionnaire Very short: 7 questions Work situation Highest level of education # of members in household Frequency of social activities Feeling (un)safe Interest in politics Attitude towards surveys Short: 9 additional questions Sex Year of birth TV watching Voluntary work Social trust Satisfied with democracy Trust in politics Immigration good/bad for country (Reasons for refusal)

Q2010, Helsinki, May 2010 Many different comparisons to be made Doorstep Cooperative main Reluctant main Cooperative doorstep (calibrated) Follow-up Main/follow-upResponseNonresponse CooperativeESSNRSESS ReluctantESSNRSESS Refusal NRS Noncontact NRS

Q2010, Helsinki, May 2010 DQS Belgium: different cooperative? ReluctantDoorstep Age Education less Labour market Household composition Safe neighbourhood Social participation less Political interest less

Q2010, Helsinki, May 2010 Results Belgium doorstep approach Criterium 1 UnweightedWeighted Chi² Age Educational level Social participation Political interest Criterium 2 (coop/refusal) Weighted cooperative respondents do not differ from unweighted cooperative respondents

Q2010, Helsinki, May 2010 Do survey responses differ between nonrespondent and cooperative respondents in follow-up and main survey in Norway? COOP follow-upCOOP main Age Sex Education level Work status Household composition Neighborhood security Social participation Political interest

Q2010, Helsinki, May 2010 Do survey responses differ between nonrespondents and cooperative respondents in follow-up and main survey in Norway? COOP follow-upCOOP main Satisfaction democracy Trust in politicians Attitude immigrants TV watching time per day Voluntary and charity work Social trust

Q2010, Helsinki, May 2010 Are the distributions of key variables independent by the types of respondents (nonrespondents and cooperative main) in Norway? (criteria 2) UnweightedWeighted Chi² Education category Work status Political interest Social participation

Q2010, Helsinki, May 2010 Are the distributions of key variables independent by the types of respondents (nonrespondents and cooperative main) in Norway? (criteria 2) UnweightedWeighted T-value How satisfied democracy Immigrants make worse/better TV watching time per day Involved in voluntary work

Q2010, Helsinki, May 2010 Is it possible to correct nonresponse bias on the basis of nonresponse survey (NRS)? On the basis of key questions on socio- demographic, behavioral & attitudinal variables, non- response bias can be measured which information to take is crucial Response propensities can take different form; Based on this approach, nonresponse bias can be corrected for (but not for all variables)

Q2010, Helsinki, May 2010 Evaluation doorstep core question approach ESS Positive Can correct for refusal Relatively inexpensive Combine with paradata/neighbourhood data Negative Context effect Refusals only Remaining refusals doorstep questionnaire Few questions only

Q2010, Helsinki, May 2010 Evaluation follow-up core question approach ESS Positive Can correct for nonresponse Include all nonrespondents Combine with paradata/neighbourhood data Negative What groups to compare (initially reluctant?) High nonresponse in follow-up study Different initial response rates Few questions only

Q2010, Helsinki, May 2010 What to do in 30+ country study? Initial situation differs Possibilities differ Use national best auxiliary information Study bias in many different ways Optimal national adjustments threaten comparability? Provide nonresponse weights to users?

Q2010, Helsinki, May 2010 More on this topic