1. Nonresponse adjustment in the European Social Survey (ESS) Ineke Stoop (SCP-Netherlands) Jaak Billiet (Uni Leuven-Belgium) Achim Koch (Gesis-Germany)

2. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 European Social Survey Content To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions Methodology To advance and consolidate improved methods of cross- national survey measurement in Europe and beyond Input harmonisation

3. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Basic facts Biennial survey 30+ countries 2.000 interviews/country Fieldwork organisation/ country Probability sampling Design weights No substitution Face-to-face Fieldwork monitoring Guidelines interviewer training Target noncontact rate: 3% Minimum number of calls Timing of calls (evening, weekend) Target response rate: 70% Guidelines response enhancement Incentives/Brochure Refusal conversion Contact form Interviewer observations Call records Standard response rate calculation

4. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Factors causing variance in nonresponse bias Survey climate Population characteristics Difficulty to obtain response House effects Fieldwork staff Fieldwork strategy Efforts to obtain high response rates Sampling frames Realised response rates Nonresponse composition noncontact/refusal/not able (language) Reasons for refusal Presence of auxiliary data to adjust for nonresponse bias

5. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Response rates ESS

6. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Assessing and correcting for bias in a cross- national study Ideal: same way everywhere Practice: availability, accessibility auxiliary data differs Sampling frames differs Usually no access to population register Quality population statistics Quality paradata Ease of response (wave data) Only approaches that COULD be used in every country?

7. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Adjusting for nonresponse bias in ESS Post-stratification Aggregated info on sex, age, education Cooperative and reluctant respondents Reasons for refusal, Number of contacts, Interviewer judgment Observational data Interviewer observation for sample and contacts Doorstep questions refusals Very short questionnaire f2f Follow-up survey respondents and nonrespondents Short/very short questionnaire mail/telephone

8. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Adjusting for nonresponse bias in ESS Post-stratification Aggregated info on sex, age, education Cooperative and reluctant respondents Reasons for refusal, Number of contacts, Interviewer judgment Observational data Interviewer observation for sample and contacts Doorstep questions refusals Very short questionnaire f2f Follow-up survey respondents and nonrespondents Short/very short questionnaire mail/telephone

9. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Bias: top and bottom of 45 items Av. ASbiasItem 0.555 Satisfaction with national government 0.666 Most of the time people try to be helpful/are mostly looking out for themselves? 0.676 Important that people are treated equally and have equal opportunities 0.737 Left-right scale 0.793 Important to be loyal to friends, devote to people close 1.857 Borrow money to make ends meet (difficult/easy) 2.077 Newspaper reading, politics/current affairs average weekday 2.082 Immigrants make country a worse/better place to live 2.189 Immigration bad/good for country’s economy 2.477 Newspaper reading, total time average weekday 3.504 Household size 3.646 Interest in politics

10. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Poststratification in ESS PositiveNegative Uniform strategy for all countriesQuality and availability variables differ Adjust all nonresponse (refusal + noncontact) Also sampling errors At least an idea of direction of nonresponse bias Only minor differences in regression parameters found Correlation PS variables and nonresponse?

11. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Adjusting for nonresponse bias in ESS Post-stratification Aggregated info on sex, age, education Cooperative and reluctant respondents Reasons for refusal, Number of contacts, Interviewer judgment Observational data Interviewer observation for sample and contacts Doorstep questions refusals Very short questionnaire f2f Follow-up survey respondents and nonrespondents Short/very short questionnaire mail/telephone

12. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Definition reluctant respondents/converted refusals Initial cooperation rate? Refusal conversion strategy None Easy All Large differences in strategies and results across countries If there are few: high initial, good job or poor start?

13. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Converted refusals in ESS 2

14. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Evaluation refusal conversion approach ESS Positive Enhances response rates Conveys message that every respondent is important Negative Privacy regulations Quality call records Different initial response rates Different efforts across countries Different success rate refusal conversion Aimed at different groups (all/easy) Few countries only

15. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Adjusting for nonresponse bias in ESS Post-stratification Aggregated info on sex, age, education Cooperative and reluctant respondents Reasons for refusal, Number of contacts, Interviewer judgment Observational data Interviewer observation for sample and contacts Doorstep questions refusals Very short questionnaire f2f Follow-up survey respondents and nonrespondents Short/very short questionnaire mail/telephone

16. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Type and quality of data Characteristics neighbourhood and dwelling Good data in about half of the ESS countries Better in later rounds Estimation of age and sex Good data in only a few of the countries Is refuser target person? Conflicting instructions contact form

17. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Content Variables measured Type of dwelling (apartment/detached/...) Physical state of building/dwellings in neighbourhood Litter or rubbish in immediate area Vandalism, graffiti or damage to property Poorer condition correlates with lower education

18. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Some results Living in apartment increases likelihood to refuse and not to be contacted (exception AT, why???) More likely to refuse and noncontact when physical condition is worse (exception AT, why???) Litter and vandalism in environment: weak effects (exception AT) Interaction between dwelling type (apartment) and physical condition in ES and SK (more likely not to participate and noncontact if apartment in bad state)

19. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Evaluation interviewer observation approach ESS Positive All sample units used (cooperative, refusals, non contacted) Moderate correlation moderately with target variables as social status (education) Negative Missing data Poor quality of coding Neighbourhood scores comparable across countries? Neighbourhood data proxy for household/ individual data

20. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Adjusting for nonresponse bias in ESS Post-stratification Aggregated info on sex, age, education Cooperative and reluctant respondents Reasons for refusal, Number of contacts, Interviewer judgment Observational data Interviewer observation for sample and contacts Doorstep questions refusals Very short questionnaire f2f Follow-up survey respondents and nonrespondents Short/very short questionnaire mail/telephone

21. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Data and method NRS sample design in 4 countries ESS 3 Target groupTimingModeIncentivesLengthResponse rate R/NR % Sample size BEInitial refusalsParallelPAPI doorstep NoVery short 45 303 CHRespondentsAfterMail/web/ CATI 10 Swiss francs Very short & short 84 1023 52 Nonrespondents NORespondentsAfterMail/web/ CATI Noshort61 487 30 Nonrespondents PLRespondentsAfterMailNotepadVery short & short 59 1208 23 Nonrespondents

22. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Basic questionnaire Very short: 7 questions Work situation Highest level of education # of members in household Frequency of social activities Feeling (un)safe Interest in politics Attitude towards surveys Short: 9 additional questions Sex Year of birth TV watching Voluntary work Social trust Satisfied with democracy Trust in politics Immigration good/bad for country (Reasons for refusal)

23. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Many different comparisons to be made Doorstep Cooperative main Reluctant main Cooperative doorstep (calibrated) Follow-up Main/follow-upResponseNonresponse CooperativeESSNRSESS ReluctantESSNRSESS Refusal NRS Noncontact NRS

24. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 DQS Belgium: different cooperative? ReluctantDoorstep Age Education less Labour market Household composition Safe neighbourhood Social participation less Political interest less

25. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Results Belgium doorstep approach Criterium 1 UnweightedWeighted Chi² Age Educational level Social participation Political interest Criterium 2 (coop/refusal) Weighted cooperative respondents do not differ from unweighted cooperative respondents

26. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Do survey responses differ between nonrespondent and cooperative respondents in follow-up and main survey in Norway? COOP follow-upCOOP main Age Sex Education level Work status Household composition Neighborhood security Social participation Political interest

27. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Do survey responses differ between nonrespondents and cooperative respondents in follow-up and main survey in Norway? COOP follow-upCOOP main Satisfaction democracy Trust in politicians Attitude immigrants TV watching time per day Voluntary and charity work Social trust

28. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Are the distributions of key variables independent by the types of respondents (nonrespondents and cooperative main) in Norway? (criteria 2) UnweightedWeighted Chi² Education category Work status Political interest Social participation

29. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Are the distributions of key variables independent by the types of respondents (nonrespondents and cooperative main) in Norway? (criteria 2) UnweightedWeighted T-value How satisfied democracy Immigrants make worse/better TV watching time per day Involved in voluntary work

30. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Is it possible to correct nonresponse bias on the basis of nonresponse survey (NRS)? On the basis of key questions on socio- demographic, behavioral & attitudinal variables, non- response bias can be measured which information to take is crucial Response propensities can take different form; Based on this approach, nonresponse bias can be corrected for (but not for all variables)

31. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Evaluation doorstep core question approach ESS Positive Can correct for refusal Relatively inexpensive Combine with paradata/neighbourhood data Negative Context effect Refusals only Remaining refusals doorstep questionnaire Few questions only

32. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 Evaluation follow-up core question approach ESS Positive Can correct for nonresponse Include all nonrespondents Combine with paradata/neighbourhood data Negative What groups to compare (initially reluctant?) High nonresponse in follow-up study Different initial response rates Few questions only

33. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 What to do in 30+ country study? Initial situation differs Possibilities differ Use national best auxiliary information Study bias in many different ways Optimal national adjustments threaten comparability? Provide nonresponse weights to users?

34. www.europeansocialsurvey.org Q2010, Helsinki, May 2010 More on this topic