NTTS conference, February 18 – New Developments in Nonresponse Adjustment Methods Fannie Cobben Statistics Netherlands Department of Methodology and Quality

NTTS conference, February 18 – In this presentation… Response selection model Use of response propensities Application to POLS 2002 Discussion

NTTS conference, February 18 – The response selection model Consists of two equations: Response equation (dichotomous) Survey item equation (continuous) The error terms are allowed to be correlated The outcome is adjusted for a possible selection bias

NTTS conference, February 18 – Extensions to response selection model Multiple selection equations, i.e. different response types Contact equation Participation equation Survey item equation Contact and participation are dependent and can both introduce a bias for the survey item If desirable, equations for other response types Categorical survey items

NTTS conference, February 18 – Response selection model – contact and participation ςi*ςi* γi*γi* P i = 1 C i = 1 P i = 0 C i = 0 Sample Non-contactContact RefusalParticipation Y i * missing Y i * = Y i Y i * missing

NTTS conference, February 18 – Advantages Model relationship with both R and Y Efficient use of auxiliary variables; paradata Closely follow fieldwork process Disadvantages Model based; dependent on distributional assumptions Issues of identification; exclusion restriction

NTTS conference, February 18 – Response propensities (1) Assume: Sample is selected from a sampling frame by some random selection procedure Two groups: R=1 response R=0 non-response X auxiliary information, available for all elements. For instance from the sampling frame. ρ(X) = P(R=1| X) is the propensity score, for instance determined by a logistic regression, i.e.

NTTS conference, February 18 – Response propensities (2) We can use the response propensities to adjust for nonresponse bias: Directly Response propensity weighting Response propensity stratification Indirectly In combination with linear weighting

NTTS conference, February 18 – Direct use of response propensities Response propensity weighting, Särndal (1981) Response propensity stratification

NTTS conference, February 18 – Indirect use of response propensities GREG – estimator with adopted inclusion probabilities

NTTS conference, February 18 – POLS 2002 Integrated Survey on Household Living Conditions (in Dutch: Permanent Onderzoek LeefSituatie) Monthly persons are selected Questions on living conditions, safety, health Basic module for persons > 12 years Datafile aggregated over 2002: n = and n r = (57%) Survey variables: Employment, Education and Religion Numerous auxiliary variables, such as age, region, house value, social insurance, ethnicity, etc.

NTTS conference, February 18 – Analysis POLS 2002 Two models: Weighting model (relation Y and R; Schouten, 2004) Age 15 + Houseval 14 + % Non-natives 8 + Ethnicity 7 + Region 15 + Type hh 4 + Telephone 2 Response model (relation R; psuedo R 2 = 2,2%) Age 15 + Houseval 14 + Urbanicity 5 + Mar_staat 4 + Ethnicity 7 + Region 15 + Type hh 4 + Telephone 2 (1) (2) Aim: Compare different response propensity methods Compare regular GREG-estimator and other methods

NTTS conference, February 18 – Propensity stratification Propensity weighting Survey itemResponse average GREG (1) (2)(1)(2)Propensity GREG Employed >= 12 hours52,4 (0,35) 53,7 (0,18) 53,5 (0,38) 53,7 (0,38) 53,7 (0,34) 53,8 (0,34) 53,8 unemployed6,7 (0,18) 6,4 (0,11) 6,4 (0,17) 6,4 (0,17) 6,4 (0,17) 6,4 (0,18) 6,4 < 12 hours40,9 (0,35) 39,9 (0,17) 40,0 (0,35) 39,9 (0,37) 39,9 (0,34) 39,8 (0,33) 39,9 Results - Employment No significant differences between method; clear difference with response average GREG and propensity weighting with same model: same results Propensity weighting and propensity GREG highest estimate employed labour force

NTTS conference, February 18 – Results - Education Propensity stratification Propensity weighting Survey itemResponse average GREG (1) (2)(1)(2)Propensity GREG Education Primary 7,2 (0,18) 6,3 (0,09) 6,3 (0,20) 6,3 (0,18) 6,3 (0,18) 6,3 (0,20) 6,3 MAVO 12,1 (0,23) 12,0 (0,15) 12,0 (0,23) 12,0 (0,22) 12,0 (0,25) 12,0 (0,24) 12,0 HAVO 19,7 (0,28) 19,9 (0,18) 19,9 (0,27) 19,9 (0,26) 19,9 (0,27) 19,9 (0,29) 19,9 VWO 7,1 (0,18) 7,2 (0,12) 7,2 (0,17) 7,2 (0,19) 7,2 (0,18) 7,2 (0,21) 7,2 MBO 30,8 (0,32) 31,1 (0,21) 31,0 (0,31) 31,1 (0,32) 31,1 (0,32) 31,0 (0,32) 31,0 HBO 16,7 (0,26) 16,9 (0,17) 16,8 (0,26) 16,9 (0,25) 16,9 (0,25) 16,9 (0,26) 16,9 WO 6,3 (0,17) 6,4 (0,11) 6,4 (0,16) 6,5 (0,18) 6,4 (0,19) 6,5 (0,18) 6,5 else 0,2 (0,03) 0,2 (0,02) 0,2 (0,03) 0,2 (0,03) 0,2 (0,03) 0,2 (0,03) 0,2

NTTS conference, February 18 – Propensity stratificiation Propensity weighting Survey itemResponse average GREG (1) (2)(1)(2)Propensity GREG Religion none37,7 (0,34) 38,5 (0,21) 38,3 (0,35) 38,4 (0,34) 38,4 (0,37) 38,5 (0,37) 38,5 R.K.33,5 (0,33) 32,4 (0,19) 32,7 (0,34) 32,6 (0,35) 32,5 (0,35) 32,5 (0,35) 32,5 Protestant21,0 (0,29) 20,4 (0,17) 20,6 (0,28) 20,5 (0,30) 20,5 (0,35) 20,4 (0,29) 20,4 Islam2,5 (0,11) 3,3 (0,06) 3,1 (0,18) 3,1 (0,11) 3,2 (0,12) 3,2 (0,12) 3,2 else5,2 (0,16) 5,4 (0,01) 5,4 (0,16) 5,3 (0,16) 5,4 (0,15) 5,4 (0,15) 5,4 Results - Religion More differences between methods Propensity weighting and propensity GREG highest estimate no religion

NTTS conference, February 18 – Conclusions Remarks: Almost the same variables in weighting model and response model Low pseudo R 2 response model Conclusions: 1.Small difference between methods 2.No reduction of variance direct response propensity methods; reduction of variance GREG-estimator 3.Propensity weighting and GREG-estimator with same model gives same results for employed and education

NTTS conference, February 18 – Thank you for your attention!

NTTS conference, February 18 – Total Survey Error Total error Sampling error Estimation error Selection error Non-sampling error Observation error Over-coverage error Measurement error Processing error Non-observation error Under-coverage error Nonresponse error

NTTS conference, February 18 – Probability based surveys (2) Under-coverage in telephone surveys Two groups: with telephone (C=1) without telephone (C=0) Non-response Two groups: Respondents (C=1) Non-respondents (C=0)

NTTS conference, February 18 – Discussion Which variables should be inserted in the different equations, and how should these variables be selected? How to construct the hierarchical structure of the model. For instance, which response types should be distinguished? Is it profitable to construct a model for every survey item? If so, how should we deal with this in practice? The response selection assumes a correlation structure between response types and survey items. This correlation has a model-based interpretation, but how should it be interpreted in practice?

NTTS conference, February 18 – Example relationship R and Y in the LFS Relation between the number of contact attempts and estimated size of the labour force 1 attempt 2 attempts 4 attempts all attempts Quarter size of the labour force