1. NTTS conference, February 18 – 20 2009 New Developments in Nonresponse Adjustment Methods Fannie Cobben Statistics Netherlands Department of Methodology and Quality

2. NTTS conference, February 18 – 20 2009 In this presentation… Response selection model Use of response propensities Application to POLS 2002 Discussion

3. NTTS conference, February 18 – 20 2009 The response selection model Consists of two equations: Response equation (dichotomous) Survey item equation (continuous) The error terms are allowed to be correlated The outcome is adjusted for a possible selection bias

4. NTTS conference, February 18 – 20 2009 Extensions to response selection model Multiple selection equations, i.e. different response types Contact equation Participation equation Survey item equation Contact and participation are dependent and can both introduce a bias for the survey item If desirable, equations for other response types Categorical survey items

5. NTTS conference, February 18 – 20 2009 Response selection model – contact and participation ςi*ςi* γi*γi* P i = 1 C i = 1 P i = 0 C i = 0 Sample Non-contactContact RefusalParticipation Y i * missing Y i * = Y i Y i * missing

6. NTTS conference, February 18 – 20 2009 Advantages Model relationship with both R and Y Efficient use of auxiliary variables; paradata Closely follow fieldwork process Disadvantages Model based; dependent on distributional assumptions Issues of identification; exclusion restriction

7. NTTS conference, February 18 – 20 2009 Response propensities (1) Assume: Sample is selected from a sampling frame by some random selection procedure Two groups: R=1 response R=0 non-response X auxiliary information, available for all elements. For instance from the sampling frame. ρ(X) = P(R=1| X) is the propensity score, for instance determined by a logistic regression, i.e.

8. NTTS conference, February 18 – 20 2009 Response propensities (2) We can use the response propensities to adjust for nonresponse bias: Directly Response propensity weighting Response propensity stratification Indirectly In combination with linear weighting

9. NTTS conference, February 18 – 20 2009 Direct use of response propensities Response propensity weighting, Särndal (1981) Response propensity stratification

10. NTTS conference, February 18 – 20 2009 Indirect use of response propensities GREG – estimator with adopted inclusion probabilities

11. NTTS conference, February 18 – 20 2009 POLS 2002 Integrated Survey on Household Living Conditions (in Dutch: Permanent Onderzoek LeefSituatie) Monthly 3.000 persons are selected Questions on living conditions, safety, health Basic module for persons > 12 years Datafile aggregated over 2002: n = 35.594 and n r = 20.168 (57%) Survey variables: Employment, Education and Religion Numerous auxiliary variables, such as age, region, house value, social insurance, ethnicity, etc.

12. NTTS conference, February 18 – 20 2009 Analysis POLS 2002 Two models: Weighting model (relation Y and R; Schouten, 2004) Age 15 + Houseval 14 + % Non-natives 8 + Ethnicity 7 + Region 15 + Type hh 4 + Telephone 2 Response model (relation R; psuedo R 2 = 2,2%) Age 15 + Houseval 14 + Urbanicity 5 + Mar_staat 4 + Ethnicity 7 + Region 15 + Type hh 4 + Telephone 2 (1) (2) Aim: Compare different response propensity methods Compare regular GREG-estimator and other methods

13. NTTS conference, February 18 – 20 2009 Propensity stratification Propensity weighting Survey itemResponse average GREG (1) (2)(1)(2)Propensity GREG Employed >= 12 hours52,4 (0,35) 53,7 (0,18) 53,5 (0,38) 53,7 (0,38) 53,7 (0,34) 53,8 (0,34) 53,8 unemployed6,7 (0,18) 6,4 (0,11) 6,4 (0,17) 6,4 (0,17) 6,4 (0,17) 6,4 (0,18) 6,4 < 12 hours40,9 (0,35) 39,9 (0,17) 40,0 (0,35) 39,9 (0,37) 39,9 (0,34) 39,8 (0,33) 39,9 Results - Employment No significant differences between method; clear difference with response average GREG and propensity weighting with same model: same results Propensity weighting and propensity GREG highest estimate employed labour force

14. NTTS conference, February 18 – 20 2009 Results - Education Propensity stratification Propensity weighting Survey itemResponse average GREG (1) (2)(1)(2)Propensity GREG Education Primary 7,2 (0,18) 6,3 (0,09) 6,3 (0,20) 6,3 (0,18) 6,3 (0,18) 6,3 (0,20) 6,3 MAVO 12,1 (0,23) 12,0 (0,15) 12,0 (0,23) 12,0 (0,22) 12,0 (0,25) 12,0 (0,24) 12,0 HAVO 19,7 (0,28) 19,9 (0,18) 19,9 (0,27) 19,9 (0,26) 19,9 (0,27) 19,9 (0,29) 19,9 VWO 7,1 (0,18) 7,2 (0,12) 7,2 (0,17) 7,2 (0,19) 7,2 (0,18) 7,2 (0,21) 7,2 MBO 30,8 (0,32) 31,1 (0,21) 31,0 (0,31) 31,1 (0,32) 31,1 (0,32) 31,0 (0,32) 31,0 HBO 16,7 (0,26) 16,9 (0,17) 16,8 (0,26) 16,9 (0,25) 16,9 (0,25) 16,9 (0,26) 16,9 WO 6,3 (0,17) 6,4 (0,11) 6,4 (0,16) 6,5 (0,18) 6,4 (0,19) 6,5 (0,18) 6,5 else 0,2 (0,03) 0,2 (0,02) 0,2 (0,03) 0,2 (0,03) 0,2 (0,03) 0,2 (0,03) 0,2

15. NTTS conference, February 18 – 20 2009 Propensity stratificiation Propensity weighting Survey itemResponse average GREG (1) (2)(1)(2)Propensity GREG Religion none37,7 (0,34) 38,5 (0,21) 38,3 (0,35) 38,4 (0,34) 38,4 (0,37) 38,5 (0,37) 38,5 R.K.33,5 (0,33) 32,4 (0,19) 32,7 (0,34) 32,6 (0,35) 32,5 (0,35) 32,5 (0,35) 32,5 Protestant21,0 (0,29) 20,4 (0,17) 20,6 (0,28) 20,5 (0,30) 20,5 (0,35) 20,4 (0,29) 20,4 Islam2,5 (0,11) 3,3 (0,06) 3,1 (0,18) 3,1 (0,11) 3,2 (0,12) 3,2 (0,12) 3,2 else5,2 (0,16) 5,4 (0,01) 5,4 (0,16) 5,3 (0,16) 5,4 (0,15) 5,4 (0,15) 5,4 Results - Religion More differences between methods Propensity weighting and propensity GREG highest estimate no religion

16. NTTS conference, February 18 – 20 2009 Conclusions Remarks: Almost the same variables in weighting model and response model Low pseudo R 2 response model Conclusions: 1.Small difference between methods 2.No reduction of variance direct response propensity methods; reduction of variance GREG-estimator 3.Propensity weighting and GREG-estimator with same model gives same results for employed and education

17. NTTS conference, February 18 – 20 2009 Thank you for your attention!

18. NTTS conference, February 18 – 20 2009 Total Survey Error Total error Sampling error Estimation error Selection error Non-sampling error Observation error Over-coverage error Measurement error Processing error Non-observation error Under-coverage error Nonresponse error

19. NTTS conference, February 18 – 20 2009 Probability based surveys (2) Under-coverage in telephone surveys Two groups: with telephone (C=1) without telephone (C=0) Non-response Two groups: Respondents (C=1) Non-respondents (C=0)

20. NTTS conference, February 18 – 20 2009 Discussion Which variables should be inserted in the different equations, and how should these variables be selected? How to construct the hierarchical structure of the model. For instance, which response types should be distinguished? Is it profitable to construct a model for every survey item? If so, how should we deal with this in practice? The response selection assumes a correlation structure between response types and survey items. This correlation has a model-based interpretation, but how should it be interpreted in practice?

21. NTTS conference, February 18 – 20 2009 Example relationship R and Y in the LFS Relation between the number of contact attempts and estimated size of the labour force 1 attempt 2 attempts 4 attempts all attempts Quarter size of the labour force