11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 1 Evaluating the Total Non-Response Errors in the European-Union Survey on Income and Living Conditions (EU-SILC): A Territorial Quality Profile Claudio QUINTANO Rosalia CASTELLANO Gennaro PUNZO European Conference on Quality in Official Statistics 2008 Rome, Italy – July 8-11, 2008

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 2 AIM OF THE WORK Evaluating the ACCURACY of the Italian Section of EU-SILC data with focus on the NON-SAMPLING ERRORS deriving from the several components of TOTAL NON-RESPONSE STEPS  A SET OF AD HOC BASIC QUALITY INDICATORS (HIERARCHICAL FRAMEWORK)  CLASSES OF SYNTHETIC QUALITY INDICATORS  BROKEN DOWN BY DIFFERENT TERRITORIAL LEVELS  ONE-WAY RANDOM EFFECTS ANOVA MODEL (Singer, 1998)  RANKING ANALYSIS TERRITORIAL QUALITY PROFILE

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 3 EU-SILC: BACKGROUND AND MAIN FEATURES OF THE ITALIAN SEGMENT PRIMARY SAMPLING UNITS (PSU) SECONDARY SAMPLING UNITS(SSU)HOUSEHOLDS A two-stages sampling design… …drawn from municipality-registers by a systematic sampling MUNICIPALITIES …STRATIFIED by demographic size, inside each NUTS2 region 288 STRATA Self-Representing Non Self-Representing First order Second order

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 4 WAVESLONGITUDINAL SUB-SAMPLES INTEGRATED DESIGN BASED ON FOUR YEARLY ROTATIONAL PANELS A (4) B (3) C (2) D1D1 B (4) D2D2 C (3) E1E1 C (4) D3D3 E2E2 F1F1 D4D4 E3E3 F2F2 G1G1 E4E4 F3F3 G2G2 H1H1 F4F4 G3G3 H2H2 I1I1 CROSS-SECTIONAL WHOLE PANEL

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 5 Source: EU-SILC Quality Reports (2004 and 2005) WavesABCDETotal – – EU-SILC SAMPLING DESIGN... Table 1 – PSU stratification... Table 2 – Longitudinal replications in terms of SSU... STRATAPSUSelf-Representing Non Self-Representing of First order Non Self-Representing of Second order Total Source: Author’s elaborations on Istat data

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 6 SAMPLINGUNITS A THEORETICAL FRAMEWORK IN-SCOPE RESPONDENTSNONRESPONDENTS NONACHIEVEMENTNONCONTACTEDREFUSED UNABLE TO RESPONSE INCORRECTADDRESS NOT LOCATED ADDRESS UNABLE TO ACCESS OUT-OF-SCOPE TEMPORARILYNOT-AT-HOME

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 7 EU-SILC BASIC QUALITY INDICATORS IN-SCOPE RATE NON-ACHIEVEMENT RATE REFUSAL RATE UNABLE-TO- RESPONSE RATE INCORRECT ADDRESS RATE NOT-LOCATED ADDRESS RATE UNABLE-TO- ACCESS RATE OUT-OF-SCOPE RATE TEMPORARILY NOT-AT-HOME RATE

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 8 A THEORETICAL FRAMEWORK IN-SCOPE RATE NON-ACHIEVEMENT RATE REFUSAL RATE UNABLE-TO- RESPONSE RATE INCORRECT ADDRESS RATE NOT-LOCATED ADDRESS RATE UNABLE-TO- ACCESS RATE OUT-OF-SCOPE RATE TEMPORARILY NOT-AT-HOME RATE

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 9 THE PROCESS OF CONTACTING THE EU-SILC SAMPLE HOUSEHOLDS... cross-sectional sample In-Scope Rate Out-of-Scope Rate Source: Author’s elaborations on Istat data Table 3 – Eligibility Rates Table 4 – Non-Contact Rates Not-Located Address Rate Unable-to-Access Rate Incorrect Address Rate Source: Author’s elaborations on Istat data 0.53% 0.32% 0.30% DECEASED INSTITUTIONALIZED TRANSFERRED Table 5 – Frame Error Rates Frame Error Rate Source: Author’s elaborations on Istat data

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research AND THEIR ACTUAL INVOLVEMENT EU-SILC cross-sectional sample Refusal Rate Unable-to-Response Rate Non-Achievement Rate Temporarily Not-at-Home Rate Temporarily Not-at-Home Rate Table 6 – Non-Participation Rates Source: Author’s elaborations on Istat data NON-ACHIEVEMENT RATE also includes a residual share of non-participant households, though successfully contacted, whose reasons are not specified

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 11 COOPERATIONRATE NON COOPERATION RATE NON ACHIEVEMENT RATE REFUSAL RATE UNABLE TO RESPONSE RATE... A SYNTHESIS OF THE BASIC QUALITY INDICATORS: A HIERACHICAL APPROACH COOPERATION RATE

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 12 NON CONTACT RATECONTACTRATE INCORRECT ADDRESS RATE NOT LOCATED ADDRESS RATE UNABLE TO ACCESS RATE... A SYNTHESIS OF THE BASIC QUALITY INDICATORS: A HIERACHICAL APPROACH CONTACT RATE

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 13 RESPONSE RATE CONTACT RATE COOPERATION RATE COMPLETION RATE... A SYNTHESIS OF THE BASIC QUALITY INDICATORS: A HIERACHICAL APPROACH x

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 14 IN-SCOPE RESPONDENTSNONRESPONDENTS NONCONTACTED CONTACTED INCORRECTADDRESS NOT LOCATED ADDRESS UNABLE TO ACCESS OUT-OF-SCOPE COOPERATING NONCOOPERATING NON ACHIEVEMENT REFUSED UNABLE TO RESPONSE SAMPLINGUNITS

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 15 EVALUATING THE ACTUAL SURVEY PARTICIPATION OVER TIME EU-SILC cross-sectional and longitudinal samples... wave 2004 ABCDTotal Cooperation Rate Contact Rate Response Rate wave 2005 BCDETotal Cooperation Rate Contact Rate Response Rate Table 7 – Synthetic Participation Rates Source: Author’s elaborations on Istat data COMPLETION RATE 75.86% 82.25% wave 2004 wave

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 16 DOES TOTAL NON-PARTICIPATION DIFFER ACROSS NATIONAL TERRITORY? Fig. 3 – Non–Response RatesFig. 4 – Non–Completion Rates Fig. 2 – Non–Contact RatesFig. 1 – Non–Cooperation Rates wave 2004 wave 2005

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 17 DOES TOTAL NON-PARTICIPATION DIFFER ACROSS NATIONAL TERRITORY? Fig. 5 – Frame Error RatesFig. 6 – Refusal Rates wave 2004 wave 2005 In order to investigate in-depth the territorial perspective, its role and significance, as well as the effects on the total non-response process... ONE-WAY RANDOM EFFECTS ANOVA MODEL (NULL MODEL)

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 18 with i = 1, 2,...,m and j = 1, 2,..., 21 municipalities ONE-WAY RANDOM EFFECTS ANOVA MODEL (NULL MODEL) NUTS2 regions fixedeffect randomeffects of belonging to the j th NUTS2 region difference between the i th municipality and the mean within the j th region measuring the effect representing the intraclasscorrelation degree of homogeneity within NUTS2 regions

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 19 Outcome Rate (2004) Non-ContactNon-CooperationNon-Response Fixed Effect: Intercept Random Effect: Region variance Residual variance (.00340) (.00009) (.00018) (.00812) (.00039)* (.00039)* (.00240) (.00880) (.00053) (.00247) Outcome Rate (2005) Non-ContactNon-CooperationNon-Response Fixed Effect: Intercept Random Effect: Region variance Residual variance (.00322) (.00008) (.00013) (.01308) (.00107) (.00265) (.01366) (.00122) (.00270) Table 8 – One-way random effects ANOVA model: main results Source: Author’s elaborations on Istat data REGIONEFFECT

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 20 Outcome Rate (2004) Frame Error RefusalNon-Completion Fixed Effect: Intercept Random Effect: Region variance Residual variance (.00317) (.00008) (.00016) (.00463) (.00013) (.00013) (.00032) (.00900) (.00056) (.00245) Outcome Rate (2005) Frame Error RefusalNon-Completion Fixed Effect: Intercept Random Effect: Region variance Residual variance (.00315) (.00008) (.00011) (.00668) (.00027) (.00116) (.01373) (.00123) (.00269) Source: Author’s elaborations on Istat data Table 9 – One-way random effects ANOVA model: main results REGIONEFFECT

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 21 In order to inspect how far is from each Italian NUTS2 region or geographical macro area to the entire national territory in terms of dissimilarities of data production process quality... SYNTHETIC QUALITY INDICATORS EXPLORING THE MAIN DIFFERENCES IN EU-SILC PARTICIPATION ACROSS NATIONAL TERRITORY SPATIALINDICES RANKING ANALYSIS BY NUTS2 REGIONS BY MACRO AREAS COGRADUATION KENDALL’S RANK CORRELATION

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 22 RANKING ANALYSIS MacroAreaNonContactMacroAreaNonCooperat.MacroAreaNonResponse CenterNorth-WestNorth-EastIslandsSouth North-WestCenterNorth-EastIslandsSouth CenterNorth-WestNorth-EastIslandsSouth MacroAreaNonContactMacroAreaNonCooperat.MacroAreaNonResponse CenterNorth-WestIslandsNorth-EastSouth North-WestCenterNorth-EastIslandsSouth North-WestCenterNorth-EastIslandsSouth Table 10 – Ranking of Italian geographical macro areas (Italy=100) - waves 2004 and 2005 waves 2004 and 2005 Source: Author’s elaborations on Istat data Kendall’s rank correlation

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 23  NON-CONTACT, OVERALL NON-RESPONSE AND, OBVIOUSLY, NON-COMPLETION RATES SIGNIFICANTLY DIFFER BY ITALIAN NUTS2 REGIONS ON BOTH THE WAVES (2004 AND 2005) CONCLUDING REMARKS...  DIFFERENTIALS ACROSS NUTS2 REGIONS ALSO CONCERN THE FRAME ERRORS AND REFUSALS AS THE TWO CRUCIAL SOURCES OF NON-CONTACTS AND NON-COOPERATION, RESPECTIVELY  NON-COOPERATION RATES SEEM TO BE STATISTICALLY DIFFERENT ACROSS NATIONAL TERRITORY ONLY IN 2005  A DOWNWARD TREND OVER WAVE IS REVEALED FOR THE OUTCOME QUALITY INDICATORS CONSIDERED ALSO DUE TO THE PANEL FRAMEWORK OF THE EU-SILC SURVEY  STAYING PUT OF THE ITALIAN REGIONS IN THE SAME (OR IN A SIMILAR) RANKING FOR THE NON-CONTACT AND NON- COOPERATION RATES AS WELL AS FOR THE FRAME ERROR AND REFUSAL RATES

11th July 2008 University of Naples “Parthenope” – Faculty of Economics Department of Statistics and Mathematics for Economic Research 24 … AND FURTHER DEVELOPMENTS  URBANICITY  POPULATION DENSITY  CRIME RATES  LACK OF SOCIAL COHESION  AND SO ON... At a sub-national level, a variety of contextual factors may influence survey participation, such as... REALLY... We deliberately neglected these aspects but we intend to examine them closely afterwards SOCIO-ECONOMIC ENVIRONMENT INFLUENCES ON SURVEY PARTICIPATION (Groves and Couper, 1998)