Giovanna Brancato, Giorgia Simeoni Istat, Italy European Conference on Quality in Official Statistics – Q2008, Rome, 8-11 July 2008 Modelling Survey Quality by Structural Equation Models Rome, 9 July 2008

Modelling Survey Quality European Conference on Quality in Official Statistics – Q2008 Rome, 9 July 2008 Quality indicators are commonly used to measure quality components, but their reliability has rarely been evaluated Objectives of the work  To build a model on quality and its components, based on Eurostat quality framework  To explore the capacity of a set of standard quality indicators to measure quality and quality components  To explore the applicability of the Structural Equation Models (SEM) to survey quality context - Useful to analyse the relationships between unobserved (quality and quality components) and observed (indicators) variables

Steps European Conference on Quality in Official Statistics – Q2008 Rome, 9 July Starting model - Complete theoretical model based on Eurostat Standard Quality Indicators 2.Theoretical models - Refinements of the starting model 3.Applications - Restricted models with available indicators

Starting model Rome, 9 July 2008 European Conference on Quality in Official Statistics – Q2008 Accuracy f(variation coefficient) Frame out-of-scope rate Unit response rate Relevance Timeliness & Punctuality Accessibility & clarity Coherence & comparability User satisfaction index Rate of available statistics f(Item response rate) Punctuality Timeliness Length of time series Metadata rate of completeness Number of publications Number of accesses to DB Quality Coherence main secondary use Eurostat Indicators

Theoretical model (1) Rome, 9 July 2008 European Conference on Quality in Official Statistics – Q2008 Accuracy f(variation coefficient) Frame out-of-scope rate Unit response rate Relevance Timeliness & Punctuality Accessibility Coherence & comparability Law regulation Area of interest f(Item response rate) Punctuality Timeliness Length of time series Number of publications Number of accesses to DB Quality Coherence exter. sources Coherence prov/final Geographical detail Law that rules the survey: From “Formal act of the owner” To”European Regulation” A score is assigned to each survey considering the thematic area it observes and matching it with the results of the 2004 CSS Geographical detail level of dissemination of survey results : From “Nation” To “Municipalities”

Theoretical model (2) Rome, 9 July 2008 European Conference on Quality in Official Statistics – Q2008 Accuracy f(variation coefficient) Frame out-of-scope rate Unit response rate Relevance Timeliness & Punctuality Accessibility Coherence & comparability Law regulation Area of interest f(Item response rate) Punctuality Timeliness Length of time series Number of publications Number of accesses to DB Coherence exter. sources Coherence prov/final Geographical detail

Rome, 9 July 2008 Information System for Survey Documentation – SIDI - SIDI documents metadata and Standard Quality Indicators of all Istat surveys and the most relevant secondary studies - Most of the indicators used in the analysis comes or derives from SIDI National Statistical Programme - PSN - Official Italian Statistical Plan of all surveys, secondary studies, research projects and statistical information systems - Most of indicators on relevance comes from PSN Istat Customer Satisfaction Survey The ‘Area of interest’ indicator has been derived using results of the mentioned survey Data Sources European Conference on Quality in Official Statistics – Q2008

Rome, 9 July 2008 Selection of all direct surveys documented in SIDI with most complete set of indicators: a total of 146 observations. Not all indicators of the theoretical model available – > construction of a simplified model with available indicators + ad hoc developed indicators SEM model standard estimation procedure assumes multivariate normality of observed variables: - Transformations (log, sqrt…) applied to continuous non normal indicators - Use of polychoric correlations and evaluation of goodness of fit by means of Satorra-Bentler  2 statistics Data and limitations European Conference on Quality in Official Statistics – Q2008

Rome, 9 July 2008 Indicators European Conference on Quality in Official Statistics – Q2008 IndicatorsApplied modelsTheoretical models Law regulation  Area of interest  Geographical detail  f(variation coefficient)  Resolved unit rate  Rate of eligible units  Response rate  f(item response rate)  Measurement error quality control action  Timeliness (Delay)  Punctuality  Periodicity  Number of publications  Number of accesses to database  On-line dissemination  Length of time series  Coherence provisional vs. final  Coherence with external sources  Meaning inverted by transformations Legenda Not available indicators Introduced indicators Transformed indicators

Analysis Strategy European Conference on Quality in Official Statistics – Q2008 Rome, 9 July 2008 Incremental approach 1. Single common factor model for Relevance 2. Single common factor model for Accuracy 3. Two factors model Relevance and Accuracy 4. Three factors model Relevance, Accuracy and Timeliness 5. Four factors model Relevance, Accuracy, Timeliness and Accessibility 6. Second-order factor model with components + Quality Rationale Simple models support in the interpretation of latent factors Parameter estimates obtained in previous models have been used as initial estimates in further models to solve problems of convergence.

Final Model European Conference on Quality in Official Statistics – Q2008 Goodness of fit 2 Satorra-Bentler  df39 RMSEA0.0 NFI CFI Unfortunately, second-order factor model did not converge with admissible parameter estimates

Conclusions - SEM model revealed a good methodological approach in this context - Applications confirm complexity of concepts and difficulty to synthesise quality indicators for some components, the interpretation of others being more straightforward - Need to collect more indicators in order to improve applications and test the theoretical models Rome, 9 July 2008

THANK YOU FOR YOUR ATTENTION!

Measurement Model for Relevance Rome, 9 July 2008 European Conference on Quality in Official Statistics – Q2008 Observations=3x4/2=6 Parameter=3+2+1=6 The model is saturated, df=0