Quality of quality: an analysis of quality indicators and their quality Aurora De Santis, Riccardo Carbini Istat, Italy European Conference on Quality in Official Statistics – Q2010 Helsinki, 06.05.2010
The evolution of the SIDI system An example of quality analysis European Conference on Quality in Official Statistics – Q2010 Summary The evolution of the SIDI system An example of quality analysis Quality of quality indicators Final remarks Helsinki, 06.05.2010
The evolution of the SIDI system An example of quality analysis European Conference on Quality in Official Statistics – Q2010 Summary The evolution of the SIDI system An example of quality analysis Quality of quality indicators Final remarks Helsinki, 06.05.2010
The evolution of the SIDI system (1) European Conference on Quality in Official Statistics – Q2010 The evolution of the SIDI system (1) Main characteristics of SIDI, the Istat Documentation System on Surveys - Process metadata and quality indicators collection for all Istat survey and each survey occasion - Integrated management of process metadata and quality indicators - Oriented to internal and external users Description of the production process and its characteristics (also in English) http://siqual.istat.it - Documentation of production processes Standard and ad hoc quality analyses Helsinki, 06.05.2010
The evolution of the SIDI system (2) European Conference on Quality in Official Statistics – Q2010 The evolution of the SIDI system (2) Helsinki, 06.05.2010
The evolution of the SIDI system (3) European Conference on Quality in Official Statistics – Q2010 The evolution of the SIDI system (3) During the last 5 years efforts have been addressed to increase the series of quality indicators collected The systematic collection of quality indicators started in 2005 with the collection of the timeliness and punctuality group for all Istat surveys From 2006 a new group for each survey was collected and the previously collected series of quality indicators were updated Helsinki, 06.05.2010
The evolution of the SIDI system (4) Titolo del convegno anche su più righe European Conference on Quality in Official Statistics – Q2010 The evolution of the SIDI system (4) Number of occasions of quality indicators available for each group at 2005 and at 2010 Group name No. of occasions at 2005 No. of occasions at 2010 No. of quality indicators at 2010 No. of series of quality indicators at 2010 Frame 143 980 9,058 1,101 Data collection 12,257 1,549 Editing and Imputation 98 336 7,687 705 Timeliness and punctuality 775 2,450 9,616 767 Costs 34 363 4,139 1,472 Helsinki, 06.05.2010
The evolution of the SIDI system (5) European Conference on Quality in Official Statistics – Q2010 The evolution of the SIDI system (5) At present, a great number of quality indicators has been collected and calculated More requirements of quality of the quality indicators collected More opportunities to exploit the information collected Helsinki, 06.05.2010
The evolution of the SIDI system An example of quality analysis European Conference on Quality in Official Statistics – Q2010 Summary The evolution of the SIDI system An example of quality analysis Quality of quality indicators Final remarks Helsinki, 06.05.2010
An example of quality analysis (1) European Conference on Quality in Official Statistics – Q2010 An example of quality analysis (1) The expanded set of data now available can be used to perform more complete analyses than in the past. Some examples of analyses on quality are reported in the paper. Mean of timeliness by year and survey periodicity Mean of frame error rate by year and data collection mode Mean of response rate by data collection mode Helsinki, 06.05.2010
Mean of timeliness by year and survey periodicity (days) European Conference on Quality in Official Statistics – Q2010 An example of quality analysis (2) Timeliness=(actual date of first release of final data) - (date of reference for the data) Mean of timeliness by year and survey periodicity (days) Monthly Quarterly Yearly 2004 254 (24) 367 (14) 595 (71) 2005 218 (26) 317 (13) 536 (66) 2006 216 (28) 315 (13) 485 (57) 2007 254 (29) 274 (13) 419 (45) 2008 242 (25) 215 (13) 294 (26) 2009 120 (16) 155 (10) 300 (4) Overall mean 224 (33) 280 (16) 494 (80) (number of different surveys the mean is based on) Helsinki, 06.05.2010
The evolution of the SIDI system An example of quality analysis European Conference on Quality in Official Statistics – Q2010 Summary The evolution of the SIDI system An example of quality analysis Quality of quality indicators Final remarks Helsinki, 06.05.2010
Quality of quality indicators (1) European Conference on Quality in Official Statistics – Q2010 Quality of quality indicators (1) Quality assurance of quality indicators Training of the quality pilots in charge of collecting and calculating quality indicators Automatic check (domain and coherence rules) implemented in SIDI Support of the central quality unit to the quality pilots Nevertheless some errors may occur, and they can have an impact Helsinki, 06.05.2010
Quality of quality indicators (2) European Conference on Quality in Official Statistics – Q2010 Quality of quality indicators (2) At present, a large set of quality indicators is calculated every year. The opportunity is the availability of series of quality indicators Control charts can be an useful method to analyse the series of quality indicators. A preliminary analysis of series of timeliness indicator values for a set of surveys using Shewhart individual control charts Helsinki, 06.05.2010
Quality of quality indicators (3) Titolo del convegno anche su più righe European Conference on Quality in Official Statistics – Q2010 Quality of quality indicators (3) Helsinki, 06.05.2010
Quality of quality indicators (4) Titolo del convegno anche su più righe European Conference on Quality in Official Statistics – Q2010 Quality of quality indicators (4) Helsinki, 06.05.2010
Final remarks The increasing use of the quality indicators collected in Istat for quality evaluation requires trustable and accurate measures. The great amount of indicators that are currently calculated every year requires further developments of our strategy. The examples reported in this paper suggest that the control chart methodology could fit this purpose. In the next years Istat could invest on further analyses and on integrating such methods in the SIDI management subsystem, in order to provide graphical tools for quality pilots as alarm bells on the quality indicator accuracy, when loading the indicators or revising them. Helsinki, 06.05.2010
Quality of quality indicators Titolo del convegno anche su più righe European Conference on Quality in Official Statistics – Q2010 Quality of quality indicators Helsinki, 06.05.2010