1. Quality of quality: an analysis of quality indicators and their quality
Aurora De Santis, Riccardo Carbini
Istat, Italy
European Conference on Quality in Official Statistics – Q2010
Helsinki,

2. The evolution of the SIDI system An example of quality analysis
European Conference on Quality in Official Statistics – Q2010
Summary
The evolution of the SIDI system
An example of quality analysis
Quality of quality indicators
Final remarks
Helsinki,

3. The evolution of the SIDI system An example of quality analysis
European Conference on Quality in Official Statistics – Q2010
Summary
The evolution of the SIDI system
An example of quality analysis
Quality of quality indicators
Final remarks
Helsinki,

4. The evolution of the SIDI system (1)
European Conference on Quality in Official Statistics – Q2010
The evolution of the SIDI system (1)
Main characteristics of SIDI, the Istat Documentation System on Surveys
- Process metadata and quality indicators collection for all Istat survey and each survey occasion
- Integrated management of process metadata and quality indicators
- Oriented to internal and external users
Description of the production
process and its characteristics
(also in English)
- Documentation of production processes
Standard and ad hoc quality analyses
Helsinki,

5. The evolution of the SIDI system (2)
European Conference on Quality in Official Statistics – Q2010
The evolution of the SIDI system (2)
Helsinki,

6. The evolution of the SIDI system (3)
European Conference on Quality in Official Statistics – Q2010
The evolution of the SIDI system (3)
During the last 5 years efforts have been addressed
to increase the series of quality indicators collected
The systematic collection of quality indicators
started in 2005 with the collection of the timeliness
and punctuality group for all Istat surveys
From 2006 a new group for each survey was collected and the previously collected series of quality indicators were updated
Helsinki,

7. The evolution of the SIDI system (4)
Titolo del convegno
anche su più righe
European Conference on Quality in Official Statistics – Q2010
The evolution of the SIDI system (4)
Number of occasions of quality indicators available for each group at 2005 and at 2010
Group name
No. of occasions at 2005
No. of occasions at 2010
No. of quality indicators at 2010
No. of series of quality indicators at 2010
Frame
143
980
9,058
1,101
Data collection
12,257
1,549
Editing and Imputation 98
336
7,687
705
Timeliness and punctuality
775
2,450
9,616
767
Costs 34
363
4,139
1,472
Helsinki,

8. The evolution of the SIDI system (5)
European Conference on Quality in Official Statistics – Q2010
The evolution of the SIDI system (5)
At present, a great number of quality indicators
has been collected and calculated
More requirements of quality of the quality indicators collected
More opportunities to exploit the information collected
Helsinki,

9. The evolution of the SIDI system An example of quality analysis
European Conference on Quality in Official Statistics – Q2010
Summary
The evolution of the SIDI system
An example of quality analysis
Quality of quality indicators
Final remarks
Helsinki,

10. An example of quality analysis (1)
European Conference on Quality in Official Statistics – Q2010
An example of quality analysis (1)
The expanded set of data now available can be used to perform more complete analyses than in the past.
Some examples of analyses on quality are reported in the paper.
Mean of timeliness by year and survey periodicity
Mean of frame error rate by year and data collection mode
Mean of response rate by data collection mode
Helsinki,

11. Mean of timeliness by year and survey periodicity (days)
European Conference on Quality in Official Statistics – Q2010
An example of quality analysis (2)
Timeliness=(actual date of first release of final data) - (date of reference for the data)
Mean of timeliness by year and survey periodicity (days)
Monthly Quarterly Yearly
(24) (14) (71)
(26) (13) (66)
(28) (13) (57)
(29) (13) (45)
(25) (13) (26)
(16) (10) (4)
Overall mean (33) (16) (80)
(number of different surveys the mean is based on)
Helsinki,

12. The evolution of the SIDI system An example of quality analysis
European Conference on Quality in Official Statistics – Q2010
Summary
The evolution of the SIDI system
An example of quality analysis
Quality of quality indicators
Final remarks
Helsinki,

13. Quality of quality indicators (1)
European Conference on Quality in Official Statistics – Q2010
Quality of quality indicators (1)
Quality assurance of quality indicators
Training of the quality pilots in charge of collecting and calculating quality indicators
Automatic check (domain and coherence rules) implemented in SIDI
Support of the central quality unit to the quality pilots
Nevertheless some errors may occur, and they can have an impact
Helsinki,

14. Quality of quality indicators (2)
European Conference on Quality in Official Statistics – Q2010
Quality of quality indicators (2)
At present, a large set of quality indicators is calculated every year.
The opportunity is the availability of series of quality indicators
Control charts can be an useful method to analyse the series of quality indicators.
A preliminary analysis of series of timeliness indicator values for a set of surveys using Shewhart individual control charts
Helsinki,

15. Quality of quality indicators (3)
Titolo del convegno
anche su più righe
European Conference on Quality in Official Statistics – Q2010
Quality of quality indicators (3)
Helsinki,

16. Quality of quality indicators (4)
Titolo del convegno
anche su più righe
European Conference on Quality in Official Statistics – Q2010
Quality of quality indicators (4)
Helsinki,

17. Final remarks
The increasing use of the quality indicators collected in Istat for quality evaluation requires trustable and accurate measures.
The great amount of indicators that are currently calculated every year requires further developments of our strategy.
The examples reported in this paper suggest that the control chart methodology could fit this purpose.
In the next years Istat could invest on further analyses and on integrating such methods in the SIDI management subsystem, in order to provide graphical tools for quality pilots as alarm bells on the quality indicator accuracy, when loading the indicators or revising them.
Helsinki,

18. Quality of quality indicators
Titolo del convegno
anche su più righe
European Conference on Quality in Official Statistics – Q2010
Quality of quality indicators
Helsinki,