1. Istat Quality Policy Marina Signore
Measurement of the quality of statistics
3-5 October 2012
Marina Signore
Istat
Division "Metadata, Quality and R&D Projects", Chief

2. ISTAT Quality Policy
Started in the 80s
Strengthened over time
Spread awareness on quality and sound basis for launching improvement actions
Improvement
actions
Standards
Measurements
Reporting
Quality of the system
Integrated
quality of
Products
Processes
Environment

3. ISTAT Quality Policy Fully harmonised with the European Framework for
European Statistics Code of Practice
Eurostat definition of Quality
Active participation to the definition of EU framework for
Quality
Leg on Quality ( )
Sponsorship on Quality ( )
Structure for managing and assessing Quality

4. Organisation for Quality
Division "Metadata, Quality and R&D Projects“
centralised Structure for Quality Assurance (management and assessment)
in charge of providing guidelines, standards and methodological
support for quality work in the production sectors
Quality Committee
established in 2010
- coordinate and supervise the activities for the assessment of quality in Istat and in the Italian National Statistical System (Sistan)
- promote and launch initiatives to communicate quality to users

5. Centralised Quality Work
since 1994
Research activity on methodologies for evaluating data quality (product quality)
Co-operation with surveys to measure non-sampling errors and also to prevent errors by process control
Training activity on quality issues at Istat and for Institutions of Sistan
Development of standard tools for documenting and monitoring production processes

6. Istat Framework for Quality
Improvement Actions
Communication on Quality to Users
Actions
Internal Audit
Self-Assessment
Assessment of Standard Quality Indicators
Assessment
Survey Assurance- Quality Control Systems
Measurements
Quality Guidelines
Documentation system
Standard Quality Indicators
Preconditions

7. Standards: Methodological Handbooks and Guidelines
What type of handbook?
Objectives to be reached
Internal needs
International requirements
Istat experience:
different types of handbooks over time (e.g. guidelines, recommended practices)
different purposes
different types of supports

8. Paper Methodological Handbooks
1989 series “Note e Relazioni”:
One handbook for each survey phase
Planning; Questionnaire design, Interviewing?, Sampling Techniques: Theory and Practice, Variance Estimation, Data Quality Control System, Graphical Representations
Thematic series:
Relevant examples
methodologies for preventing, monitoring and evaluating the interviewer effect
the methodology and experiences in telephone interviews for social surveys
methodologies and techniques for data disclosure control

9. EU Recommended Practices
Following 2001 LEG recommendations
Harmonised at European level
Developed in partnership by EU NSIs
Istat coordination
QDET - Handbook of recommended practices for questionnaire design and testing in the ESS
EDIMBUS - Handbook of recommended practices for editing and imputation in cross-sectional business surveys in
the ESS

10. Quality Guidelines Released in 2009
Principles to be followed when planning, running, assessing a survey and for managing process and prouct quality
Basis for auditing and self-assessment
Available on Istat website
English version will be released by December 2012

11. Standards: Generalised Software
Standardisation of survey operations
Incorporating sound methodologies
Developed by Istat (or bought from other NSIs)
Available for all survey phases
Technical manuals and training courses
Available freely on Istat website

12. Measurements DATQAM: Prerequisite for quality assessment
PDCA-cycle: decisions based on facts
where to improve
effectiveness of improvements
Process
key process variables
by-product of production process
Product
ex-post control surveys
costly

13. Product Quality Work in co-operation with surveys on specific problems
- designing control surveys (e.g. reinterviews)
- suggesting new or most appropriate methodologies to use
- analyses of the results (in co-operation)
Some examples:
- measuring coverage errors in telephone surveys
- measuring response errors in the Population Census
- measuring the interviewer effect on final data
- measuring non response bias and variance in mail business surveys
- analyses of multi-mode data collection and their impact on data quality

14. Process Measurements
Long tradition starting in the ’80s Currently performed by most surveys Survey monitoring also by means of developping information systems
Process and product quality indicators are produced and they feed Istat centralised information system for quality (SIDI/SIQual)

15. Documentation
To keep track of data, metadata and quality indicators Support to planning surveys Support to assessing survey quality Support correct use of data

16. Strategy for internal documentation
Concerns both metadata and standard quality indicators (process and product oriented quality indicators)
Concerns all Istat surveys (direct, based on administrative surveys) and secondary studies
Istat centralised information system for quality (SIDI/SIQual)

17. Quality Assessment Policy and Tools
Recently launched activity (end 2009)
Relies on the quality work done so far
Improvement-oriented
Strategy for systematic quality assessment
Indirect quality assessment: Analyses of Standard Quality Indicators
- Direct quality assessment: Auditing and Self-assessment

18. Indirect Quality Assessment
Regular monitoring of the quality of all the processes and products to support survey managers in achieving quality targets
Based on the analysis of metadata and standard quality indicators stored in the SIDI information system
Report for the Istat’s top management with commented analyses of the information in SIDI (July 2011; July 2012)
Level of quality reached; changes from 2001 to 2011; over the last five years; changes from the first report
Focus on timeliness and accuracy (coverage and total nonresponse errors). Focus on CATI and mixed modes data collection in 2011
Information provided: average of indicators for groups of processes sharing some given characteristics; trends over time, etc

19. Monthly (Institutions)
Example of quality report
Timeliness mean value in days – active surveys by periodicity ( )
timeliness
Annual
Quarterly
Monthly (businesses)
Monthly (Institutions)
mean n
General mean
531 70 86 13 69 15
245 8
Timeliness (days) by year and periodicity (active surveys)

20. Follow-up activities after first report
Dissemination of quality analyses from SIDI within the Institute started in fall 2011.
- Preparation of a specific report for each subject matter directorate with the same analyses of the general report but tailored to the processes of that sector
Presentation and discussion of the report with the Director and the Chiefs of the Divisions within the directorate
In one occasion it was decided to organize a meeting with a single Division for informing staff of the quality aspects of their surveys

21. Direct Quality Assessment
Aimed at verifying the compliance of statistical processes to standards and Guidelines
Based on a combination of internal audits and self-assessments
Detailed investigation of statistical processes and their products
Verify compliance to principles stated in the Istat’s Quality Guidelines

22. The Procedure - The survey manager fills-in Audit Self-assessment Who
Team of 3 experts (auditors)
Survey manager
How
- Study of preliminary docs (quality reports: process and products; both compiled using the data and metadata stored in the SIDI/Siqual system)
- Auditors interview the survey manager using the audit questionnaire
- The survey manager fills-in
the self-assessment
questionnaire
Output
Final report (3-5 pages):
Part A: Results of the audit (team of auditors)
Part B: Improvement actions (survey manager)
Part A: Results of the self-assessment (survey manager)
revised by 1 reviewer

23. The Procedure
Procedure and tools finalised after tests in 2010 and 2011
Period
Audits
Self-assessments
Tot.
Feb-Jun 2010 2 5 7
Oct Apr. 2011 14
Main results of tests:
Revision of questionnaires and Guidelines (seasonal adj.; revision policy; survey with admin. data)
Estimation of the workload => simplification of the original self-assessment procedure
Changes in the procedure, also to take into account Istat’s new organization

24. The Procedure Direct assessment on a regular basis, starting in 2012
January- June
Running of the assessment: 5 audits and 9 self-assessments each year
June – December
Approaval of Improvement actions by the Quality Committee
Flagging the improvement actions in the Annual Program of Activities
From 2013
Starting regular follow-up of the implementation of improvement actions

25. Quality Assessment: Conclusions
Direct assessment
J goes in depth into processes
J permits to identify best practices
J permits to improve the single process
L Workload of actors involved is substantial
L Limited number of processes can be evaluated annually
Indirect assessment
J regular monitoring of processes and products
J raises awareness of quality issues and targets
J provides comparable information at low cost
L does not allow to go in depth
L Depends on the information in SIDI (incomplete and not updated data)
Combination of the approaches permits to overcome the drawbacks of a single approach but at the price of a higher effort

26. Communicating Quality
Recently launched activity for a structured and regular communication of quality aspects to users
New section on Istat website
Released in 2011
Istat Quality Policy
Work done
Codes of Statitistics
Tools
Links to documents
Link to SIQual

27. Final remarks
Quality is a long term commitment and shpuld be seen as an investment
Istat is collecting the results of past investments
However we are aware that there is still a lot of work to do
Future plans:
To consolidate what has been recently launched
To start disseminate in a systematic way quality indicators

28. References
Anitori P., Brancato G., Gismondi R., Signore M. (2000), "The effects of Late Respondents on Retail Trade Monthly Indexes", Proceedings of the International Conference on Establishment Surveys-II, Survey Methods for Business, Farms and Institutions, giugno, Buffalo, USA, CD-ROM. Barbieri G e Signore M. (1999), "Statistical research activities at Istat (Statistics Italy)", Research in Official Statistics, Vol. n. 1, pp Bocci L., Muratore M.G., Signore M., Tagliacozzo G. (2002), “The interviewer effect on the Data Collection of Sensitive Questions”, Atti della XLI Riunione Scientifica della SIS, Sessione Data Quality and accuracy, pp Brancato G., De Angelis R., Mazza L., Signore M. (1998), "Feasibility and effectivness of CATI follow-up in ISTAT Small Business Survey" Proceedings of the 3rd International Conference on Methodological Issues in Official Statistics, Stoccolma, ottobre, pp Brancato G, D’Orazio M, Fortini M. (2004) “Response error estimation in presence of Record Linkage error: the case of the Italian Population Census”. Proceedings of the European Conference on Quality and Methodology in Official Statistics (Q2004). Mainz, May CD ROM Brancato G., Corazziari I., Dattilo B., Di Filippo P., Muratore M.G., Perez M., Simeoni G. (2005) “What are we missing? The effects on the estimations of no phone households in Italy” Paper presentato alla Second International Conference on Telephone Survey Methodology. Florida, January Brancato G., Pellegrini C., Signore M., Simeoni G., (2004) “Standardising, Evaluating and Documenting Quality: the Implementation of Istat Information System for Survey Documentation – SIDI”, International Conference on Methodologies and Quality in Statistics, Mainz

29. References
Signore M., Brancato G., Simeoni G. (2008), “Multi-mode data collection: What can still be expected?”, invited paper, Proceedings of Statistics Canada Symposium 2008, Data Collection: Challenges, Achievements and New Directions. Signore M., D’Orazio M., Carbini R. (2012), “Quality Assessment at Istat:The Combined Use of Standard Quality Indicator Analysis and Audit Procedures”, European Conference on Quality in Official Statistics (Q2012), Athens