1. Istat Quality Policy
ESTP Training Course “Quality Management and Survey Quality Measurement” Rome, 24 – 27 September 2013
Marina Signore
Director of Research
Chief, Division "Metadata, Quality and R&D Projects“
Istat

2. ISTAT Quality Policy
Started in the 90s
Strengthened over time
Fully harmonised with the European Framework for Quality:
European Statistics Code of Practice
Eurostat definition of Quality
Active participation to the definition of EU framework for Quality
Leg on Quality ( )
Sponsorship on Quality ( )
Structure for managing and assessing Quality

3. Organisation for Quality
Division “Metadata, Quality and R&D Projects”
- centralised Structure for Quality Assurance (management and assessment)
- in charge of providing guidelines, standards and methodological support for quality work in the production sectors
Quality Committee
established in 2010 and renewed in 2012
- coordinate and supervise the activities for the assessment of quality in Istat and in the Italian National Statistical System (Sistan)
- promote and launch initiatives to communicate quality to users

4. Istat Framework for Quality
Improvement Actions
Communication on Quality to Users
Actions
Internal Audit
Self-Assessment
Assessment of Standard Quality Indicators
Assessment
Survey Assurance- Quality Control Systems
Measurements
Quality Guidelines
Documentation system
Standard Quality Indicators
Preconditions

5. Standards: Methodological Handbooks and Guidelines
What type of handbook?
Objectives to be reached
Internal needs
International requirements
Istat experience:
different types of handbooks over time (e.g. guidelines, recommended practices)
different purposes
different types of supports

6. Paper Methodological Handbooks
1989 series “Note e Relazioni”:
One handbook for each survey phase
Planning; Questionnaire design, Interviewing, Sampling Techniques: Theory and Practice, Variance Estimation, Data Quality Control System, Graphical Representations
Thematic series:
Relevant examples
methodologies for preventing, monitoring and evaluating the interviewer effect
the methodology and experiences in telephone interviews for social surveys
methodologies and techniques for data disclosure control

7. EU Recommended Practices
Following 2001 LEG on Quality recommendations
Harmonised at European level
Developed in partnership by EU NSIs
Istat coordination
QDET - Handbook of recommended practices for questionnaire design and testing in the ESS
EDIMBUS - Handbook of recommended practices for editing and imputation in cross-sectional business surveys in the ESS

8. Quality Guidelines Released in 2010
Principles to be followed when planning, running, assessing a survey and for managing process and product quality
Basis for auditing and self-assessment
Available on Istat website since 2011
English version released in February

9. Measurements DATQAM: Prerequisite for quality assessment
PDCA-cycle: decisions based on facts
where to improve
effectiveness of improvements
Product
ex-post control surveys
costly
Process
key process variables
by-product of production process

10. Product Quality Work in co-operation with surveys on specific problems
- designing control surveys (e.g. reinterviews)
- suggesting new or most appropriate methodologies to use
- analyses of the results (in co-operation)
Some examples:
- measuring coverage errors in telephone surveys
- measuring response errors in the Population Census
- measuring the interviewer effect on final data
- measuring non response bias and variance in mail business surveys
- analyses of multi-mode data collection and their impact on data quality

11. Process Measurements
Long tradition starting in the ’90s Currently performed by most surveys Survey monitoring also by means of developing information systems
Process and product quality indicators are produced and they feed Istat centralised information system for quality (SIDI/SIQual)

12. Documentation To keep track of data, metadata and quality indicators
Support to planning surveys
Support to assessing survey quality
Support correct use of data

13. Strategy for internal documentation
Concerns both metadata and standard quality indicators (process and product oriented quality indicators)
Concerns all Istat surveys (direct, based on administrative surveys) and secondary studies
Istat centralised information system for quality SIDI/SIQual

14. The SIDI system
SIDI is an information system devoted to support the survey managers in the quality control activity:
- to monitor the production process
- to analyse quality over time and comparisons among surveys on Istat Intranet
Quality tool for different users
Survey managers
Centralised quality managers
Istat top management
Eurostat and other international organisations

15. SIDI Standard Quality Indicators
Process-oriented SQI
Indirect measures of Accuracy
- frame errors
- nonresponse
- coding
- editing and imputation
Information on survey costs
A set of SQI for each sub-component
Harmonised with Eurostat SQI
Product-oriented SQI
Timeliness and punctuality
Comparability
Coherence
A set of SQI for each sub-component
Harmonised with Eurostat SQI

16. The SIDI system: organisation
Implementation started in November 2001
Net of quality pilots at survey level
quality pilots are trained on quality and documentation of metadata and evaluation of SQI
formally appointed
one edition of training course every year
more than 150 quality pilots trained so far
Referee at directorate/division level
working group coordinated centrally
collect internal users’ needs
coordinate the work of the quality pilots
Increased awareness of quality problems
Increased quality culture

17. The SIDI system
Documented processes 224 surveys, 155 active ones 101 secondary studies, 80 active ones
Quality indicators
Timeliness %
Coverage and nonresponse %
Resources %
Comparability %
Editing and imputation %
Coherence preliminary/final estimates 16 %
Coherence with other sources %
Revision policy %

18. The SIQual system is the navigation system of SIDI Internally:
equipped with standard enquiry functionalities for analysing SQI and metadata
Externally:
only metadata on survey process are available both in Italian and in English
link with data dissemination system (I.stat)
reuse of metadata for short methodological notes
Future work:
To implement sofware for automatic compilation of Eurostat quality reports using metadata and SQI in the system

19. The SIQual system

20. Quality Assessment Policy and Tools
Recently launched activity (since 2010)
Relies on the quality work done so far
Improvement-oriented
Strategy for systematic quality assessment
Indirect quality assessment: Analyses of Standard Quality Indicators
- Direct quality assessment: Auditing and Self-assessment

21. Indirect Quality Assessment
Regular monitoring of the quality of all the processes and products to support survey managers in achieving quality targets
Based on the analysis of metadata and standard quality indicators stored in the SIDI information system
Report for the Istat’s top management with commented analyses of the information in SIDI (July 2011; July 2012)
Level of quality reached; changes from 2001 to 2011; over the last five years; changes from the first report
Focus on timeliness and accuracy (coverage and total nonresponse errors). Focus on CATI and mixed modes data collection in 2011
Information provided: average of indicators for groups of processes sharing some given characteristics; trends over time, etc

22. Monthly (Institutions)
Example of quality report
Timeliness mean value in days – active surveys by periodicity ( )
timeliness
Annual
Quarterly
Monthly (businesses)
Monthly (Institutions)
mean n
General mean
531 70 86 13 69 15
245 8
Timeliness (days) by year and periodicity (active surveys)

23. Follow-up activities after first report
Dissemination of quality analyses from SIDI within the Institute started in fall 2011
- Preparation of a specific report for each subject matter directorate with the same analyses of the general report but tailored to the processes of that sector
Presentation and discussion of the report with the Director and the Chiefs of the Divisions within the directorate
In one occasion it was decided to organize a meeting with a single Division for informing staff of the quality aspects of their surveys

24. Direct Quality Assessment
Aimed at verifying the compliance of statistical processes to principles stated in Istat’s Quality Guidelines
Based on a combination of internal audits and self-assessments
Detailed investigation of statistical processes and their products
Aimed at identifying improvement actions

25. The Procedure Testing phases in 2010 and 2011
Since 2012, direct assessment started on a regular basis
Jan- June: Running of audits and self-assessments
June – December
Approval of Improvement actions by the Quality Committee
Flagging the improvement actions in the Annual Program of Activities
From 2013: Starting regular follow-up of the implementation of improvement actions
Period
Audits
Self-assessments
Tot.
2010 and 2011 7 14
Jan-Jun 2012 5 9
Jan-Jun 2013 17 25 42

26. The Procedure - The survey manager fills-in Audit Self-assessment Who
Team of 3 experts (auditors)
Survey manager
How
- Study of preliminary docs (quality reports: process and products; both compiled using the data and metadata stored in the SIDI/Siqual system)
- Auditors interview the survey manager using the audit questionnaire
- The survey manager fills-in
the self-assessment
questionnaire
Output
Final report (3-5 pages):
Part A: Results of the audit (team of auditors)
Part B: Improvement actions (survey manager)
Part A: Results of the self-assessment (survey manager)
revised by 1 reviewer

27. Quality Assessment: Conclusions
Direct assessment
J goes in depth into processes
J permits to identify best practices
J permits to improve the single process
L Workload of actors involved is substantial
L Limited number of processes can be evaluated annually
Indirect assessment
J regular monitoring of processes and products
J raises awareness of quality issues and targets
J provides comparable information at low cost
L does not allow to go in depth
L Depends on the information in SIDI (incomplete and not updated data)
Combination of the approaches permits to overcome the drawbacks of a single approach but at the price of a higher effort

28. Communicating Quality
Recently launched activity for a structured and regular communication of quality aspects to users
New section on Istat website
Released in 2011
Istat Quality Policy
Work done
Codes of Statistics
Tools
Links to documents
Link to SIQual

29. Final remarks
Quality is a long term commitment and should be seen as an investment
Istat is collecting the results of past investments
However we are aware that there is still a lot of work to do
Future plans:
To consolidate what has been recently launched
To start disseminate in a systematic way quality indicators

30. References
Anitori P., Brancato G., Gismondi R., Signore M. (2000), "The effects of Late Respondents on Retail Trade Monthly Indexes", Proceedings of the International Conference on Establishment Surveys-II, Survey Methods for Business, Farms and Institutions, giugno, Buffalo, USA, CD-ROM.
Barbieri G e Signore M. (1999), "Statistical research activities at Istat (Statistics Italy)", Research in Official Statistics, Vol. n. 1, pp
Bocci L., Muratore M.G., Signore M., Tagliacozzo G. (2002), “The interviewer effect on the Data Collection of Sensitive Questions”, Atti della XLI Riunione Scientifica della SIS, Sessione Data Quality and accuracy, pp
Brancato G., De Angelis R., Mazza L., Signore M. (1998), "Feasibility and effectivness of CATI follow-up in ISTAT Small Business Survey" Proceedings of the 3rd International Conference on Methodological Issues in Official Statistics, Stoccolma, ottobre, pp
Brancato G, D’Orazio M, Fortini M. (2004) “Response error estimation in presence of Record Linkage error: the case of the Italian Population Census”. Proceedings of the European Conference on Quality and Methodology in Official Statistics (Q2004). Mainz, May CD ROM
Brancato G., Corazziari I., Dattilo B., Di Filippo P., Muratore M.G., Perez M., Simeoni G. (2005) “What  are we missing? The effects on the estimations of no phone households in Italy” Paper presentato alla Second International Conference on Telephone Survey Methodology. Florida, January 2006.
Brancato G., Pellegrini C., Signore M., Simeoni G., (2004) “Standardising, Evaluating and Documenting Quality: the Implementation of Istat Information System for Survey Documentation – SIDI”, International Conference on Methodologies and Quality in Statistics, Mainz

31. References
Signore M., Brancato G., Simeoni G. (2008), “Multi-mode data collection: What can still be expected?”, invited paper, Proceedings of Statistics Canada Symposium 2008, Data Collection: Challenges, Achievements and New Directions. Signore M., D’Orazio M., Carbini R. (2012), “Quality Assessment at Istat:The Combined Use of Standard Quality Indicator Analysis and Audit Procedures”, European Conference on Quality in Official Statistics (Q2012), Athens