Istat Quality Policy Marina Signore Measurement of the quality of statistics 3-5 October 2012   Marina Signore Istat Division "Metadata, Quality and R&D Projects", Chief

ISTAT Quality Policy Started in the 80s Strengthened over time Spread awareness on quality and sound basis for launching improvement actions Improvement actions Standards Measurements Reporting Quality of the system Integrated quality of Products Processes Environment

ISTAT Quality Policy Fully harmonised with the European Framework for European Statistics Code of Practice Eurostat definition of Quality Active participation to the definition of EU framework for Quality Leg on Quality (1999-2001) Sponsorship on Quality (2009-2011) Structure for managing and assessing Quality

Organisation for Quality Division "Metadata, Quality and R&D Projects“ centralised Structure for Quality Assurance (management and assessment) in charge of providing guidelines, standards and methodological support for quality work in the production sectors Quality Committee established in 2010 - coordinate and supervise the activities for the assessment of quality in Istat and in the Italian National Statistical System (Sistan) - promote and launch initiatives to communicate quality to users

Centralised Quality Work since 1994 Research activity on methodologies for evaluating data quality (product quality) Co-operation with surveys to measure non-sampling errors and also to prevent errors by process control Training activity on quality issues at Istat and for Institutions of Sistan Development of standard tools for documenting and monitoring production processes

Istat Framework for Quality Improvement Actions Communication on Quality to Users Actions Internal Audit Self-Assessment Assessment of Standard Quality Indicators Assessment Survey Assurance- Quality Control Systems Measurements Quality Guidelines Documentation system Standard Quality Indicators Preconditions

Standards: Methodological Handbooks and Guidelines What type of handbook? Objectives to be reached Internal needs International requirements Istat experience: different types of handbooks over time (e.g. guidelines, recommended practices) different purposes different types of supports

Paper Methodological Handbooks 1989 series “Note e Relazioni”: One handbook for each survey phase Planning; Questionnaire design, Interviewing?, Sampling Techniques: Theory and Practice, Variance Estimation, Data Quality Control System, Graphical Representations 2000- 2005 Thematic series: Relevant examples methodologies for preventing, monitoring and evaluating the interviewer effect the methodology and experiences in telephone interviews for social surveys methodologies and techniques for data disclosure control

EU Recommended Practices Following 2001 LEG recommendations Harmonised at European level Developed in partnership by EU NSIs Istat coordination QDET - Handbook of recommended practices for questionnaire design and testing in the ESS EDIMBUS - Handbook of recommended practices for editing and imputation in cross-sectional business surveys in the ESS

Quality Guidelines Released in 2009 Principles to be followed when planning, running, assessing a survey and for managing process and prouct quality Basis for auditing and self-assessment Available on Istat website English version will be released by December 2012

Standards: Generalised Software Standardisation of survey operations Incorporating sound methodologies Developed by Istat (or bought from other NSIs) Available for all survey phases Technical manuals and training courses Available freely on Istat website

Measurements DATQAM: Prerequisite for quality assessment PDCA-cycle: decisions based on facts where to improve effectiveness of improvements Process key process variables by-product of production process Product ex-post control surveys costly

Product Quality Work in co-operation with surveys on specific problems - designing control surveys (e.g. reinterviews) - suggesting new or most appropriate methodologies to use - analyses of the results (in co-operation) Some examples: - measuring coverage errors in telephone surveys - measuring response errors in the Population Census - measuring the interviewer effect on final data - measuring non response bias and variance in mail business surveys - analyses of multi-mode data collection and their impact on data quality

Process Measurements Long tradition starting in the ’80s Currently performed by most surveys Survey monitoring also by means of developping information systems Process and product quality indicators are produced and they feed Istat centralised information system for quality (SIDI/SIQual)

Documentation To keep track of data, metadata and quality indicators Support to planning surveys Support to assessing survey quality Support correct use of data

Strategy for internal documentation Concerns both metadata and standard quality indicators (process and product oriented quality indicators) Concerns all Istat surveys (direct, based on administrative surveys) and secondary studies Istat centralised information system for quality (SIDI/SIQual)

Quality Assessment Policy and Tools Recently launched activity (end 2009) Relies on the quality work done so far Improvement-oriented Strategy for systematic quality assessment Indirect quality assessment: Analyses of Standard Quality Indicators - Direct quality assessment: Auditing and Self-assessment

Indirect Quality Assessment Regular monitoring of the quality of all the processes and products to support survey managers in achieving quality targets Based on the analysis of metadata and standard quality indicators stored in the SIDI information system Report for the Istat’s top management with commented analyses of the information in SIDI (July 2011; July 2012) Level of quality reached; changes from 2001 to 2011; over the last five years; changes from the first report Focus on timeliness and accuracy (coverage and total nonresponse errors). Focus on CATI and mixed modes data collection in 2011 Information provided: average of indicators for groups of processes sharing some given characteristics; trends over time, etc

Monthly (Institutions) Example of quality report Timeliness mean value in days – active surveys by periodicity (2001-2011) timeliness Annual Quarterly Monthly (businesses) Monthly (Institutions) mean n General mean 531 70 86 13 69 15 245 8 Timeliness (days) by year and periodicity (active surveys)

Follow-up activities after first report Dissemination of quality analyses from SIDI within the Institute started in fall 2011. - Preparation of a specific report for each subject matter directorate with the same analyses of the general report but tailored to the processes of that sector Presentation and discussion of the report with the Director and the Chiefs of the Divisions within the directorate In one occasion it was decided to organize a meeting with a single Division for informing staff of the quality aspects of their surveys

Direct Quality Assessment Aimed at verifying the compliance of statistical processes to standards and Guidelines Based on a combination of internal audits and self-assessments Detailed investigation of statistical processes and their products Verify compliance to principles stated in the Istat’s Quality Guidelines

The Procedure - The survey manager fills-in Audit Self-assessment Who Team of 3 experts (auditors) Survey manager How - Study of preliminary docs (quality reports: process and products; both compiled using the data and metadata stored in the SIDI/Siqual system) - Auditors interview the survey manager using the audit questionnaire - The survey manager fills-in the self-assessment questionnaire Output Final report (3-5 pages): Part A: Results of the audit (team of auditors) Part B: Improvement actions (survey manager) Part A: Results of the self-assessment (survey manager)   revised by 1 reviewer

The Procedure Procedure and tools finalised after tests in 2010 and 2011 Period Audits Self-assessments Tot. Feb-Jun 2010 2 5 7 Oct. 2010-Apr. 2011 14 Main results of tests: Revision of questionnaires and Guidelines (seasonal adj.; revision policy; survey with admin. data) Estimation of the workload => simplification of the original self-assessment procedure Changes in the procedure, also to take into account Istat’s new organization

The Procedure Direct assessment on a regular basis, starting in 2012 January- June Running of the assessment: 5 audits and 9 self-assessments each year June – December Approaval of Improvement actions by the Quality Committee Flagging the improvement actions in the Annual Program of Activities From 2013 Starting regular follow-up of the implementation of improvement actions

Quality Assessment: Conclusions Direct assessment J goes in depth into processes J permits to identify best practices J permits to improve the single process L Workload of actors involved is substantial L Limited number of processes can be evaluated annually Indirect assessment J regular monitoring of processes and products J raises awareness of quality issues and targets J provides comparable information at low cost L does not allow to go in depth L Depends on the information in SIDI (incomplete and not updated data) Combination of the approaches permits to overcome the drawbacks of a single approach but at the price of a higher effort

Communicating Quality Recently launched activity for a structured and regular communication of quality aspects to users New section on Istat website Released in 2011 http://www.istat.it/en/about-istat/quality Istat Quality Policy Work done Codes of Statitistics Tools Links to documents Link to SIQual

Final remarks Quality is a long term commitment and shpuld be seen as an investment Istat is collecting the results of past investments However we are aware that there is still a lot of work to do Future plans: To consolidate what has been recently launched To start disseminate in a systematic way quality indicators

References Anitori P., Brancato G., Gismondi R., Signore M. (2000), "The effects of Late Respondents on Retail Trade Monthly Indexes", Proceedings of the International Conference on Establishment Surveys-II, Survey Methods for Business, Farms and Institutions, 17-21 giugno, Buffalo, USA, CD-ROM. Barbieri G e Signore M. (1999), "Statistical research activities at Istat (Statistics Italy)", Research in Official Statistics, Vol. n. 1, pp. 109-114. Bocci L., Muratore M.G., Signore M., Tagliacozzo G. (2002), “The interviewer effect on the Data Collection of Sensitive Questions”, Atti della XLI Riunione Scientifica della SIS, Sessione Data Quality and accuracy, pp299-302 Brancato G., De Angelis R., Mazza L., Signore M. (1998), "Feasibility and effectivness of CATI follow-up in ISTAT Small Business Survey" Proceedings of the 3rd International Conference on Methodological Issues in Official Statistics, Stoccolma, 12-13 ottobre, pp. 151-154 Brancato G, D’Orazio M, Fortini M. (2004) “Response error estimation in presence of Record Linkage error: the case of the Italian Population Census”. Proceedings of the European Conference on Quality and Methodology in Official Statistics (Q2004). Mainz, 24-26 May 2004. CD ROM Brancato G., Corazziari I., Dattilo B., Di Filippo P., Muratore M.G., Perez M., Simeoni G. (2005) “What are we missing? The effects on the estimations of no phone households in Italy” Paper presentato alla Second International Conference on Telephone Survey Methodology. Florida, January 2006. Brancato G., Pellegrini C., Signore M., Simeoni G., (2004) “Standardising, Evaluating and Documenting Quality: the Implementation of Istat Information System for Survey Documentation – SIDI”, International Conference on Methodologies and Quality in Statistics, Mainz

References Signore M., Brancato G., Simeoni G. (2008), “Multi-mode data collection: What can still be expected?”, invited paper, Proceedings of Statistics Canada Symposium 2008, Data Collection: Challenges, Achievements and New Directions. Signore M., D’Orazio M., Carbini R. (2012), “Quality Assessment at Istat:The Combined Use of Standard Quality Indicator Analysis and Audit Procedures”, European Conference on Quality in Official Statistics (Q2012), Athens