Istat Quality Policy ESTP Training Course “Quality Management and Survey Quality Measurement” Rome, 24 – 27 September 2013 Marina Signore Director of Research Chief, Division "Metadata, Quality and R&D Projects“ Istat

ISTAT Quality Policy Started in the 90s Strengthened over time Fully harmonised with the European Framework for Quality: European Statistics Code of Practice Eurostat definition of Quality Active participation to the definition of EU framework for Quality Leg on Quality (1999-2001) Sponsorship on Quality (2009-2011) Structure for managing and assessing Quality

Organisation for Quality Division “Metadata, Quality and R&D Projects” - centralised Structure for Quality Assurance (management and assessment) - in charge of providing guidelines, standards and methodological support for quality work in the production sectors Quality Committee established in 2010 and renewed in 2012 - coordinate and supervise the activities for the assessment of quality in Istat and in the Italian National Statistical System (Sistan) - promote and launch initiatives to communicate quality to users

Istat Framework for Quality Improvement Actions Communication on Quality to Users Actions Internal Audit Self-Assessment Assessment of Standard Quality Indicators Assessment Survey Assurance- Quality Control Systems Measurements Quality Guidelines Documentation system Standard Quality Indicators Preconditions

Standards: Methodological Handbooks and Guidelines What type of handbook? Objectives to be reached Internal needs International requirements Istat experience: different types of handbooks over time (e.g. guidelines, recommended practices) different purposes different types of supports

Paper Methodological Handbooks 1989 series “Note e Relazioni”: One handbook for each survey phase Planning; Questionnaire design, Interviewing, Sampling Techniques: Theory and Practice, Variance Estimation, Data Quality Control System, Graphical Representations 2000- 2005 Thematic series: Relevant examples methodologies for preventing, monitoring and evaluating the interviewer effect the methodology and experiences in telephone interviews for social surveys methodologies and techniques for data disclosure control

EU Recommended Practices Following 2001 LEG on Quality recommendations Harmonised at European level Developed in partnership by EU NSIs Istat coordination QDET - Handbook of recommended practices for questionnaire design and testing in the ESS EDIMBUS - Handbook of recommended practices for editing and imputation in cross-sectional business surveys in the ESS

Quality Guidelines Released in 2010 Principles to be followed when planning, running, assessing a survey and for managing process and product quality Basis for auditing and self-assessment Available on Istat website since 2011 English version released in February 2013 http://www.istat.it/en/tools/data-quality/guidelines

Measurements DATQAM: Prerequisite for quality assessment PDCA-cycle: decisions based on facts where to improve effectiveness of improvements Product ex-post control surveys costly Process key process variables by-product of production process

Product Quality Work in co-operation with surveys on specific problems - designing control surveys (e.g. reinterviews) - suggesting new or most appropriate methodologies to use - analyses of the results (in co-operation) Some examples: - measuring coverage errors in telephone surveys - measuring response errors in the Population Census - measuring the interviewer effect on final data - measuring non response bias and variance in mail business surveys - analyses of multi-mode data collection and their impact on data quality

Process Measurements Long tradition starting in the ’90s Currently performed by most surveys Survey monitoring also by means of developing information systems Process and product quality indicators are produced and they feed Istat centralised information system for quality (SIDI/SIQual)

Documentation To keep track of data, metadata and quality indicators Support to planning surveys Support to assessing survey quality Support correct use of data

Strategy for internal documentation Concerns both metadata and standard quality indicators (process and product oriented quality indicators) Concerns all Istat surveys (direct, based on administrative surveys) and secondary studies Istat centralised information system for quality SIDI/SIQual

The SIDI system SIDI is an information system devoted to support the survey managers in the quality control activity: - to monitor the production process - to analyse quality over time and comparisons among surveys on Istat Intranet Quality tool for different users Survey managers Centralised quality managers Istat top management Eurostat and other international organisations

SIDI Standard Quality Indicators Process-oriented SQI Indirect measures of Accuracy - frame errors - nonresponse - coding - editing and imputation Information on survey costs A set of SQI for each sub-component Harmonised with Eurostat SQI Product-oriented SQI Timeliness and punctuality Comparability Coherence A set of SQI for each sub-component Harmonised with Eurostat SQI

The SIDI system: organisation Implementation started in November 2001 Net of quality pilots at survey level quality pilots are trained on quality and documentation of metadata and evaluation of SQI formally appointed one edition of training course every year more than 150 quality pilots trained so far Referee at directorate/division level working group coordinated centrally collect internal users’ needs coordinate the work of the quality pilots Increased awareness of quality problems Increased quality culture

The SIDI system Documented processes 224 surveys, 155 active ones 101 secondary studies, 80 active ones Quality indicators Timeliness 100 % Coverage and nonresponse 100 % Resources 66 % Comparability 60 % Editing and imputation 44 % Coherence preliminary/final estimates 16 % Coherence with other sources 9 % Revision policy 7 %

The SIQual system is the navigation system of SIDI Internally: equipped with standard enquiry functionalities for analysing SQI and metadata Externally: only metadata on survey process are available both in Italian and in English link with data dissemination system (I.stat) reuse of metadata for short methodological notes Future work: To implement sofware for automatic compilation of Eurostat quality reports using metadata and SQI in the system

The SIQual system http://siqual.istat.it/SIQual/lang.do?language=UK

Quality Assessment Policy and Tools Recently launched activity (since 2010) Relies on the quality work done so far Improvement-oriented Strategy for systematic quality assessment Indirect quality assessment: Analyses of Standard Quality Indicators - Direct quality assessment: Auditing and Self-assessment

Indirect Quality Assessment Regular monitoring of the quality of all the processes and products to support survey managers in achieving quality targets Based on the analysis of metadata and standard quality indicators stored in the SIDI information system Report for the Istat’s top management with commented analyses of the information in SIDI (July 2011; July 2012) Level of quality reached; changes from 2001 to 2011; over the last five years; changes from the first report Focus on timeliness and accuracy (coverage and total nonresponse errors). Focus on CATI and mixed modes data collection in 2011 Information provided: average of indicators for groups of processes sharing some given characteristics; trends over time, etc

Monthly (Institutions) Example of quality report Timeliness mean value in days – active surveys by periodicity (2001-2011) timeliness Annual Quarterly Monthly (businesses) Monthly (Institutions) mean n General mean 531 70 86 13 69 15 245 8 Timeliness (days) by year and periodicity (active surveys)

Follow-up activities after first report Dissemination of quality analyses from SIDI within the Institute started in fall 2011 - Preparation of a specific report for each subject matter directorate with the same analyses of the general report but tailored to the processes of that sector Presentation and discussion of the report with the Director and the Chiefs of the Divisions within the directorate In one occasion it was decided to organize a meeting with a single Division for informing staff of the quality aspects of their surveys

Direct Quality Assessment Aimed at verifying the compliance of statistical processes to principles stated in Istat’s Quality Guidelines Based on a combination of internal audits and self-assessments Detailed investigation of statistical processes and their products Aimed at identifying improvement actions

The Procedure Testing phases in 2010 and 2011 Since 2012, direct assessment started on a regular basis Jan- June: Running of audits and self-assessments June – December Approval of Improvement actions by the Quality Committee Flagging the improvement actions in the Annual Program of Activities From 2013: Starting regular follow-up of the implementation of improvement actions Period Audits Self-assessments Tot. 2010 and 2011 7 14 Jan-Jun 2012 5 9 Jan-Jun 2013 17 25 42

The Procedure - The survey manager fills-in Audit Self-assessment Who Team of 3 experts (auditors) Survey manager How - Study of preliminary docs (quality reports: process and products; both compiled using the data and metadata stored in the SIDI/Siqual system) - Auditors interview the survey manager using the audit questionnaire - The survey manager fills-in the self-assessment questionnaire Output Final report (3-5 pages): Part A: Results of the audit (team of auditors) Part B: Improvement actions (survey manager) Part A: Results of the self-assessment (survey manager)   revised by 1 reviewer

Quality Assessment: Conclusions Direct assessment J goes in depth into processes J permits to identify best practices J permits to improve the single process L Workload of actors involved is substantial L Limited number of processes can be evaluated annually Indirect assessment J regular monitoring of processes and products J raises awareness of quality issues and targets J provides comparable information at low cost L does not allow to go in depth L Depends on the information in SIDI (incomplete and not updated data) Combination of the approaches permits to overcome the drawbacks of a single approach but at the price of a higher effort

Communicating Quality Recently launched activity for a structured and regular communication of quality aspects to users New section on Istat website http://www.istat.it/en/about-istat/quality Released in 2011 Istat Quality Policy Work done Codes of Statistics Tools Links to documents Link to SIQual

Final remarks Quality is a long term commitment and should be seen as an investment Istat is collecting the results of past investments However we are aware that there is still a lot of work to do Future plans: To consolidate what has been recently launched To start disseminate in a systematic way quality indicators

References Anitori P., Brancato G., Gismondi R., Signore M. (2000), "The effects of Late Respondents on Retail Trade Monthly Indexes", Proceedings of the International Conference on Establishment Surveys-II, Survey Methods for Business, Farms and Institutions, 17-21 giugno, Buffalo, USA, CD-ROM. Barbieri G e Signore M. (1999), "Statistical research activities at Istat (Statistics Italy)", Research in Official Statistics, Vol. n. 1, pp. 109-114. Bocci L., Muratore M.G., Signore M., Tagliacozzo G. (2002), “The interviewer effect on the Data Collection of Sensitive Questions”, Atti della XLI Riunione Scientifica della SIS, Sessione Data Quality and accuracy, pp299-302 Brancato G., De Angelis R., Mazza L., Signore M. (1998), "Feasibility and effectivness of CATI follow-up in ISTAT Small Business Survey" Proceedings of the 3rd International Conference on Methodological Issues in Official Statistics, Stoccolma, 12-13 ottobre, pp. 151-154 Brancato G, D’Orazio M, Fortini M. (2004) “Response error estimation in presence of Record Linkage error: the case of the Italian Population Census”. Proceedings of the European Conference on Quality and Methodology in Official Statistics (Q2004). Mainz, 24-26 May 2004. CD ROM Brancato G., Corazziari I., Dattilo B., Di Filippo P., Muratore M.G., Perez M., Simeoni G. (2005) “What  are we missing? The effects on the estimations of no phone households in Italy” Paper presentato alla Second International Conference on Telephone Survey Methodology. Florida, January 2006. Brancato G., Pellegrini C., Signore M., Simeoni G., (2004) “Standardising, Evaluating and Documenting Quality: the Implementation of Istat Information System for Survey Documentation – SIDI”, International Conference on Methodologies and Quality in Statistics, Mainz

References Signore M., Brancato G., Simeoni G. (2008), “Multi-mode data collection: What can still be expected?”, invited paper, Proceedings of Statistics Canada Symposium 2008, Data Collection: Challenges, Achievements and New Directions. Signore M., D’Orazio M., Carbini R. (2012), “Quality Assessment at Istat:The Combined Use of Standard Quality Indicator Analysis and Audit Procedures”, European Conference on Quality in Official Statistics (Q2012), Athens