Istat Quality Policy Marina Signore

Slides:



Advertisements
Similar presentations
Implementation of the CoP in SLOVENIA Cooperation with data users Genovefa RUŽIĆ Deputy Director-General.
Advertisements

Quality Guidelines for statistical processes using administrative data European Conference on Quality in Official Statistics Q2014 Giovanna Brancato, Francesco.
The quality framework of European statistics by the ESCB Quality Conference Vienna, 3 June 2014 Aurel Schubert 1) European Central Bank 1) This presentation.
Information System for Quality Documentation A Short Presentation for the ESTP Course “Data Dissemination and Publication of Statistics” by Sonia Vittozzi.
OECD Short-Term Economic Statistics Working PartyJune Analysis of revisions for short-term economic statistics Richard McKenzie OECD OECD Short.
The use and convergence of quality assurance frameworks for international and supranational organisations compiling statistics The European Conference.
Giovanna Brancato, Marina Signore Istat Work Session on Statistical Metadata (METIS) Metadata and Quality Indicators Reuse for Quality reporting Geneva,
Implementing ESS standards for reference metadata and quality reporting at Istat Work Session on Statistical Metadata Topic (i): Metadata standards and.
European Conference on Quality in Official Statistics (Q2010) 4-6 May 2010, Helsinki, Finland Brancato G., Carbini R., Murgia M., Simeoni G. Istat, Italian.
Quality assurance activities at EUROSTAT CCSA Conference Helsinki, 6-7 May 2010 Martina Hahn, Eurostat.
8-11-Jul-07 How to increase quality of Principal European Economic Indicators? Roberto Barcellan, Brian Newson, Klaus Wurm Eurostat.
Marina Signore Head of Service “Audit for Quality Istat Assessing Quality through Auditing and Self-Assessment Signore M., Carbini R., D’Orazio M., Brancato.
OECD Short-Term Economic Statistics Working PartyJune Richard McKenzie OECD OECD / Eurostat taskforce on performing revisions analysis for sub-annual.
Overview of quality work in Statistics Denmark Kirsten Wismer.
Quality issues on the way from survey to administrative data: the case of SBS statistics of microenterprises in Slovakia Andrej Vallo, Andrea Bielakova.
Short-Term Economic Statistics Working PartyJune Short Term Economic Statistics Timeliness Framework Richard McKenzie OECD.
Supporting Researchers and Institutions in Exploiting Administrative Databases for Statistical Purposes: Istat’s Strategy G. D’Angiolini, P. De Salvo,
1 C. ARRIBAS, D. LORCA, A. SALINERO & A. COLMENERO Measuring statistical quality at the Spanish National Statistical Institute.
Implementation of the European Statistics Code of Practice Yalta September 2009 Pieter Everaers, Eurostat.
Developing and applying business process models in practice Statistics Norway Jenny Linnerud and Anne Gro Hustoft.
Regional Seminar on Promotion and Utilization of Census Results and on the Revision on the United Nations Principles and Recommendations for Population.
United Nations Oslo City Group on Energy Statistics OG7, Helsinki, Finland October 2012 ESCM Chapter 8: Data Quality and Meta Data 1.
Sponsorship on Quality The final report Zsuzsanna Kovács Expert Group Meeting on National Quality Assurance Frameworks UNSD, New York, September.
Short-Term Economic Statistics Working PartyJune Short Term Economic Statistics Timeliness Framework Richard McKenzie OECD.
13 November, 2014 Seminar on Quality Reports QUALITY REPORTS EXPERIENCE OF STATISTICS LITHUANIA Nadiežda Alejeva Head, Price Statistics.
1 Recent developments in quality related matters in the ESS High level seminar for Eastern Europe, Caucasus and Central Asia countries Claudia Junker,
Session topic (i) – Editing Administrative and Census data Discussants Orietta Luzi and Heather Wagstaff UNECE Worksession on Statistical Data Editing.
Quality declarations Study visit from Ukraine 19. March 2015
United Nations Statistics Division
Short Training Course on Agricultural Cost of Production Statistics
Implementation of Quality indicators for administrative data
Towards more flexibility in responding to users’ needs
The ESS vision, ESSnets and SDMX
Quality assurance in official statistics
Italian National Institute of Statistics Modernisation Story
Data Collection techniques Marina Signore
Survey phases, survey errors and quality control system
Survey phases, survey errors and quality control system
Overview of the ESS quality framework and context
Quality in the Spanish Statistical Office Alfredo Cristóbal Cristóbal
Item 10 – Conclusions of the meeting
ESSnet Projects Pascal JACQUES Unit/B5 Methodology and research
Aurora De Santis, Riccardo Carbini Istat, Italy
Quality Management Frameworks
ESS Standardisation State of play
Towards 2021 Population Census in the Republic of Serbia
Introduction to Quality Concepts
Quality assessment ESTP Training Course “Quality Management and survey Quality Measurement” Rome, 24 – 27 September 2013 Giorgia Simeoni Researcher Unit.
Assessment of Quality in Statistics GLOBAL ASSESSMENTS, PEER REVIEWS AND SECTOR REVIEWS IN THE ENLARGEMENT AND ENP COUNTRIES Mirela Kadic, Project Manager.
Draft Methodology for impact analysis of ESS.VIP Projects
Sub-Regional Workshop on International Merchandise Trade Statistics Compilation and Export and Import Unit Value Indices 21 – 25 November Guam.
Quality Assurance in the European Statistical System
Palestinian Central Bureau of Statistics
Istat Quality Policy ESTP Training Course “Quality Management and Survey Quality Measurement” Rome, 24 – 27 September 2013 Marina Signore Director of.
The European Statistics Code of Practice - a Basis for Eurostat’s Quality Assurance Framework Marie Bohatá Deputy Director General, Eurostat ... Strategic.
Quality Reporting in CBS
United Nations Statistics Division
The new quality strategy in the modernised Italian National Statistical Institute Giovanna Brancato Giorgia Simeoni, Antonia Boggia,
Streamlining statistical production
Karin Blix, Statistics Denmark,
GSBPM AND ISO AS QUALITY MANAGEMENT SYSTEM TOOLS: AZERBAIJAN EXPERIENCE Yusif Yusifov, Deputy Chairman of the State Statistical Committee of the Republic.
2.7 Annex 3 – Quality reports
MIMOD – Project overview
Meeting of the EHIS Technical Group Luxembourg January 2012
Policy Group on Statistical Cooperation October 2014, Antalya
European Statistics Code of Practice
Q European Conference on Quality in Official Statistics
Overview of the ESS quality framework and context
ESS conceptual standards for quality reporting
GSBPM Giorgia Simeoni, Istat,
Presentation transcript:

Istat Quality Policy Marina Signore Measurement of the quality of statistics 3-5 October 2012   Marina Signore Istat Division "Metadata, Quality and R&D Projects", Chief

ISTAT Quality Policy Started in the 80s Strengthened over time Spread awareness on quality and sound basis for launching improvement actions Improvement actions Standards Measurements Reporting Quality of the system Integrated quality of Products Processes Environment

ISTAT Quality Policy Fully harmonised with the European Framework for European Statistics Code of Practice Eurostat definition of Quality Active participation to the definition of EU framework for Quality Leg on Quality (1999-2001) Sponsorship on Quality (2009-2011) Structure for managing and assessing Quality

Organisation for Quality Division "Metadata, Quality and R&D Projects“ centralised Structure for Quality Assurance (management and assessment) in charge of providing guidelines, standards and methodological support for quality work in the production sectors Quality Committee established in 2010 - coordinate and supervise the activities for the assessment of quality in Istat and in the Italian National Statistical System (Sistan) - promote and launch initiatives to communicate quality to users

Centralised Quality Work since 1994 Research activity on methodologies for evaluating data quality (product quality) Co-operation with surveys to measure non-sampling errors and also to prevent errors by process control Training activity on quality issues at Istat and for Institutions of Sistan Development of standard tools for documenting and monitoring production processes

Istat Framework for Quality Improvement Actions Communication on Quality to Users Actions Internal Audit Self-Assessment Assessment of Standard Quality Indicators Assessment Survey Assurance- Quality Control Systems Measurements Quality Guidelines Documentation system Standard Quality Indicators Preconditions

Standards: Methodological Handbooks and Guidelines What type of handbook? Objectives to be reached Internal needs International requirements Istat experience: different types of handbooks over time (e.g. guidelines, recommended practices) different purposes different types of supports

Paper Methodological Handbooks 1989 series “Note e Relazioni”: One handbook for each survey phase Planning; Questionnaire design, Interviewing?, Sampling Techniques: Theory and Practice, Variance Estimation, Data Quality Control System, Graphical Representations 2000- 2005 Thematic series: Relevant examples methodologies for preventing, monitoring and evaluating the interviewer effect the methodology and experiences in telephone interviews for social surveys methodologies and techniques for data disclosure control

EU Recommended Practices Following 2001 LEG recommendations Harmonised at European level Developed in partnership by EU NSIs Istat coordination QDET - Handbook of recommended practices for questionnaire design and testing in the ESS EDIMBUS - Handbook of recommended practices for editing and imputation in cross-sectional business surveys in the ESS

Quality Guidelines Released in 2009 Principles to be followed when planning, running, assessing a survey and for managing process and prouct quality Basis for auditing and self-assessment Available on Istat website English version will be released by December 2012

Standards: Generalised Software Standardisation of survey operations Incorporating sound methodologies Developed by Istat (or bought from other NSIs) Available for all survey phases Technical manuals and training courses Available freely on Istat website

Measurements DATQAM: Prerequisite for quality assessment PDCA-cycle: decisions based on facts where to improve effectiveness of improvements Process key process variables by-product of production process Product ex-post control surveys costly

Product Quality Work in co-operation with surveys on specific problems - designing control surveys (e.g. reinterviews) - suggesting new or most appropriate methodologies to use - analyses of the results (in co-operation) Some examples: - measuring coverage errors in telephone surveys - measuring response errors in the Population Census - measuring the interviewer effect on final data - measuring non response bias and variance in mail business surveys - analyses of multi-mode data collection and their impact on data quality

Process Measurements Long tradition starting in the ’80s Currently performed by most surveys Survey monitoring also by means of developping information systems Process and product quality indicators are produced and they feed Istat centralised information system for quality (SIDI/SIQual)

Documentation To keep track of data, metadata and quality indicators Support to planning surveys Support to assessing survey quality Support correct use of data

Strategy for internal documentation Concerns both metadata and standard quality indicators (process and product oriented quality indicators) Concerns all Istat surveys (direct, based on administrative surveys) and secondary studies Istat centralised information system for quality (SIDI/SIQual)

Quality Assessment Policy and Tools Recently launched activity (end 2009) Relies on the quality work done so far Improvement-oriented Strategy for systematic quality assessment Indirect quality assessment: Analyses of Standard Quality Indicators - Direct quality assessment: Auditing and Self-assessment

Indirect Quality Assessment Regular monitoring of the quality of all the processes and products to support survey managers in achieving quality targets Based on the analysis of metadata and standard quality indicators stored in the SIDI information system Report for the Istat’s top management with commented analyses of the information in SIDI (July 2011; July 2012) Level of quality reached; changes from 2001 to 2011; over the last five years; changes from the first report Focus on timeliness and accuracy (coverage and total nonresponse errors). Focus on CATI and mixed modes data collection in 2011 Information provided: average of indicators for groups of processes sharing some given characteristics; trends over time, etc

Monthly (Institutions) Example of quality report Timeliness mean value in days – active surveys by periodicity (2001-2011) timeliness Annual Quarterly Monthly (businesses) Monthly (Institutions) mean n General mean 531 70 86 13 69 15 245 8 Timeliness (days) by year and periodicity (active surveys)

Follow-up activities after first report Dissemination of quality analyses from SIDI within the Institute started in fall 2011. - Preparation of a specific report for each subject matter directorate with the same analyses of the general report but tailored to the processes of that sector Presentation and discussion of the report with the Director and the Chiefs of the Divisions within the directorate In one occasion it was decided to organize a meeting with a single Division for informing staff of the quality aspects of their surveys

Direct Quality Assessment Aimed at verifying the compliance of statistical processes to standards and Guidelines Based on a combination of internal audits and self-assessments Detailed investigation of statistical processes and their products Verify compliance to principles stated in the Istat’s Quality Guidelines

The Procedure - The survey manager fills-in Audit Self-assessment Who Team of 3 experts (auditors) Survey manager How - Study of preliminary docs (quality reports: process and products; both compiled using the data and metadata stored in the SIDI/Siqual system) - Auditors interview the survey manager using the audit questionnaire - The survey manager fills-in the self-assessment questionnaire Output Final report (3-5 pages): Part A: Results of the audit (team of auditors) Part B: Improvement actions (survey manager) Part A: Results of the self-assessment (survey manager)   revised by 1 reviewer

The Procedure Procedure and tools finalised after tests in 2010 and 2011 Period Audits Self-assessments Tot. Feb-Jun 2010 2 5 7 Oct. 2010-Apr. 2011 14 Main results of tests: Revision of questionnaires and Guidelines (seasonal adj.; revision policy; survey with admin. data) Estimation of the workload => simplification of the original self-assessment procedure Changes in the procedure, also to take into account Istat’s new organization

The Procedure Direct assessment on a regular basis, starting in 2012 January- June Running of the assessment: 5 audits and 9 self-assessments each year June – December Approaval of Improvement actions by the Quality Committee Flagging the improvement actions in the Annual Program of Activities From 2013 Starting regular follow-up of the implementation of improvement actions

Quality Assessment: Conclusions Direct assessment J goes in depth into processes J permits to identify best practices J permits to improve the single process L Workload of actors involved is substantial L Limited number of processes can be evaluated annually Indirect assessment J regular monitoring of processes and products J raises awareness of quality issues and targets J provides comparable information at low cost L does not allow to go in depth L Depends on the information in SIDI (incomplete and not updated data) Combination of the approaches permits to overcome the drawbacks of a single approach but at the price of a higher effort

Communicating Quality Recently launched activity for a structured and regular communication of quality aspects to users New section on Istat website Released in 2011 http://www.istat.it/en/about-istat/quality Istat Quality Policy Work done Codes of Statitistics Tools Links to documents Link to SIQual

Final remarks Quality is a long term commitment and shpuld be seen as an investment Istat is collecting the results of past investments However we are aware that there is still a lot of work to do Future plans: To consolidate what has been recently launched To start disseminate in a systematic way quality indicators

References Anitori P., Brancato G., Gismondi R., Signore M. (2000), "The effects of Late Respondents on Retail Trade Monthly Indexes", Proceedings of the International Conference on Establishment Surveys-II, Survey Methods for Business, Farms and Institutions, 17-21 giugno, Buffalo, USA, CD-ROM. Barbieri G e Signore M. (1999), "Statistical research activities at Istat (Statistics Italy)", Research in Official Statistics, Vol. n. 1, pp. 109-114. Bocci L., Muratore M.G., Signore M., Tagliacozzo G. (2002), “The interviewer effect on the Data Collection of Sensitive Questions”, Atti della XLI Riunione Scientifica della SIS, Sessione Data Quality and accuracy, pp299-302 Brancato G., De Angelis R., Mazza L., Signore M. (1998), "Feasibility and effectivness of CATI follow-up in ISTAT Small Business Survey" Proceedings of the 3rd International Conference on Methodological Issues in Official Statistics, Stoccolma, 12-13 ottobre, pp. 151-154 Brancato G, D’Orazio M, Fortini M. (2004) “Response error estimation in presence of Record Linkage error: the case of the Italian Population Census”. Proceedings of the European Conference on Quality and Methodology in Official Statistics (Q2004). Mainz, 24-26 May 2004. CD ROM Brancato G., Corazziari I., Dattilo B., Di Filippo P., Muratore M.G., Perez M., Simeoni G. (2005) “What are we missing? The effects on the estimations of no phone households in Italy” Paper presentato alla Second International Conference on Telephone Survey Methodology. Florida, January 2006. Brancato G., Pellegrini C., Signore M., Simeoni G., (2004) “Standardising, Evaluating and Documenting Quality: the Implementation of Istat Information System for Survey Documentation – SIDI”, International Conference on Methodologies and Quality in Statistics, Mainz

References Signore M., Brancato G., Simeoni G. (2008), “Multi-mode data collection: What can still be expected?”, invited paper, Proceedings of Statistics Canada Symposium 2008, Data Collection: Challenges, Achievements and New Directions. Signore M., D’Orazio M., Carbini R. (2012), “Quality Assessment at Istat:The Combined Use of Standard Quality Indicator Analysis and Audit Procedures”, European Conference on Quality in Official Statistics (Q2012), Athens