First results from the in-depth surveys on quality assurance frameworks and quality reporting Conference on Data Quality for International Organisations.

Slides:



Advertisements
Similar presentations
Quality Improvement in the ONS Cynthia Z F Clark Frank Nolan Office for National Statistics United Kingdom.
Advertisements

TEN-T Info Day for AP and MAP Calls 2012 EVALUATION PROCESS AND AWARD CRITERIA Anna Livieratou-Toll TEN-T Executive Agency Senior Policy & Programme Coordinator.
Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November.
Enhancing Data Quality of Distributive Trade Statistics Workshop for African countries on the Implementation of International Recommendations for Distributive.
The quality framework of European statistics by the ESCB Quality Conference Vienna, 3 June 2014 Aurel Schubert 1) European Central Bank 1) This presentation.
Quality evaluation and improvement for Internal Audit
1 The South African Statistical Quality Assessment Framework (SASQAF) Presentation made at the Conference on Data Quality for International Organisations.
Recent international developments in Energy Statistics United Nations Statistics Division International Workshop on Energy Statistics September 2012,
United Nations Economic Commission for Europe Statistical Division Applying the GSBPM to Business Register Management Steven Vale UNECE
The use and convergence of quality assurance frameworks for international and supranational organisations compiling statistics The European Conference.
United Nations Statistics Division
European Conference on Quality in Official statistics, Rome 8-11 July 2008 Quality framework in European Trade Statistics Anne Berthomieu International.
European Conference on Quality in Official Statistics (Q2010) 4-6 May 2010, Helsinki, Finland Brancato G., Carbini R., Murgia M., Simeoni G. Istat, Italian.
Quality assurance activities at EUROSTAT CCSA Conference Helsinki, 6-7 May 2010 Martina Hahn, Eurostat.
Integration Development Programme in the Field of Statistics of the Eurasian Economic Union for EEC THE EURASIAN ECONOMIC COMMISSION.
8-11-Jul-07 How to increase quality of Principal European Economic Indicators? Roberto Barcellan, Brian Newson, Klaus Wurm Eurostat.
REFERENCE METADATA FOR DATA TEMPLATE Ales Capek EUROSTAT.
Marina Signore Head of Service “Audit for Quality Istat Assessing Quality through Auditing and Self-Assessment Signore M., Carbini R., D’Orazio M., Brancato.
Initial thoughts on a Global Strategy for the Implementation of the SEEA Central Framework Ivo Havinga United Nations Statistics Division.
Essential SNA Project being developed from 2011 to 2013.
Quality assurance / Quality control system for the Greek GHG emissions inventory Yannis Sarafidis, Elena Georgopoulou UNFCCC Workshop on National Systems.
The ECB Statistical Quality Framework and Quality Assurance Procedures: An assessment in the light of the attempt to harmonise frameworks of international.
CZECH STATISTICAL OFFICE 1 The Quality Metadata System In the Czech Statistical Office Work Session on Statistical Metadata (METIS)
Assessing the Capacity of Statistical Systems Development Data Group.
CZECH STATISTICAL OFFICE Na padesátém 81, CZ Praha 10, Czech Republic 1 Subsystem QUALITY in Statistical Information System Czech.
process information Coordination of National Statistical Systems Seminar on the Implementation of Fundamental Principles Konrad Pesendorfer.
Chapter 9: Data quality and metadata Ilaria DiMatteo United Nations Statistics Division The 4 th meeting of the Oslo Group on energy statistics Ottawa,
Introduction 1. Purpose of the Chapter 2. Institutional arrangements Country Practices 3. Legal framework Country Practices 4. Preliminary conclusions.
European Conference on Quality in Official Statistics 8-11 July 2008 Mr. Hing-Wang Fung Census and Statistics Department Hong Kong, China (
United Nations Statistics Division Work Programme on Economic Census Vladimir Markhonko, Chief Trade Statistics Branch, UNSD Youlia Antonova, Senior Statistician,
CARICOM. January 30- February 1, 2013 Kingston, Jamaica REGIONAL SEMINAR ON DEVELOPING A PROGRAMME for the IMPLEMENTATION of the SYSTEM OF NATIONAL ACCOUNTS.
Implementation of the European Statistics Code of Practice Yalta September 2009 Pieter Everaers, Eurostat.
Eurostat/UNSD Conference on International Outreach and Coordination in National Accounts for Sustainable Development and Growth 6-8 May, Luxembourg These.
African Centre for Statistics United Nations Economic Commission for Africa Data compilation and management proposal for Africa Statistical Commission.
Consultant Advance Research Team. Outline UNDERSTANDING M&E DATA NEEDS PEOPLE, PARTNERSHIP AND PLANNING 1.Organizational structures with HIV M&E functions.
United Nations Oslo City Group on Energy Statistics OG7, Helsinki, Finland October 2012 ESCM Chapter 8: Data Quality and Meta Data 1.
Quality Frameworks: Implementation and Impact Notes by Michael Colledge.
UNSD Recent international developments in Energy Statistics.
General Recommendations on STS Carsten Boldsen Hansen Economic Statistics Section, UNECE UNECE Workshop on Short-Term Statistics (STS) and Seasonal Adjustment.
The implementation programme for the 2008 SNA and supporting statistics UNSD-Regional Commissions Coordination Meeting on Integrated Economic Statistics.
Report on the breakout session on Rapid Estimates Roberto Barcellan European Commission - Eurostat.
Copyright 2010, The World Bank Group. All Rights Reserved. Principles, criteria and methods Part 1 Quality management Produced in Collaboration between.
Quality at a Glance: Documentation of Quality Indicators at Statistics Austria European Conference on Quality in Official Statistics Rome, 8-11 July 2008.
13 November, 2014 Seminar on Quality Reports QUALITY REPORTS EXPERIENCE OF STATISTICS LITHUANIA Nadiežda Alejeva Head, Price Statistics.
A Training Course for the Analysis and Reporting of Data from Education Management Information Systems (EMIS)
Relationship between Short-term Economic Statistics Expert Group Meeting on Short-Term Statistics February 2016 Amman, Jordan.
1 Recent developments in quality matters in the ESS High level seminar for Eastern Europe, Caucasus and Central Asia countries Claudia Junker, Eurostat,
1 Recent developments in quality related matters in the ESS High level seminar for Eastern Europe, Caucasus and Central Asia countries Claudia Junker,
United Nations Statistics Division Developing a short-term statistics implementation programme Expert Group Meeting on Short-Term Economic Statistics in.
Maria João Zilhão Planning and Quality Control Unit « High Level Seminar “Quality Matters in Statistics” June, Athens, Greece Implementation of the.
4–6 September 2013, Vilnius, Lithuania High-Level Seminar for Eastern Europe, Caucasus and Central Asia Countries QUALITY FRAMEWORK AT.
21 June 2011 High level seminar for EECCA on “Quality matters in statistics” High level seminar for EECCA on “Quality matters in statistics” The Code of.
Eurostat Quality reporting on energy statistics Framework and experience at EU level United Nations Oslo Group on Energy Statistics Aguascalientes (Mexico),
United Nations Statistics Division
Gerhardt Bouwer Statistics South Africa
Quality assurance in official statistics
4.1. Data Quality 1.
Measuring Data Quality and Compilation of Metadata
Overview of the ESS quality framework and context
Quality assessment ESTP Training Course “Quality Management and survey Quality Measurement” Rome, 24 – 27 September 2013 Giorgia Simeoni Researcher Unit.
Assessment of Quality in Statistics GLOBAL ASSESSMENTS, PEER REVIEWS AND SECTOR REVIEWS IN THE ENLARGEMENT AND ENP COUNTRIES Mirela Kadic, Project Manager.
Sub-Regional Workshop on International Merchandise Trade Statistics Compilation and Export and Import Unit Value Indices 21 – 25 November Guam.
Energy Statistics Compilers Manual
Palestinian Central Bureau of Statistics
The European Statistics Code of Practice - a Basis for Eurostat’s Quality Assurance Framework Marie Bohatá Deputy Director General, Eurostat ... Strategic.
United Nations Statistics Division
2.7 Annex 3 – Quality reports
European Statistics Code of Practice
Overview of the ESS quality framework and context
ESS conceptual standards for quality reporting
Presentation transcript:

First results from the in-depth surveys on quality assurance frameworks and quality reporting Conference on Data Quality for International Organisations April 2006, Newport (UK) Håkan Linden, Statistical Governance, Quality and Evaluation European Commission, Eurostat

Background Key areas of work in the Eurostat project on “the use and convergence of international quality assurance frameworks” sponsored by the CCSA. Problem statement – quality assurance frameworks: - lack of use of quality assurance frameworks in most international organisations - existence of different quality frameworks and benchmarking activities for national statistics reported Problem statement – quality reporting: - existence of different tools and practices for collecting quality metadata from the data providers - existence of different formats to inform the users on the quality of statistics

Overview in-depth surveys Questionnaire templates developed by the task-teams for structuring the work and collecting the state-of-art on quality frameworks and quality reporting activities. All members of the CCSA included in the surveys Data collection: 20 February 2006 – 31 March international organisations replied – thank you! First results for this conference Report with detailed results and recommendations to be prepared for the 2 nd CCSA meeting 2006

Quality frameworks - the template Section 6. Quality Assurance Frameworks in place - brief description of the quality assurance framework - legal basis - last update - coverage of institutional environment, statistical processes and outputs - quality definition - quality requirements/ targets - procedures for evaluation of the adherence to the quality framework - main strengths and weaknesses of the quality framework Section 7. If a quality framework not yet in place - brief description of the current situation of the quality work - how weaknesses are identified, impact assessed and improvement actions - main obstacles for development and implementation of a quality framework Section 8. For all international organisations - the future plans for quality improvements - additional comments/ recommendations etc. for this project

Quality frameworks - results 5 (out of 12) organisations have quality assurance frameworks in place 1 organisation is currently developing an encompassing quality assurance framework 6 organisations have not yet begun to develop formalised quality assurance frameworks

Quality frameworks - in place “Legal” basis: - endorsed by (internal) Statistical Policy Group and Statistical Committee - Data Quality Standard under development - Article - Policy document - Adherence to the “Principles Governing Statistical Activities” - Recommendation Last update: 2003 (new update in 2006); May 2004; July 2003; May 2005; 2003.

Quality frameworks - in place Coverage of institutional environment/ settings: - all statistical activities/ major activities reviewed every 5- years - statistical data under “direct coordination” but extension of the coverage planned - 2 organisations actually cover the institutional settings!

Quality frameworks - in place Coverage of statistical processes: - definition of data requirements - evaluation of other data currently available - statistical design and planning - data and metadata collection - data and metadata processing - compilation and estimation - disclosure control - data and metadata dissemination

Quality frameworks - in place Coverage of statistical output: - all - limited to a few domains - economic monetary statistics

Quality frameworks - in place Quality requirements/ targets: (i) Institutional environment/ settings: - benchmarks for observation of international good practice or standard with reference to for example the UN Fundamental Principles of Official Statistics, the UN Handbook of Statistical Organization, and the Quality Declaration for the ESS (ii) Statistical processes: - a set of best practices is defined for each process. Each activity must be compared with these practices - benchmarks … - standardisation and harmonisation of tools, reports and development of policies (like revision policy) (iii) Statistical outputs: - a set of best practices is defined for each process. Each activity must be compared with these practices - punctuality (annually agreed timetables) - regulations

Quality frameworks - in place Procedures in place for the evaluation of the adherence to the framework: (i) Institutional environment/ settings: -surveillance missions may help to identify institutional and legal issues underlying problems in data quality -audit exercises -self-assessments -peer reviews -annual report on the implementation and protection of confidential information (ii) Statistical processes: - quality reviews of ongoing activities on a 5 year rolling basis (self-evaluation of strengths and weaknesses by comparing with best-practices) (iii) Statistical outputs: - compliance reports - regular reports on gaps in statistics (as input for medium-term work-pgm) - quality reports/ quality profiles

Quality frameworks - in place Strengths: - Quality reviews (self-evaluation of strengths and weaknesses by responsible persons, comparing their activities with best-practices) on a rolling basis give a good framework for discussing quality problems between domain managers and statistics and IT- experts - Data quality monitoring at each stage of a statistical production process makes it possible to identify and address data quality problems. Incorporated “feed-back loops” serve as a mechanism for improving data quality - The implementation of quality framework (phased/ staggered approach) ease the development of new work processes related to data collection and processing, IT, and data dissemination - Existence of transparent and comprehensive legislative framework for the production of statistics ease the monitoring of data quality - Quality manager appointed to coordinate all quality work

Quality frameworks – in place Weaknesses: - The resources allocated to corporate quality work are insufficient - The resources to help solve quality problems are scarce - The implementation of process-oriented data quality management requires strong co-ordination and support (IT and statistics) - Lack of standardisation of data treatment procedures and insufficient documentation of methods

Quality frameworks – in place The main constraints on reaching optimal data quality in the statistical processes: - The resources - Difficult to apply the same quality concept (e.g. dimensions) for less developed statistical systems as for more developed systems - The improvement of data quality is viewed as a gradual process that need to take into account resources constraints and establish priorities - No direct contact with respondents - The quality of the country data

Quality frameworks - not (yet) in place The current quality work: - implementation of new Statistical Information System (data validation/ consistency checks and metadata management) - procedures for validation of statistics/ estimations by countries before publication - regular review of data collection activities - internal quality reports - methodological aspects are documented and used to identify best practices - project on “process documentation” to identify weaknesses and define best practices

Quality frameworks - not (yet) in place Identification of weaknesses: - manual data clearance procedures are well functioning but proves to be time consuming and intervenes at a late stage in report preparation - data weaknesses are identified when in-house economists draft their analytical reports based on the statistics - data weaknesses are identified on the basis of metadata available and by comparisons with similar statistics from other organisations - the impact of data weaknesses is assessed using statistical “judgement”, improvement actions are discussed between statistician(s) and the supervisor and implemented according to agreed plans - weaknesses in data are pointed out by the users (user feed-back or user surveys)

Quality frameworks - not (yet) in place Main obstacles/ critical issues for implementation of quality assurance framework: - economists and analysts perceive data quality requirements too excessive and seen as extreme refinements imposed by statisticians - relevant IT tools should support the quality framework to efficient process of data and quality control (QF might be seen as a concept impossible to apply) -time and resources -A well-designed quality framework for organisations compiling statistics from different international organisations would ease the implementation

Quality frameworks – future plans (1) - review of existing quality framework - inform senior managers on data quality issues and principles governing international statistics - training of staff on quality issues and data processing - data collection by electronic means will increase and make it possible to implement more automatic data validation - SDMX is expected to be a mean for accessing data and metadata from countries and other international organisations - improved accessibility to data for users - adapt the common quality assurance framework that will be proposed by CCSA - establishment of a statistical data quality group

Quality frameworks – future plans (2) - promoting data and metadata dissemination - regular monitoring, assessment and reporting based on the statistical data quality framework - peer reviews of member countries

Quality reporting – the template Section 6. Metadata about quality collected from data providers - brief description of the quality reporting activties - legal basis - quality dimensions covered - frequency - procedures/ mechanisms in place for quailty reporting - type of quality information (qualitative/ quantitative) - the use of the quality information - strengths and weaknesses of the quality reporting - quality constraints - quailty improvements Section 7. Metadata about the quality of released statistics - brief description of the ways of informing the users on the quality - based on standardised reference metadata - type of quality information disseminated (qualitative, quantitative) - strengths and weaknesses of the system of informing users on quality Section 8. For all international organisations - the future plans for improving quality reporting - additional comments/ recommendations etc. for this project

Quality reporting - results “Legal” basis: - Guidelines for good practices - Ratification of Convention for data reporting and its implementation - indirect through Council decision - Standards - Regulations - Recommendation

Quality reporting - results Coverage of quality dimensions (1): - relevance, accuracy, credibility, timeliness, accessibility, interpretability, coherence - relevance, accuracy, timeliness, punctuality, accessibility, clarity, comparbility, coherence, completeness, and sound metadata - prerequisites of quality, assurances of integrity, methological soundness, accuracy and reliability, servicability, accessibility - methodological soundness, accuracy, reliability, consistency, timeliness, and punctuality - relevance, accuracy, timeliness, punctuality, accessibility, clarity, comparability, and coherence

Quality reporting - results Coverage of quality dimensions (2): - relevance, accuracy, interpretability, and coherence - completeness, accuracy, reliability, comparability, adherence to standards - data accuracy, comparability and consistency - data consistency (aggregation) and comparability with standards - methodological soundness (coverage) and conistency over time

Quality reporting - results Frequency of the quality reporting: - no common rules - rolling programme - once a year - approx. every 5 years - twice a year - each time data submitted/ collected - quarterly - according to requirements in Regulations - as part of projects on country assistance/ country reviews

Quality reporting - results Procedures/ mechanisms in place for the quality reporting: - the quality framework provides the theoretical and practical guidance - metadata questionnaire attached to statistical data questionnaire - manuals/ guidelines/ handbooks - clerical and computerised edits - standardised reporting forms for metadata - glossary on quality - bulletin board on data collection, dissmeination and quality of statistics - standardised data quality assessment framework tool (part of formalised data quality progam) for assessing member countries data quality - regulations - metadata common vocabulary

Quality reporting - results Type of quality information collected: - almost all information provided are qualitative - very few quantitative measures, like: - response rates - data completeness - data timeliness - revisions binary indicators - set of standard quality indicators - overall assessments: - allocation of “colour-codes” to country data - quality ratings (four grade scale)

Quality reporting - results Strengths: - decentralised meaning that activity managers can collect the metadata - metadata reporting by NSO’s are done in a systematic manner - allows checking compliance against international standards - a tool for monitoring where there are trade-offs between data quality problems and allocation of resources - standard framework for metadata in general and metadata about quality in place covering many countries - good cooperation with data providers - quality criteria and quality reporting requirements included in legislations for member countries

Quality reporting - results Weaknesses: - un-coordinated - too decentralised (no coordination) - no standards/ standardised reporting - no central monitoring - needs further embedding in overall culture of the organisation - reporting NSO’s not aware of international standard methodology, concepts, definitions and classifications - no standardisation of metadata and rules to be followed - metadata are not processed automatically - infrequent collection of information - burden on countries - lack of resources - self-assessments of countries not yet done - not all statistics yet covered

Quality reporting - results Coverage of quality constraints: - none (9 out of 12) - data sharing and coordination among data producing agencies - confidentiality - resources (financial, staff, technical) and efficient use - planning of statistical programs - covered by a special “merit and costs” procedure - burden on respondents

Quality reporting - results Coverage of quality improvement actions: - no (8 out of 12) - sometimes - explicit sections on “plans for improvement” and “specific recommendations for improvement”

Quality reporting - results Future plans for improving quality reporting from data providers: - continued use of electronic dissemination - investigation on the use of SDMX standards - development of a metadatabase that allows automatic handling of metadata for the processing, tabulation and dissemination - revision of the quality reporting framework in order to improve standardisation - data quality assessment framework is being incorporated into the data dissemination standards - development of a coherent quality framework with the aim to introduce a harmonised definition of quality - workshops and seminars for countries on strengthening the capacity to produce data

Quality information for the users Ways of informing the users on the quality: - quality metadata is embedded in the dissemination system - attached notes to datasets and tables - metadata included in “Sources and Methods” publications - standardised “country notes” on methods and definitions for each release - Data Dissemination Standard(s) - dissemination of data only possible if metadata are filled in - annual quality report published summarising all economic statistics - annual internal quality report for each statistical area - quality profiles for key statistical indicators - domain specific quality reports

Quality information for the users Future plans for improving quality information about the published statistics: - implementation of new Statistical Information System designed to fully support metadata (harmonisation and automatisation) - more use of pre-coded response for metadata to improve standardisation - adoption of systematic criteria is under consideration (dependent on international work) - coverage of all statistical domains - data quality requirements to be better integrated in the statistical database dissemination system