Lecture 1: Definition of quality in statistics Outi Ahti-Miettinen Statistics Finland ESTP course on Quality Management in Statistical Agencies – Introductory course Statistics Finland, Helsinki 24-27 April, 2018 THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

Contents of the session Definition of quality in statistics Definition of quality Brief history of quality management Quality in statistics production Principles of GSBPM Generic statistical business process model

Definition of Quality -1 What is good quality? Often used phrases: ”Conformance with requirements/standards” ”Fit for purpose” - products in use ”Zero defects” - functions properly Perspectives: Production - control & monitoring Supply & marketing - production for use Customer - justifies the quality ISO definition for quality: ”Degree to which a set of inherent characteristics fulfills requirements”

Definition of Quality -2

Brief history of quality management -1 Starting point: Industrial revolution Introduction of mass production - uprise in early 20th century Manufacturing: The end to the old team work Machines and their users tiny parts in production Need to standardize the work process: Production time important in mass production Quality control and inspection Quality systems

Brief history of quality management -2 Early advocates: organization of work & organization theories Winslow Taylor and Henry Ford Later famous statisticians: Walter A. Shewhart and W. Edwards Deming 1891-1967 1900-1993

Brief history of quality management -3 Shewhart initiated Control charts Advocated the use of measurements Developed statistical theory to process control Deming had a crucial role in developing systematic production quality measurement Theory: Plan-Do-Check-Act cycle (Originally by Shewhart…); 14 points; Seven deadly diseases etc. Many applications in industry Eye on enterprise management

Brief history of quality management -4

What is quality management? Improving your organisation Identify good practices –> Retain Identify bad practices –> Replace Support of the top management Measuring results Improving processes, methods and practices

Continuous Quality Improvement Plan-Do-Check-Act cycle

Quality in statistics production -1 Timeline over 100 years 1890s: Herman Hollerith invented the punch card tabulation machine while working at the US Census Bureau. It was a starting point for development in statistical computing… like any industrial process 1920s and 1930s: statistical quality control theories (Shewhart, Deming, Dodge, Roming…) 1930s and 1940s: data process of censuses and sample surveys 1950: first UN recommendations ”The Preparation of Sampling Survey Reports”

Quality in statistics production -2 1980s and 1990s: general awareness of quality Statistics Canada: Quality Guidelines (1985) Statistical agencies develop their own policies on quality US: Federal Committee On Measuring and Reporting the Quality of Survey Data Quality policies in International organisations: IMF, OECD Europe: Regulations and agreements on quality reporting WG on Assessment of Quality in Statistics since 1998 LEG on Quality, 1999-2001

Quality in statistics production -3 And from 2000: The European Statistics Code of Practice (2005, rev 2011) Regulation on European Statistics No 223/2009 so called ’Statistical Law’ (revised 05/2015) The Eurostat Quality Assurance Framework 2011 Implementation of the Fundamental Principles of Official Statistics (UN, 2003) Declaration of Good Practices in Technical Cooperation in Statistics (UN, 1999) Data Quality Assessment Framework - A Factsheet, Statistics Department DQAF (2006, IMF) The Eurostat Quality Assurance Framework 2012 Statistical Data Quality in the UNECE (2010), Steven Vale

Quality in statistics production -4 Development in the European Statistical System (ESS): the mission, vision and values Code of Practice & European Statistical Law the ESS quality dimensions Standard quality indicators Quality assurance framework & tools Quality assessment plan Quality declaration

Quality of European statistics

ESS Code of Practice (revised 11/2017) Institutional environment Professional independence Mandate for data collection and Access to Data Adequacy of resources Commitment to quality Statistical confidentiality and Data protections Impartiality and objectivity Statistical processes Sound methodology Appropriate statistical procedures Non-excessive burden on respondents Cost-effectiveness Statistical output Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility 1bis Coordination and cooperation

OECD Good statistical practice (11/2015) Put in place a clear legal and institutional framework for official statistics Ensure professional independence of National Statistical Authorities Ensure adequacy of human financial and technical resources Protect the privacy of data providers Ensure the right to access administrative sources Ensure the impartiality, objectivity and transparency Employ sound methodology and commit to professional standards Commit to the quality of statistical outputs and processes Ensure user-friendly data access and dissemination Establish responsibilities for co-ordination of statistical activities Commit to international co-operation Encourage exploring innovative methods

OECD Good Practices based on ES CoP Example: ES CoP Principle 1: Professional Independence Indicator 1.7: The National Statistical Institute and Eurostat and, where appropriate, other statistical authorities, comment publicly on statistical issues, including criticisms and misuses of statistics as far as considered suitable. OECD Recommendation 2: Ensure professional independence of National Statistical Authorities. Indicator 2.10: The NSO and where appropriate, other National Statistical Authorities, comment publicly on statistical issues, including criticisms and misuses of statistics as far as considered suitable (ECoP).

Example: Quality Criteria of Official Statistics of Finland vs Example: Quality Criteria of Official Statistics of Finland vs. ES CoP Principles Professional independence Mandate for data collection Adequacy of resources Commitment to quality Statistical confidentiality Impartiality and objectivity Sound methodology Appropriate statistical procedures Non-excessive burden on respond Cost-effectiveness Relevance Accuracy and reliability Timeliness and punctuality Coherence and comparability Accessibility OSF Criteria: 1. Impartiality and transparency 2. Quality control 3. Confidentiality 4. Efficiency 5. Relevance 6. Accuracy and reliability 7. Timeliness and punctuality 8. Coherence and comparability 9. Accessibility and clarity 19

Principles of GSBPM -1 Generic Statistical Business Process Model The GSBPM provides a basis for statistical organizations to agree on standard terminology to develop statistical metadata systems and processes A flexible tool to describe and define the set of business processes needed to produce official statistics

Principles of GSBPM -2 The GSBPM applies to all activities undertaken by producers of official statistics, at both the national and international levels, which result in data outputs. It is independent of the data source, so it can be used for the description and quality assessment of processes based on surveys, censuses, administrative records, and other non-statistical or mixed sources.

Describing a process v5.0 Archive (Phase 8, v4.0) has been incorporated into the over-arching process of data and metadata management.

GSBPM: Sub-processes with interest group contacts Quality Management / Metadata Management 1 Specify Needs 2 Design 3 Build 4 Collect 5 Process 6 Analyse 7 Disseminate 1.1 Identify needs 1.2 Consult & confirm need 1.3 Establish output objectives 1.5 Check data availability 1.6 Prepare business case 2.1 outputs 2.2 Design variable descriptions 2.4 Design frame & sample 2.5 Design processing & analysis 2.6 Design production systems & workflow 4.1 Create frame & select sample 4.2 Set up collection 4.3 Run collection 4.4 Finalise collection 5.1 Integrate data 5.2 Classify & code 5.3 Review & validate 5.5 Derive new variables & units 5.7 Calculate aggregates 6.1 Prepare draft outputs 6.2 Validate outputs 6.3 Interpret & explain outputs 6.4 Apply disclosure control 6.5 Finalise outputs 7.1 Update output systems 7.2 Produce dissemination products 7.3 Manage release of dissemination products 7.5 Manage user queries 7.4 Promote dissemination products 5.6 Calculate weights 2.3 Design collection 8 Evaluate 8.1 Gather evaluation inputs 8.2 Conduct evaluation 8.3 Agree an action plan 1.4 Identify concepts 3.6 Test statistical business process 3.2 Build or enhance process components 3.4 Configure workflows 3.5 Test production system 3.1 Build collection instrument 5.4 Edit & impute 5.8 Finalise data files Main Consult with users needs Consult with research 3.3 Build or enhance dissemination components Process owner

How to get started on GSBPM ? UNECE: http://www1.unece.org/stat/platform/display/metis/The+Generic+Statistical+Business+Process+Model

References Eurostat: Quality of statistics http://ec.europa.eu/eurostat/web/quality/overview Eurostat: Quality tools and standards http://ec.europa.eu/eurostat/web/quality/quality-reporting UNECE GSBPM: http://www1.unece.org/stat/platform/display/metis/The+Generic+Statistical+Business+Process+Model UNECE Statistics: http://www.unece.org/stats/stats_h.html IMF DQAF: http://dsbb.imf.org/Pages/DQRS/DQAF.aspx Quality Framework for OECD Statistical Activities: http://www.oecd.org/std/qualityframeworkforoecdstatisticalactivities.htm