1. Lecture 1: Definition of quality in statistics
Outi Ahti-Miettinen Statistics Finland
ESTP course on Quality Management in Statistical Agencies – Introductory course
Statistics Finland, Helsinki April, 2018
THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

2. Contents of the session
Definition of quality in statistics
Definition of quality
Brief history of quality management
Quality in statistics production
Principles of GSBPM Generic statistical business process model

3. Definition of Quality -1
What is good quality?
Often used phrases:
”Conformance with requirements/standards”
”Fit for purpose” - products in use
”Zero defects” - functions properly
Perspectives:
Production - control & monitoring
Supply & marketing - production for use
Customer - justifies the quality
ISO definition for quality: ”Degree to which a set of inherent characteristics fulfills requirements”

4. Definition of Quality -2

5. Brief history of quality management -1
Starting point:
Industrial revolution
Introduction of mass production - uprise in early 20th century
Manufacturing:
The end to the old team work
Machines and their users tiny parts in production
Need to standardize the work process:
Production time important in mass production
Quality control and inspection
Quality systems

6. Brief history of quality management -2
Early advocates:
organization of work & organization theories
Winslow Taylor and Henry Ford
Later famous statisticians:
Walter A. Shewhart and W. Edwards Deming

7. Brief history of quality management -3
Shewhart initiated
Control charts
Advocated the use of measurements
Developed statistical theory to process control
Deming had a crucial role in developing systematic production quality measurement
Theory: Plan-Do-Check-Act cycle (Originally by Shewhart…); 14 points; Seven deadly diseases etc.
Many applications in industry
Eye on enterprise management

8. Brief history of quality management -4

9. What is quality management?
Improving your organisation
Identify good practices –> Retain
Identify bad practices –> Replace
Support of the top management
Measuring results
Improving processes, methods and practices

10. Continuous Quality Improvement Plan-Do-Check-Act cycle

11. Quality in statistics production -1
Timeline over 100 years
1890s: Herman Hollerith invented the punch card tabulation machine while working at the US Census Bureau. It was a starting point for development in statistical computing… like any industrial process
1920s and 1930s: statistical quality control theories (Shewhart, Deming, Dodge, Roming…)
1930s and 1940s: data process of censuses and sample surveys
1950: first UN recommendations ”The Preparation of Sampling Survey Reports”

12. Quality in statistics production -2
1980s and 1990s: general awareness of quality
Statistics Canada: Quality Guidelines (1985)
Statistical agencies develop their own policies on quality
US: Federal Committee On Measuring and Reporting the Quality of Survey Data
Quality policies in International organisations: IMF, OECD
Europe: Regulations and agreements on quality reporting
WG on Assessment of Quality in Statistics since 1998
LEG on Quality,

13. Quality in statistics production -3
And from 2000:
The European Statistics Code of Practice (2005, rev 2011)
Regulation on European Statistics No 223/2009 so called ’Statistical Law’ (revised 05/2015)
The Eurostat Quality Assurance Framework 2011
Implementation of the Fundamental Principles of Official Statistics (UN, 2003)
Declaration of Good Practices in Technical Cooperation in Statistics (UN, 1999)
Data Quality Assessment Framework - A Factsheet, Statistics Department DQAF (2006, IMF)
The Eurostat Quality Assurance Framework 2012
Statistical Data Quality in the UNECE (2010), Steven Vale

14. Quality in statistics production -4
Development in the European Statistical System (ESS):
the mission, vision and values
Code of Practice & European Statistical Law
the ESS quality dimensions
Standard quality indicators
Quality assurance framework & tools
Quality assessment plan
Quality declaration

15. Quality of European statistics

16. ESS Code of Practice (revised 11/2017)
Institutional environment
Professional independence
Mandate for data collection and Access to Data
Adequacy of resources
Commitment to quality
Statistical confidentiality and Data protections
Impartiality and objectivity
Statistical processes
Sound methodology
Appropriate statistical procedures
Non-excessive burden on respondents
Cost-effectiveness
Statistical output
Relevance
Accuracy and reliability
Timeliness and punctuality
Coherence and comparability
Accessibility
1bis Coordination
and cooperation

17. OECD Good statistical practice (11/2015)
Put in place a clear legal and institutional framework for official statistics
Ensure professional independence of National Statistical Authorities
Ensure adequacy of human financial and technical resources
Protect the privacy of data providers
Ensure the right to access administrative sources
Ensure the impartiality, objectivity and transparency
Employ sound methodology and commit to professional standards
Commit to the quality of statistical outputs and processes
Ensure user-friendly data access and dissemination
Establish responsibilities for co-ordination of statistical activities
Commit to international co-operation
Encourage exploring innovative methods

18. OECD Good Practices based on ES CoP
Example: ES CoP Principle 1: Professional Independence Indicator 1.7: The National Statistical Institute and Eurostat and, where appropriate, other statistical authorities, comment publicly on statistical issues, including criticisms and misuses of statistics as far as considered suitable. OECD Recommendation 2: Ensure professional independence of National Statistical Authorities. Indicator 2.10: The NSO and where appropriate, other National Statistical Authorities, comment publicly on statistical issues, including criticisms and misuses of statistics as far as considered suitable (ECoP).

19. Example: Quality Criteria of Official Statistics of Finland vs
Example: Quality Criteria of Official Statistics of Finland vs. ES CoP Principles
Professional independence
Mandate for data collection
Adequacy of resources
Commitment to quality
Statistical confidentiality
Impartiality and objectivity
Sound methodology
Appropriate statistical procedures
Non-excessive burden on respond
Cost-effectiveness
Relevance
Accuracy and reliability
Timeliness and punctuality
Coherence and comparability
Accessibility
OSF Criteria: 1. Impartiality and transparency 2. Quality control 3. Confidentiality 4. Efficiency 5. Relevance 6. Accuracy and reliability 7. Timeliness and punctuality 8. Coherence and comparability 9. Accessibility and clarity 19

20. Principles of GSBPM -1 Generic Statistical Business Process Model
The GSBPM provides a basis for statistical organizations to agree on standard terminology to develop statistical metadata systems and processes
A flexible tool to describe and define the set of business processes needed to produce official statistics

21. Principles of GSBPM -2
The GSBPM applies to all activities undertaken by producers of official statistics, at both the national and international levels, which result in data outputs.
It is independent of the data source, so it can be used for the description and quality assessment of processes based on surveys, censuses, administrative records, and other non-statistical or mixed sources.

22. Describing a process
v5.0
Archive (Phase 8, v4.0) has been incorporated into the over-arching process of data and metadata management.

23. GSBPM: Sub-processes with interest group contacts
Quality Management / Metadata Management 1
Specify Needs 2
Design 3
Build 4
Collect 5
Process 6
Analyse 7
Disseminate
1.1
Identify needs
1.2
Consult & confirm need
1.3
Establish output objectives
1.5
Check data availability
1.6
Prepare business case
2.1
outputs
2.2
Design variable descriptions
2.4
Design frame & sample
2.5
Design processing &
analysis
2.6
Design production systems & workflow
4.1
Create frame & select sample
4.2
Set up collection
4.3
Run
collection
4.4
Finalise collection
5.1
Integrate data
5.2
Classify & code
5.3
Review & validate
5.5
Derive new variables & units
5.7
Calculate aggregates
6.1
Prepare draft outputs
6.2
Validate outputs
6.3
Interpret & explain outputs
6.4
Apply disclosure control
6.5
Finalise outputs
7.1
Update output systems
7.2
Produce dissemination products
7.3
Manage release of dissemination products
7.5
Manage user queries
7.4
Promote dissemination products
5.6
Calculate weights
2.3
Design collection 8
Evaluate
8.1
Gather evaluation inputs
8.2
Conduct evaluation
8.3
Agree an action plan
1.4
Identify concepts
3.6
Test
statistical business process
3.2
Build or enhance process components
3.4
Configure workflows
3.5
Test production system
3.1
Build collection instrument
5.4
Edit & impute
5.8
Finalise data files
Main
Consult with users needs
Consult with research
3.3
Build or enhance dissemination components
Process owner

24. How to get started on GSBPM ?
UNECE:

25. References
Eurostat: Quality of statistics
Eurostat: Quality tools and standards
UNECE GSBPM:
UNECE Statistics:
IMF DQAF:
Quality Framework for OECD Statistical Activities: