1. Quality Management Frameworks
ESTP Training Course “Quality Management and survey Quality Measurement” Rome, 24 – 27 September 2013
Marina Signore
Director of Research
Chief, Division "Metadata, Quality and R&D Projects“
Istat
Maria João Zilhão
Unit Director
Planning, Control and Quality Unit
Statistics Portugal

2. Quality Assurance Framework
General definition
Quality assurance is referred to as “the part of quality management focusing on proving confidence that quality requirements will be fulfilled (ISO 9000)
QA comprises al measures that make sure that:
Product quality requirements are being explicitly documented
Processes are defined
Product and process quality are regularly monitored and documented
Users are informed on the quality of products
A procedure is implemented so that improvements measures are being planned, implemented and evaluated
A QAF defines a set of concrete measures and how these general objectives should be achieved within a given organisation

3. Quality Assurance Framework
At international level
the objective is to guarantee a certain number of minimum requirements, mostly in an international context
the European Statistics Code of Practice and ESS QAF
the Data Quality Assurance Framework (DAQF) of IMF
the Latin American and Caribbean Code of Good Statistical Practice
the UN generic template and guidelines for QAF
At national level
the objective is to establish, in a specific organisation, a system of methods and tools guaranteeing the adherence to minimum requirements
Statistics Canada Quality Assurance Framework

4. The scope of QAF User needs Management systems & leadership
Statistical products
Support processes
Statistical processes
Institutional environment

5. ESS Quality Assurance Framework (QAF)
Developed by the Sposorship on Quality in order to support the implementation of the CoP
QAF provides the requirements, methods and tools in order to implement each indicator/principle of the CoP (quality commitment, statistical processes, statistical products)
at Institutional level
at product/survey level
Support NSIs to improve compliance to the CoP
Support the assessment of compliance to the CoP
Basis for defining minimum requirements at ESS level
Important tool for the next round of peer reviews

6. P4 Commitment to Quality
P7 Sound Methodology
P8 Appropriate Statistical Procedures
P9 Non-excessive Burden on Respondents
P10 Cost Effectiveness
P11 Relevance
P12 Accuracy and Reliability
P13 Timeliness and Punctuality
P14 Coherence and Comparability
P15 Acessibility and Clarity

7. QAF for P4: Quality Commitment
Indicator 4.1: Quality policy is defined and made available to the public. An organizational structure and tools are in place to deal with quality management.
Methods at institutional level
1. A quality commitment statement
A Quality Commitment Statement is made publicly available, laying out principles and commitments related to quality in statistics which are consistent with the goals set out in the Mission and Vision statements.
2. An organizational structure for managing quality
There is a clear organizational structure for managing quality within the statistical authority. Examples of such a structure are:
Quality Committee;
Quality Manager;
Centralized Quality unit;
Other structures (e.g. a selected group of staff trained as “quality pilots” to act as project/processes coach/advisers).
3. Definition of Quality guidelines
Guidelines are defined on how to implement quality management within the statistical production process, comprising:
A description of the statistical production process and the identification of documentation for each stage, following the Business Process Model for Statistics or any other equivalent process representation;
A description of the methods to monitor the quality of each stage of the statistical production process.
Statistical authority means Eurostat, national statistical institutes (NSIs) and other national authorities responsible for the development, production and dissemination of European statistics, where appropriate (cf. Regulation (EC) No 223/2009, Articles 4 and 5).

8. QAF for P4: Quality Commitment (cont.)
Indicator 4.1: Quality policy is defined and made available to the public. An organizational structure and tools are in place to deal with quality management.
Methods at institutional level 4. Availability of Quality guidelines. Quality guidelines, as defined above, are made available to all users at least in a summary version. 5. An infrastructure for documentation. An appropriate infrastructure is in place in order to ensure updated documentation on quality. 6. Training courses. Specific training courses support the quality policy and are available to relevant staff on a regular basis.

9. QAF for P4: Quality Commitment (cont.)
Indicator 4.2: Procedures are in place to plan and monitor the quality of the statistical production process.
Methods at institutional level
1. Methodological and technical support and general tools. Methodological and technical support and general tools are provided by specialized / dedicated units, namely Quality, Methodology and IT, for implementing process quality monitoring/quality assurance plan.
Methods at product/survey level
2. Procedures to monitor process quality. Procedures are in place to monitor the quality of different stages of the statistical production, e.g. according to a quality assurance plan or a similar scheme, like the establishment of regular expert group meetings.
3. A quality assurance plan. The quality assurance plan, or any other similar scheme, describes the working standards, the formal obligations (such as laws and internal rules) and the set of quality control actions to prevent and monitor errors, to evaluate quality indicators and to control different points at each stage of the statistical process.
4. The quality assurance plan or any other similar scheme:
takes user’s needs into account and checks the relevance of the statistical process;
ensures effective technical and organizational design;
assures the quality of data collection, including the use of administrative data;
assures the quality of data treatment (coding, editing, imputation and estimation);
ensures the systematic examination of possible trade-offs within quality;
makes information accessible and comprehensible to users, and collects reactions/feedback from users;
ensures suitable metadata is provided to users to aid their understanding of quality.

10. QAF for P4: Quality Commitment (cont.)
Indicator 4.3: Product quality is regularly monitored, assessed with regard to possible trade-offs, and reported according to the quality criteria for European Statistics.
Methods at institutional level
1. Procedures to monitor product quality. Procedures based on quality reporting are in place to internally monitor product quality. Results are analyzed regularly and senior management is informed in order to decide improving actions.
2. User satisfaction surveys. User Satisfaction Surveys or other indirect methods are implemented on a regular basis and their results are made public and incorporated where useful in Quality Reports, since they monitor “Relevance”, amongst other dimensions.
Methods at product/survey level
3. User oriented quality reports. User oriented quality reports are made available to the public.
4. Producer oriented quality reports. Producer oriented quality reports are published regularly (periodicity to be determined: e.g. by the specific Regulation and the survey life cycle), bearing in mind the standards for reference metadata and quality indicators, in particular the Single Integrated Metadata Structure (SIMS).
5. Product quality monitoring. Users and producers quality reporting are used for regular quality monitoring over time.

11. QAF for P4: Quality Commitment (cont.)
Indicator 4.4: There is a regular and thorough review of the key statistical outputs using also external experts where appropriate.
Methods at institutional level
1. A plan for implementing Quality Reviews. An appropriate Plan for implementing Quality Reviews (such as Auditing and Self-Assessment) is defined/implemented regularly for key statistical outputs and systematically in the case of processes reengineering.
2. A structure for Quality Reviews. An appropriate structure for carrying out Quality Reviews is in place for internal audits and self-assessments.
3. Training of internal auditors. Internal auditors are trained in auditing techniques and behavior.
4. Reference documentation. Quality reviews have as reference documentation:
Quality guidelines/quality assurance plan, or a similar scheme;
Producer oriented quality reports and/or user oriented quality reports;
Self-assessment questionnaires filled by producers;
Reports from audit interviews;
Questionnaires completed by respondents and/or users;
Any other satisfaction survey.
5. Action plans. The findings of the quality reviews result in action plans.
6. Feedback from users. Feedback from different users is used as input to action plans (making use of User Satisfaction Surveys or Focus groups).
7. Deployment of outside experts. Outside experts are deployed to review key statistical domains (e.g. Data Review of Standards and Codes (ROSC) by the IMF).
8. Benchmarking. Benchmarking on key statistical processes with other statistical authorities is carried out to identify good practices.

12. UN Generic Quality Assurance Framework
Developed by the UN Statistical Commission
Work carried out by an Expert Group (2010 – 2012)
14 countries from all over the world and representatives from International organisations
Outputs
a template for a generic quality assurance framework
mapping to existing quality frameworks
development of guidelines to accompany the template
glossary of the main quality related terms
online inventory of documents and links to national and international quality tools and references

13. The template
The template for generic quality assurance framework is a guidance for developing such a framework
It is of utmost importance for those countries where a national or supranational framework is missing
It is organised in five sections
Quality context
Quality concepts and frameworks
Quality assurance guidelines or NQAF lines
Quality assessment and reporting
Quality and other management frameworks
It is developed in the Guidelines

14. The guidelines for the template
Quality context
A short text explains the benefits and challenges of quality management
Quality concepts and frameworks
Link to the Glossary of quality related terms compiled by the expert group and reference to mapping to existing frameworks.
In particular, the 19 Quality assurance guidelines (or NQAF lines) are mapped to the EU statistics CoP, the International Monetary Fund’s Data Quality Assurance Framework, the Latin American and Caribbean Code of Good Statistical Practice, Statistics Canada Quality Assurance Framework
This exercise is important to stress the existing synergies beyond the differences in wording, ordering, grouping,…..

15. The guidelines for the template
Quality assurance guidelines – NQAF lines
19 lines have been identified as key elements to be considered in building up a NQAF – corresponding to principles of EU CoP
Guidelines are provided for each of the 19 lines consisting in:
a description of the NQAF line (meaning)
a description of the elements to be assured both at agency level and at the programme (survey/process) design stage and at programme implementation stage
supporting mechanism to be put in place
Quality assessment and reporting
How to measure process and product quality and how to communicate it to users, on obtaining feedback from users and on conducting auditing and self-assessment
Quality and other management frameworks
Relationships with performance management, resources management, ethical standards, continuous improvement and governance

16. NQAF – Quality assurance guidelines
Managing the statistical system
NQAF 1 – Coordinating the national statistical system
NQAF 2 – Managing relationships with data users and data providers
NQAF 3 – Managing statistical standards
Managing the institutional environment
NQAF 4 – Assuring professional independence
NQAF 5 – Assuring Impartiality and Objectivity
NQAF 6 – Assuring transparency
NQAF 7 – Assuring statistical confidentiality and security
NQAF 8 – Assuring the quality commitment
NQAF 9 – Assuring adequacy of resources

17. NQAF – Quality assurance guidelines
Managing statistical processes
NQAF 10 – Assuring methodological soundness
NQAF 11 – Assuring cost-effectiveness
NQAF 12 – Assuring soundness of implementation
NQAF 13 – Managing the respondent burden
Managing statistical outputs
NQAF 14 – Assuring relevance
NQAF 15 – Assuring accuracy and reliability
NQAF 16 – Assuring timeliness and punctuality
NQAF 17 – Assuring accessibility and clarity
NQAF 18 – Assuring coherence and comparability
NQAF 19 – Managing metadata

18. NQAF 8 – Assuring the quality commitment
Description:
Statistical agencies should be dedicated to assuring quality in their work, and systematically and regularly identify strengths and weaknesses to continuously improve process and product quality. Processes, staff and facilities should be in place for ensuring that the data produced are commensurate with their quality objectives.
Elements to be assured:
At the agency level
Does the statistical agency’s policy or message about its commitment to quality in statistics clearly convey and promote the shared concern for quality of all of its staff, and include information about tradeoffs affecting the statistical work program? Is this policy made public?
Is a culture of continuous improvement promoted that systematically fosters the documentation of methodology and processes and the exchange of good statistical practices, as well as the monitoring, assessment and improvement of the quality of statistical operations?
Is there a specific person or persons who have been assigned explicit responsibility for the management of quality within the statistical agency?
Does the statistical agency follow any of the externally recognized processes or activities that focus on quality?

19. NQAF 8 – Assuring the quality commitment
Are guidelines for implementing quality management defined which: (a) describe the entire statistical process and identify relevant documentation for each stage of production; (b) describe the methods for monitoring the quality of each stage of the statistical production process; and (c) identify the indicators (quality measures) for evaluating the quality of the main stages of production?
Are the statistical agency’s quality guidelines made available to external users, at least in a summary version?
Are measures in place for conducting periodic quality reviews of key products to assess adherence to internal guidelines and international standards? Is top management informed of the results in order to define improvement actions?
Are procedures in place to ensure that the required documentation on quality is regularly updated?
Are staff training and development programs in place to ensure that the staff is well aware of the statistical agency’s quality policy and has an understanding as to how quality may be achieved; are the statistical agency’s internal auditors trained in auditing techniques and behavior?
Does the management of the statistical agency ensure that compiling areas or subject matter units have access to necessary tools and specialized methodological and technical support as needed to help implement their strategies for improving data development, production and dissemination?
Is benchmarking of key statistical processes with other statistical agencies carried out to identify good practices?

20. NQAF 8 – Assuring the quality commitment
Elements to be assured:
At the program design stage
Is a quality assurance plan or similar mechanism in place that describes the working standards, the formal obligations (such as laws and internal rules) and quality control actions to prevent, monitor and evaluate errors and to control different points at each stage of the statistical process?
Are procedures in place to monitor the quality of different stages of the statistical production, e.g. according to a quality assurance plan or similar mechanism?
Are there mechanisms in place to take users’ needs into account?
Are trade-offs within quality systematically examined?
At the program implementation stage
Are there mechanisms in place to assure the quality of data collection (including the use of administrative data) and data editing?
At the post-collection evaluation stage
Are metadata and quality indicators or measures prepared and provided to users to help them assess the quality of the released data?
Are outside experts called in to conduct some of the quality reviews?
Are mechanisms in place to collect and follow up on users’ reactions and feedback?
Are user satisfaction surveys implemented regularly, and are their results made public?

21. NQAF 8 – Assuring the quality commitment
Supporting mechanisms:
The commitment to quality is likely to be more effectively assured if the following supporting mechanisms are in place:
A written quality policy, declaration or commitment statement is publicly available.
A staff awareness “campaign” is undertaken to emphasize the statistical agency’s commitment to quality.
A quality manager, quality committee, unit or group of coaches or advisers is assigned responsibility for quality management.
Guidelines, methodological manuals and handbooks on recommended practices are made available.

22. Comparing NQAF and EU CoP
What are the main differences in the two approaches?
How is quality assured in the two frameworks?
……..

23. Comparing NQAF and EU CoP
Managing the statistical system
I 2.1; I 2.2; I 2.3; I 3.3; I 3.4;
NQAF 1 – Coordinating the national statistical system
I 2.3; I 7.7; I 9.1; I 9.2; I 9.3; I 9.4; I 9.5; I 9.6; I 11.1; I 11.2; I 11.3; I 15.6; I 15.7
NQAF 2 – Managing relationships with data users and data providers
NQAF 3 – Managing statistical standards
Managing the institutional environment
P1 (I 1.1 – I 1.8)
NQAF 4 – Assuring professional independence
P6 (I 6.1 – I 6.8)
NQAF 5 – Assuring Impartiality and Objectivity
I 6.3; I 6.4; I 6.5; I 6.6; I 8.6
NQAF 6 – Assuring transparency
P5 (I 5.1 – I 5.6)
NQAF 7 – Assuring statistical confidentiality and security
P4 (I 4.1 – I 4.4)
NQAF 8 – Assuring the quality commitment
P3 (I 3.1 – I 3.4)
NQAF 9 – Assuring adequacy of resources

24. Comparing NQAF and EU CoP
Managing statistical processes
P7 (I 7.1 – I 7.7); I 8.2; I 8.3; I 8.4; I 8.5; I 8.6
NQAF 10 – Assuring methodological soundness
P10 (I 10.1 – I 10.4)
NQAF 11 – Assuring cost-effectiveness
I 8.1; I 8.2; I 8.3; I 8.4; I 8.5; I 8.7; I 8.8; I 8.9
NQAF 12 – Assuring soundness of implementation
I 8.7; I 8.8; I 8.9;
P9 (I 9.1 – I 9.6);
NQAF 13 – Managing the respondent burden
Managing statistical outputs
P11 (I 11.1 – I 11.3)
NQAF 14 – Assuring relevance
I 8.6
P12 (I 12.1 – I 12.3)
NQAF 15 – Assuring accuracy and reliability
P13 (I 13.1 – I 13.5)
NQAF 16 – Assuring timeliness and punctuality
P15 (I 15.1 – I 15.7)
NQAF 17 – Assuring accessibility and clarity
P14 (I 14.1 – I 14.5)
NQAF 18 – Assuring coherence and comparability
I 15.1; I 15.2
NQAF 19 – Managing metadata

25. Comparing NQAF and EU CoP QAF
EU CoP- P4 Commitment to Quality
NQAF 8 – Assuring quality commitment
Statistical Authorities are committed to quality. They systematically and regularly identify strengths and weaknesses to continuously improve process and product quality.
Statistical agencies should be dedicated to assuring quality in their work, and systematically and regularly identify strengths and weaknesses to continuously improve process and product quality. Processes, staff and facilities should be in place for ensuring that the data produced are commensurate with their quality objectives
Indicator 4.1: Quality policy is defined and made available to the public. An organizational structure and tools are in place to deal with quality management
---
Indicator 4.2: Procedures are in place to plan and monitor the quality of the statistical production process.
Indicator 4.3: Product quality is regularly monitored, assessed with regard to possible trade-offs, and reported according to the quality criteria for European Statistics.
Indicator 4.4: There is a regular and thorough review of the key statistical outputs using also external experts where appropriate
Methods at institutional level
Elements to be assured:
At the agency level

26. Comparing NQAF and EU CoP QAF
EU CoP- P4 Commitment to Quality
NQAF 8 – Assuring quality commitment
Methods at product/survey level
Elements to be assured at program level
Specified for each indicator
Specified for the process phase
At the program design stage
At the program implementation stage
At the post-collection evaluation stage
Included in the methods/tools at institutional or product/survey level
Supporting mechanisms

27. The generic quality assurance framework
Can be useful to develop a QAF at National level?
What are in your views the advantages of a NQAF?
How could a NQAF relate to the EU CoP and its QAF?

28. NQAF From the work of the UN Expert Group on NQAF
Systematic quality management typically takes the form of a quality assurance framework. Quality management measures adopted by NSIs are sooner or later translated into a formalised quality assurance framework
The objective is to have an overarching framework that would provide context for quality concerns, activities and initiatives, and explain the relationships between the various procedures and tools.
Such a framework has proved to be very useful in providing a single place to record and reference the full range of current quality concepts, policies and practices

29. NQAF
Main benefits:
it provides systematic mechanism for facilitating the on-going identification of quality problems and possible actions for their resolutions. At the same time, it serves to stimulate and maximise interaction among staff throughout the organisation
it gives greater transparency to the processes by which quality is assured and reinforces the image of the office as a credible provider of good quality statistics
it provides a basis for creating and maintain a quality culture within the organisation and contains reference material that can be helpful for training
it supports quality improvements and their maintenance over time

30. Final remarks – ESS level
Quality framework is represented by EU Statistics Code of Practice
ESS QAF has been developed for enhancing the implementation of the EU CoP in Member States
ESS QAF can be a guidance to support systematic quality activity at national level
It is not an easy task to link quality frameworks to the NSI core processes and to contextualise what is already in place
Long term investment that needs to be supported by top management
Frameworks are also useful to pick up ideas and tools and start a more systematic approach