1. Process Quality in ONS Rachel Skentelbery, Rachael Viles & Sarah Green
Methodology Directorate,
Office for National Statistics

2. Aim of Presentation
This presentation aims to give the audience an appreciation of the importance of process quality within the ONS and how we are looking to implement this within the organisation

3. Overview Background to Process Quality at ONS
ONS Process Quality Initiatives
ONS Process Quality Handbook
Process Quality Measures
Implementation

4. ‘product quality will follow from improvements in process quality’
Process Quality at ONS
Leadership expert group (LEG) final report (Eurostat (2002)) states that:
‘in theory, good product quality can be achieved through evaluations and rework’
This is costly and time consuming and so instead, it is believed that:
‘product quality will follow from improvements in process quality’
Quality is one of the key principles in NSCoP and runs through all our frameworks
At EU Level the requirement to improve measurability of statistical outputs and processes in embedded in the European Statistical System
Process quality is important since it is a time and cost effective way to meet product/output quality criteria – this would otherwise be unachievable within the constraints of a business environment.

5. Impact of Process Quality at ONS
Reputation
Business
Quality
Customer Perceived Quality
So we know why we are driven by our and Euro frameworks that we should be looking at process quality but how does process quality fit into ONS as an organisation?
At ONS we implement process quality to ensure quality outputs for out customers
Extremely important for a corporate face of organisation
Process Quality
Product Quality

6. Definitions Process: Process Quality: Process variables:
A series of actions or steps towards achieving a particular end
Process Quality:
An assessment of how far each steps meets defined criteria
Process variables:
Factors that can vary with each repetition of the process
These are quite broad and generic definitions – they are not definitive
Products or outputs are generated by an underlying process or sequence of processes, and so the product quality is likely to be affected by the process quality.

7. ONS Process Quality Initiatives
Process Quality Project in ONS
Summary of the European Handbook on Improving Quality by Analysis of Process Variables
A defined set of process quality measures
Project aims to develop a CQI framework for the statistical production processes, and tools to enable this with the view to roll out across ONS and wider GSS
Summarised version of the EU HB for ONS with updated guidance on improving quality by analysis of process variables
Measures to be developed through the course of the project
So how have we addressed these directives in ONS?

8. ONS Process Quality Handbook
Aims:
To assist the implementation of Continuous Quality Improvement techniques into ONS
To provide guidance on applying the technique for each step in the Statistical Value Chain

9. Overview of Process Quality Handbook
Key Process Variables are factors that:
can vary with each repetition of the process
have a large effect on key characteristics
Monitoring the process variables is split into 3 stages within the handbook:
The way we judge process quality is using Key Process Variables
We have defined a process variable earlier as a factor that can vary with each repetition of the process, those that are key are those which have a large effect on key characteristics of the output
Key Process Variables are the main focus of the HB

10. Steps for Continuous Quality Improvement
Identify key characteristics of the processes that will to make the output fit user requirements
Develop a process map
Determine key process variables that will impact on the key characteristics
Evaluate how to measure the key process variables
Monitor stability of the key process variables
Determine if the process produces an output that over time continuously meets user requirements
Establish a system for the continuous monitoring of the process
So we have a defined set of steps for implementing process quality in ONS in order to achieve CQI.
The first step is to identify the key characteristics of the process that will make the output fit user requirements. It is not enough to identify a process, if we are to implement process quality and effectively achieve CQI we must understand the output characteristics that are important to users, both internal and external, and make the output a success – these are key characteristics. Very often these are defined by users but some examples of key characteristics may be: X, Y, Z.
Once a process has been identified for monitoring and the key characteristics identified, the next step is to map the process by developing a comprehensive flow chart of the process, known as a process map. Process maps can be used to:
take an objective look at a process
identify who owns the process or parts of it
identify sources of variation in the process
identify ways to monitor the process
identify waste - any activity that requires resource but adds no value
An effective process map will establish the following:
pictorial representation of a process
clearly defined start and end points
the flow of the process and decision points within it
owners of each process
what actually happens in a process - not what 'ideally' should occur.
Potential sources of information during process mapping are: existing documentation eg survey procedures manuals and specifications; statistical output managers; those who carry out the tasks - the process owners.
The handbook includes the three main points I discussed on the previous slide (identification, measuring and analysis of the key process variables) I will expand on these with example in a moment.
The next important thing to consider is if the process is consistent over time – we might be able to produce an output to user requirements once but can we do it time after time – we need to ensure this happens otherwise measuring the key process variables is a waste of time as we will never achieve a state of CQI.
Once we have established all the steps then we can maintain a system for CQI

11. 3. Determine Key Process Variables
Not efficient or effective to measure all process variables
Identify those key process variables
Useful Tools:
Cause & Effect diagrams
Pareto Charts
We need to identify the key process variables from the process map
Key Process Variables are factors that:
can vary with each repetition of the process
have a large effect on key characteristics
Useful tools to identify these are cause and effect diagrams and pareto charts

12. Pareto Chart of Contribution to Total Item Non-response by Question
Example from ONS E-Commerce Survey
Pareto chart showing that out of X questions we can identify 4 questions that contribute 33% of the total item non-response and so these will be our the process variables that will be key for us to consider.
As part of work to improve the quality of the E-commerce questionnaire, the Data Collection Branch of Methodology Group Division, fully reviewed the questionnaire for This included carrying out a post-implementation evaluation, aiming to:
assess how successful revisions to the questionnaire had been in terms of:
- meeting data requirements;
- respondent response to the various questions;
highlight areas for further improvement.
To achieve these aims the evaluation team gathered feedback from the Survey Processing Centre (SPC) who scan and capture data responsible for scanning and capturing the data, and the Data Validation Branch (DVB) who validate the data and answer respondent queries responsible for data validation and answering respondent queries. As well as common key process variables related to response rates, the evaluation team used feedback from the SPC
(following analysis of batch errors in scanning) to identify several key process variables to measure and analyse.
The full list includes percentages of:
a) item non-response;
b) unnecessary response;
c) entered a mark in both yes and no boxes;
d) left both yes and no boxes blank;
e) entered a numeric value in a mark box;
f) entered a mark in a numeric field;
g) completed more than one mark box where only one is expected;
h) entered a non-relevant mark in or across mark boxes.

13. 4. Measure Key Process Variables
How accurately key process variables can be measured?
Measurement should require minimum effort
Key points:
what can be measured that is useful to understanding and monitoring the system
can the measures be used to assess changes in the quality
what does success look like
can the measures lead to potential for improvement
After identifying the key process variables, it is important to know how accurately they can be measured. Ideally this measurement should require as little effort as possible on the part of the producers. The key point to consider when evaluating how to measure the key process variables are:
what can be measured that is useful to understanding and monitoring the system
can the measures be used to assess changes in the quality
what does success look like
can the measures lead to potential for improvement

14. Example of Measuring Key Process Variables
Example from ONS UK Census: Occupation Coding
Key process variable:
- accuracy of coding (consistency rate) for each Estimation Area (EA)
2% sample of codes was verified, for each EA, independently by coders
Sample data were used to estimate process variables:
Example from UK Census on occupation coding:
What can be measured that is useful to understanding and monitoring the system?
We have identified a KPV as consistency rate and we have collected data that means we can measure it
Can the measures be used to assess changes in the quality?
We can see how the consistency rate changes over time and compare to see if consistency is improving at any time
What does success look like?
Success will be consistently reaching an agreed level of consistency in the accuracy of coding
Can the measures lead to potential for improvement?
Yes, if we see consistency rate is too low or drops then we can find places in the process where we can look to rectify this – more checking for example
Evaluate measurement capability:
Is consistency a good measure of accuracy?
What is the sampling error of our measurements?

15. 5. Monitor Stability of Key Process Variables
Need to analyse the key process variables
Process stability:
a state where the process variation consists entirely of random components
i.e. the variation is not systematic
Useful Tool:
- Control charts
Once key process variables have been identified and measured in the previous 2 steps, analysis of them can begin. First, they are tested for stability: process stability is a state where the process variation consists entirely of random components i.e. the variation is not systematic.

16. Control Chart of Coding Consistency Rate for Occupation Question
Example from ONS UK Census
So from the previous example we would want to consider the consistency rate over time and can do this using a control chart
From here we can establish if over time the process produces an output that continuously meets user needs and develop a process for continuous improvement

17. The way forward Processes fall into two categories:
those amenable to the whole process quality approach
those where it is difficult to identify measurable process variables
Identify those areas where this approach can be used
Support output areas implementation
Compendia of quality – asking business areas what process quality checks they have in place and look at their quality measurement and monitoring.

18. Process Quality Measures
Enable producers to measure and monitor quality of processes over time
Responsive to problems, highlight areas for improvement
Quality assurance for users
Standardised approach, comparability
All very well having a handbook but we need to develop a definitive set of quality measures for people to use alongside this

19. Process Quality Measures in ONS
Currently developing a comprehensive list of process quality measures that will link to:
Each step of Statistical Value Chain
The Eurostat Process quality attributes:
Efficiency
Effectiveness
Robustness
Flexibility
Transparency
Integration
Will look to derive a subset of Key Process Quality Measures
Equivalent to the ‘Guidelines for Measuring Statistical Quality’ but with the focus on process quality measures

20. Implementation Across ONS
Pilot use of Handbook
Bespoke support to output areas from ONS Quality Centre
E-learning course for process quality tools
Expert led training sessions
Quality strategy targets
Pilot – working with output areas to see where tools and concepts will work and to ensure implementation and continued use to ensure CQI
Quality strategy targets – by 2011 key statistical outputs to publish annual process quality measures – self imposed target

21. Review ONS specific Process Quality Handbook:
Developed
To be rolled out across ONS after piloting
Process Quality Measures:
- Key Quality Measures
Blended approach to implementation
E-learning
Expert led training
Support and help to areas
Quality strategy targets

22. Further Reference Points
Eurostat Handbook on Improving Quality by Analysis of Process Variables
Contact me directly at in ONS Quality Centre: