1. Quality assurance in population and housing censuses
Session 6
Quality assurance in population and housing censuses
UNSD presentation

2. Overview Importance of a quality assurance programme
The quality assurance circle
The role of managers
Dimensions of quality
Quality assurance by census phase
Evaluation
Process quality
Data quality

3. Importance of a quality assurance programme
P&R recommends that “Each country must have a quality assurance and improvement programme in place to measure the quality of each stage of the census” (P&R 2.169)
A major objective of a quality assurance programme is to ensure that quality assessment is consistently incorporated in all phases of the census, focusing on efforts in controlling the occurrence of errors and taking remedial actions to ensure the highest quality of both the processes and their outcomes.
A quality assurance programme should also be viewed as a quality improvement programme, and without such a programme, the census data when finally produced may contain many errors, which can severely diminish its usefulness
The quality assurance and improvement system should be developed as part of the overall census programme - integrated with other census plans, schedules and procedures, - and established at all phases of census operations, including planning, pre‐enumeration, enumeration, coding, data capture, editing, tabulation and data dissemination.

4. Importance of a quality assurance programme
Quality is the outcome of processes, and deficiencies in quality (for example, delays in processing or lack of accuracy in the results) are usually the result of deficiencies in process rather than the actions of individuals working in that process.
Quality is relative, and is based on what is acceptable to data users, or fit for the purpose, rather than on a concept of absolute perfection
The key to quality assurance and improvement is the ability to regularly measure the timeliness and accuracy of a given process so that the process can be improved when a fall in quality is indicated.
The focus of quality assurance is to prevent errors from reoccurring, to detect errors easily and inform the workers so that they do not continue --this simple feedback loop is represented in the “quality assurance circle”

5. Quality assurance circle
The quality assurance circle is a schematic representation of the iterative process by which quality is improved
Quality assurance circle is particularly applicable to tasks that are highly repetitive such as the processing phase of the census
It is less applicable in processes that are one-off or time-constrained (eg. enumeration) as there is less opportunity to measure performance, identify problems and implement corrective actions
The emphasis of the quality circle is on improving the process that caused the “error”, which may be any of the cost, timeliness or accuracy attributes falling below specified levels.
Implement corrective
action
Identify most important problems
Identify causes
of problems
Measure
quality

6. The role of managers
Managers have a vital role in establishing quality – their main roles include:
Establishing a culture within the census agency that has a focus on quality issues and to obtain the commitment of staff to strive to achieve high‐quality goals
Creating an environment in which everyone has the opportunity to contribute to quality improvement
Ensuring that clients’ expectations are known and that these expectations are built into planning objectives and into the systems that are to deliver them
Ensuring that processes for implementing quality assurance programmes are documented and such documentation provide information on:
how quality is going to be measured
who is involved in identifying root causes of problems with quality,
how the process improvements are going to be implemented

7. Dimensions of quality Quality is a multidimensional concept
Outputs of any statistical exercise should possess some or all of the following six main attributes:
Relevance
Accuracy
Timeliness
Accessibility
Interpretability
Comparability
Some of these dimensions are inter-dependent and involve trade-off (eg. timeliness and accuracy)
Additional dimensions of quality: Coherence, completeness

8. Dimensions of quality: Relevance
The relevance of statistical information is the degree to which it meets the needs of users -- and suggests the need to avoid the collection and production of data for which there is no significant use
This dimension is important in census content development and dissemination
Relevance is a qualitative assessment of the value of the census data produced, including in terms of meeting the mandate of the agency, legislated requirements

9. Dimensions of quality: Accuracy
The accuracy of statistical information is the degree to which those data correctly estimate or describe the quantities or characteristics that the statistical activity was designed to measure
It is usually characterized in terms of errors in statistical estimates introduced by major sources of errors such as coverage, sampling, non-response, processing, etc

10. Dimensions of quality: Timeliness
Timeliness refers the length of time between the census reference day and the date on which the information becomes available
It represents the degree to which information is released in a time period that still permits the information to be of value to users
It often involves a trade-off with accuracy
Census results are often made available over several release dates, so to provide an assessment of timeliness, major information releases should have specified publication dates in the dissemination schedule

11. Dimensions of quality: Accessibility
The accessibility of statistical information refers to the ease with which it can be obtained
Takes into account the suitability of the form in which the information is available to users, the media of dissemination
Availability of metadata
Where data products are provided at cost, the affordability of the information to users also affects accessibility

12. Dimensions of quality: Interpretability
The interpretability of statistical information reflects the availability of supplementary information and metadata necessary to interpret and use it
Usually it covers the underlying concepts, definitions, variables, classifications used, the methodology of data collection and processing and indications of the accuracy of the information

13. Dimensions of quality: Comparability
The comparability of statistical information reflects the degree to which statistical information is comparable across countries, regions within a country, and time
Usually underlying concepts, definitions, classifications used, the methodology of data collection and processing provide information on comparability

14. Additional dimensions of data quality
Coherence
Coherence reflects the degree to which the census information can be successfully brought together with other statistical information within an integrated framework over time
The use of standard concepts, definitions and classifications – possibly agreed at the international level - promotes coherence
Completeness – an extension of relevance
Completeness reflects the degree to which statistics serve users as completely as possible, taking restricted resources and respondent burden in to account

15. Quality assurance by census phase
The quality circle can be applied to the entire census cycle with:
Performance in the previous phase being evaluated at any 
given level of detail; 
Problems with quality ranked in order of importance; 
Root  causes  identified  and  corrective  action  implemented
The following slides outline the ways in which the concept of the quality circle is applied to the census cycle, with focus on some procedures for improving the quality of important phases including form design, enumeration, processing and dissemination

16. Quality assurance by census phase
Topic selection
The first step in managing the quality of census statistics to ensure that the product will be relevant to users and meets requirements outlined in census legislations
The key process is extensive consultation with actual and potential users
Consultation with users would include: consultations with key government departments and agencies; advice from professional advisory committees in major subject matter areas; user feedback; ad hoc consultations with interested groups;

17. Quality assurance by census phase
Form Design and Testing
The next quality management task concerns the testing of each census question and the testing of the design of the form.
There are several key internal stakeholders of the form design process, whose requirements need to be taken into account:
The dissemination team -- to ensure that the questions asked will deliver the data to meet the needs of users;
The subject matter specialist team;
The team responsible for development of the data capture or processing system - especially for data collection using scanning systems or an electronic questionnaire
The field operations team -- which is responsible for training the enumeration workforce and printing the form;
The respondents -- to ensure that the forms are easy to complete in mail‐out/mail‐back based self enumeration

18. Quality assurance by census phase
Printing and distribution of census questionnaires/materials
Estimation of number of questionnaires and census materials is critical step for cost-effective way of printing census materials
Establishment of a system for monitoring the quality of the work done by the printing company
Regular check of the quality of the printed documents
Monitoring the progress in printing
Monitoring distribution of census materials from printing house to final destination – by number of questionnaire and other census materials

19. Quality assurance by census phase
Recruitment and training of the field staff
Criteria for selection of enumerators/supervisors
Clear procedures for recruitment
Training materials for ensuring a standard training programme
Monitoring and supervision of training and recruitment of enumerators/supervisors by relevant census supervisors

20. Quality assurance by census phase
Enumeration
Checking the work of enumerators - coverage/content
Collecting periodical information from enumerators/ supervisors to assess the progress in enumeration
Collecting information on number of enumerated population/housing units, refusal, housing units with no contact
Monitoring non-response and follow-up rates
Assessment for the risk of not completing enumeration as scheduled
Special procedures for refusal/housing units with no contact

21. Quality assurance by census phase
Data capture, coding and editing
Processing procedures should be developed with a view to minimizing the risk of erroneously cancelling, losing or artificially creating households during all phases of data processing
Procedures for monitoring the quality of each phase
Repeating certain procedures based on the sample of batches/records and comparing two datasets
Identifying systematic errors
Assessment of the quality of the procedures and difficulties faced

22. Quality assurance by census phase
Dissemination
The dissemination area is responsible for the timely delivery of products and services to the census data users
Therefore insufficient planning and resources for this phase can have the effect of delaying the release of the data and thus compromising the overall achievement of the census objectives
Management of the quality in census dissemination is driven by concerns to:
deliver relevant products and services while maintaining accuracy of the data, and timeliness and predictability of data release within agreed cost constraints

23. Evaluation
P&R recommends that “a complete evaluation takes place and is documented at the end of each phase of the census - particularly for phases such as enumeration, so that the organizational learning inherent in the quality circle is carried forward to the next census”
Evaluation of the overall census operation is vital for identifying strengths and weaknesses of census phases, including planning, enumeration, data processing and dissemination, and also for the purpose of analysing the quality of census statistics, which are the major output of these processes.
A comprehensive evaluation programme should include:
Evaluation of census processes – operational assessments
Evaluation of data quality - assessment of coverage and content errors
The results of evaluations of census operation for both operational aspects and the quality of data should be made available to the stakeholders

24. Evaluation – Operational aspects/processes
Operational assessments provide valuable information on strengths and weaknesses of past operational procedures which should be carefully reviewed prior to the development of the next census
Operational assessments should:
document operational errors
explain the effectiveness of operations and procedures and their likely impact on overall quality of census
Census evaluation with all dimensions of quality requires a comprehensive evaluation programme for assessing and documenting the outcomes of each process using appropriate and customized methodologies – these methodologies should be planned well in advance, in the planning phase of the census

25. Evaluation – Operational aspects/processes
P&R recommends that “the census evaluation programme should be undertaken by subject specialists according to the agreed goals and methodologies covering all possible dimensions of quality.” Some areas for evaluation include:
Identification of the deficiencies and achievements in data capture, coding and editing;
Relevance of census data to user needs and satisfaction of users with dissemination tools and products (based on information collected through user consultation);
Achievements and difficulties in use of new technologies and methodologies and identification of improvements for the next census;
Realization of the census calendar, including the calendar of releasing census results, and, in the case of changes to the calendar, the reasons and consequences.

26. Evaluation – Data quality
P&R recommends that “Evaluation of the accuracy of the census data should be undertaken, to the extent possible, by conducting a post‐enumeration survey for measuring coverage and content errors, by comparing the census results with similar data from other sources (surveys and administrative records in a similar time frame & previous census results) and by applying demographic analysis”
The purposes of evaluating the accuracy of the data are to inform users on the quality of the current census data and to assist in future improvements.
Evaluation of data accuracy may enable the identification of any problem areas that have not been previously detected through the quality management processes in earlier phases of the census

27. Conclusions
Quality assurance and improvement systems should be developed as part of the overall census programme, and integrated with other census plans, schedules and procedures
The systems should be established at all phases of census operations, including planning, pre‐enumeration, enumeration, document flow, coding, data capture, editing, tabulation and data dissemination.
Quality management procedures for each phase of the census should be assessed with appropriate methods