Quality Assurance in Population and Housing Censuses United Nations Regional Workshop on the 2020 World Programme on Population and Housing Censuses: International Standards and Contemporary Technologies, Colombo, Sri Lanka, 8-11 May 2018 Session 6 Quality Assurance in Population and Housing Censuses Meryem Demirci United Nations Statistics Division

Overview Importance of a quality assurance programme The quality assurance circle The role of managers Quality assurance by census phase Dimensions of quality Evaluation of quality Process quality Data quality

Importance of a quality assurance programme P&R recommends that “Each country must have a quality assurance and improvement programme to measure the quality of each stage of the census” (P&R 2.169) A major goal of quality assurance programme is to detect errors during the process and take remedial actions- a quality improvement programme The system should be managed in an integrated way and established at all phases of census operations including planning, questionnaire design, mapping, enumeration, flow of census materials, data capture, coding, editing/imputation, dissemination Quality of census processes will determine the quality of census results

Importance of a quality assurance programme Deficiencies in quality are usually the results of deficiencies in process rather than the actions of individuals working in that process The focus of quality assurance is: to monitor and measure the process quality, to detect errors early to improve procedures for achieving the targets to prevent errors from reoccurring –if possible-, The key to quality assurance and improvement is to be able to regularly measure the cost, timeliness and quality of a given process

The role of managers Managers have a vital role in establishing quality – their main roles include: Establishing a culture within the census agency that has a focus on quality issues –having a programme for all statistical operation, training all staff Ensuring the commitment of staff to strive to achieve high‐quality goals Creating an environment in which everyone has the opportunity to contribute to quality improvement Ensuring that processes for implementing quality assurance programmes are documented and such documentation provide information on: how quality is going to be measured who is involved in identifying root causes of problems with quality, how the process improvements are going to be implemented

Quality assurance circle Many census tasks are highly repetitive and a quality assurance programme can be established Careful design should identify: The types of errors that may occur What information is required to identify such errors How this information will be collected in a timely manner during live operations Who will be responsible for monitoring What action will be taken Implement corrective action Identify most important problems Identify causes of problems Measure quality

Quality assurance by census phase Quality assurance circle should be applied to the entire census cycle with: Performance in the previous phase being evaluated at any given level of detail Impacts of quality problems occurred in previous phase (s) should be evaluated Comparing with positive experience of the previous census Root  causes  identified  and  corrective  action  implemented

Quality assurance by census phase Topic selection The first step in managing the quality of census statistics to ensure that the product will be relevant to users and meets with the requirements outlined in census legislations The key process is extensive consultation with actual and potential users- user consultation programme Consultation with users in different formats: consultations with key government departments and agencies; advice from professional advisory committees in major subject matter areas; user feedback; ad hoc consultations with interested groups; Methods: Meetings, surveys Assessment of quality –preparing a report on user consultation programme, achievements, failure in inclusion of some topics and reasons

Quality assurance by census phase Form Design and Testing Second quality management task: testing of each census question and testing of the design The results of each test being analyzed and necessary changes made using quality circle approach Evaluation of improvements in census questionnaire and satisfaction of key internal stakeholders The dissemination team- ensuring to meet with the needs of users The subject matter specialist team; The team of the data capture or processing system The field operations team The respondents -- to ensure that the forms are easy to complete- self enumeration mail‐out/mail‐back and Internet Assessment report on satisfaction and improvements

Quality assurance by census phase Printing and distribution of census questionnaires/materials Establishment of a system for monitoring the quality of the work done by the printing company Regular check of the quality of the printed documents Monitoring the progress in printing Monitoring distribution of census materials from printing house to final destination – by timing and number of questionnaire and other census materials Assessment on the quality of printing and timely distribution of enough materials and collecting all materials from the field

Quality assurance by census phase Enumeration Checking the quality of the work of enumerators - coverage/content Collecting periodical information from enumerators/ supervisors to assess the progress in enumeration- through management and information system Collecting information on number of enumerated population/housing units, refusal, housing units with no contact Monitoring non-response and follow-up rates and implementing special procedures Monitoring the duration needed for completing enumeration- if not as scheduled, implementing remedial action Quality assessment based on information collected during enumeration-especially on census coverage and the quality of the work of field staff

Quality assurance by census phase Data capture, coding and editing/imputation Procedures should be developed with a view to minimizing the risk of erroneously cancelling, losing or artificially creating households/individuals during data processing-validation of enumerated population/households in different phases Procedures for monitoring the quality of each phase Repeating certain procedures based on the sample of batches/records and comparing two datasets Identifying systematic errors Assessment of the quality of data capture, coding and editing/imputation to ensure these procedures do not create significant errors in census data – Data capture error, Coding error, Imputation rate/Dissimilarity index

Quality assurance by census phase Dissemination The dissemination team is responsible for the timely delivery of products and services to the census data users Management of the quality in census dissemination is driven by concerns to: deliver relevant products and services while maintaining accuracy of the data and timeliness Assessment report on timely dissemination of census products and relevant tools for ensuring that the results are accessible to users

Dimensions of data quality Quality is a multidimensional concept Quality of outputs of any statistical exercise should be assessed with some or all of the following six main attributes: Relevance Accuracy Timeliness Accessibility Interpretability Comparability and coherence Some of these dimensions are inter-dependent and involve trade-off (eg. timeliness and accuracy)

Dimensions of quality: Relevance The relevance of statistical information is the degree to which it meets the needs of users -- and suggests the need to avoid the collection and production of data for which there is no significant use This dimension is important in census content development and dissemination Relevance is a qualitative assessment of the value of the census data produced, including in terms of meeting the mandate of the agency, legislated requirements and users` needs

Dimensions of quality: Accuracy The accuracy of statistical information is the degree to which those data correctly estimate or describe the quantities or characteristics that the statistical activity was designed to measure It is usually characterized in terms of errors in statistical estimates introduced by major sources of errors such as coverage error, content error, item non-response, imputation rate, etc. PES, information collected in field enumeration and data processing

Dimensions of quality: Timeliness Timeliness refers the length of time between the census reference day and the date on which the information becomes available There is a trade-off against accuracy Often for a census there are several release dates to be considered in dissemination schedule-temporary results, final results, thematic reports, online dissemination database, etc. Major information releases should have publication dates announced well in advance

Dimensions of quality: Accessibility The accessibility of statistical information refers to the ease with which it can be obtained The suitability of the form in which the information is available to users, the media of dissemination Where data products are provided at cost, the affordability of the information to users also affects accessibility

Dimensions of quality: Interpretability The interpretability of statistical information reflects the availability of supplementary information and metadata necessary to interpret and use it Usually it covers the underlying concepts, definitions, variables, classifications used, the methodology of data collection and processing and indications of the accuracy of the information

Dimensions of quality: Comparability The comparability of statistical information reflects the degree to which statistical information is comparable across countries and time Usually underlying concepts, definitions, classifications used, the methodology of data collection and processing provide information on comparability Using international recommendations on concepts, definitions, classifications and census methods increases comparability

Dimensions of data quality Coherence Coherence reflects the degree to which the census information can be successfully brought together with other statistical information within an integrated framework over time The use of standard concepts, definitions and classifications – possibly agreed at the international level - promotes coherence

Evaluation P&R recommends that “a complete evaluation takes place and is documented at the end of each phase of the census” Evaluation of the overall census operation is vital for identifying strengths and weaknesses of census phases, including planning, enumeration, data processing and dissemination, and also for the purpose of analysing the quality of census statistics, which are the major output of these processes. A comprehensive evaluation programme should include: Evaluation of census processes – operational assessments Evaluation of the quality of products- assessment of coverage and content errors and dimension of the quality The results of evaluations of census operation for both operational aspects and the quality of data should be made available to the stakeholders

Evaluation – Operational aspects/processes P&R recommends that “the census evaluation programme should be undertaken by subject specialists according to the agreed goals and methodologies covering all possible dimensions of quality.” Some areas for evaluation include: Identification of the deficiencies and achievements in data capture, coding and editing; Relevance of census data to user needs and satisfaction of users with dissemination tools and products (based on information collected through user consultation); Achievements and difficulties in use of new technologies and methodologies and identification of improvements for the next census; Realization of the census calendar, including the calendar of releasing census results, and, in the case of changes to the calendar, the reasons and consequences.

Evaluation – Data quality P&R recommends that “Evaluation of the accuracy of the census data should be undertaken, to the extent possible, by: conducting a post‐enumeration survey for measuring coverage and content errors, comparing the census results with similar data from other sources (surveys and administrative records in a similar time frame & previous census results) and applying demographic analysis The purposes of evaluating the accuracy of the data are to inform users on the quality of the current census data and to assist in future improvements.

Conclusions Quality assurance and improvement systems should be developed as part of the overall census programme, and integrated with other census plans, schedules and procedures The systems should be established at all phases of census operations, including planning, pre‐enumeration, enumeration, document flow, coding, data capture, editing, tabulation and data dissemination. Quality management procedures for each phase of the census should be assessed with appropriate methods