1. Statistical Expertise for Sound Decision Making Quality Assurance for Census Data Processing Jean-Michel Durr 28/1/20111Fourth meeting of the TCG - Lubjana

2. Statistical Expertise for Sound Decision Making Overview Data processing cycle Quality Assurance for Processing: – Objectives – QA Framework: Quality management system Setting the minimum standard Continuous quality improvement 28/1/2011Fourth meeting of the TCG - Lubjana2

3. Statistical Expertise for Sound Decision Making Data processing cycle 28/1/2011Fourth meeting of the TCG - Lubjana3

4. Statistical Expertise for Sound Decision Making The data processing cycle Sequence of activities between the enumeration phase and the dissemination phase Data processing cycle involves many different interdependant activities Largely depends on the technology used: for ex. Coding may take place before of after data capture 28/1/2011Fourth meeting of the TCG - Lubjana4

5. Statistical Expertise for Sound Decision Making 28/1/2011Fourth meeting of the TCG - Lubjana5

6. Statistical Expertise for Sound Decision Making Data processing cycle Receipt and registration: – Forms received at the processing centres are registered to ensure that all enumeration areas are accounted for – Need to coordinate with managers in field operations to monitor the deliveries 28/1/2011Fourth meeting of the TCG - Lubjana6

7. Statistical Expertise for Sound Decision Making Data processing cycle Preliminary checking: – Regardless of the technology employed, some type of checking of the forms is necessary – Can vary from superficial checks to ensure that the forms are in adequate condition to be read by scanners to transcription of damaged forms and manual editing of responses 28/1/2011Fourth meeting of the TCG - Lubjana7

8. Statistical Expertise for Sound Decision Making Data processing cycle Coding: – Coding assigns classification codes to responses on the census form – Coding can be an automated system, computer assisted, clerical or a combination of all three 28/1/2011Fourth meeting of the TCG - Lubjana8

9. Statistical Expertise for Sound Decision Making Data processing cycle Data capture: – system used to capture information from the census form and create a computer data file – Can include: Key entry Optical mark recognition Intelligent character recognition PDA/Internet 28/1/2011Fourth meeting of the TCG - Lubjana9

10. Statistical Expertise for Sound Decision Making Data processing cycle Editing: – Procedure for detecting errors in and between data records, during and after data collection and capture, and on adjusting individual items – Systematic inspection of invalid and inconsistent responses, and subsequent manual or automatic correction, according to predetermined rules 28/1/2011Fourth meeting of the TCG - Lubjana10

11. Statistical Expertise for Sound Decision Making Data processing cycle Validation: – Validation is the final check of data to ensure that the quality of the data meets agreed minimum standards – Tabulations of the final database: To ensure internal coherence To compare with other sources (see demographic methods) 28/1/2011Fourth meeting of the TCG - Lubjana11

12. Statistical Expertise for Sound Decision Making Quality Assurance 28/1/2011Fourth meeting of the TCG - Lubjana12

13. Statistical Expertise for Sound Decision Making Quality Assurance Objectives: – During the processing of census data, assuming that the criterion of relevance has already been met, the emphasis should be on: Data accuracy Budget Timeliness – Necessary trade off between the three 28/1/2011Fourth meeting of the TCG - Lubjana13

14. Statistical Expertise for Sound Decision Making Quality Assurance Framework Quality management system Setting the minimum standard Continuous quality improvement 28/1/2011Fourth meeting of the TCG - Lubjana14

15. Statistical Expertise for Sound Decision Making Quality Management System Units of work selected: – Too costly to control all units => use of sampling – Not only clerical work but also automated processes (OCR) should be included – Outsourced or not Method of operation Rejected units of work 28/1/2011Fourth meeting of the TCG - Lubjana15

16. Statistical Expertise for Sound Decision Making Use of sampling Basic rules: Sampling rates relatively high at the beginning gradually decreasing as operators become more proficient All operators should have their first workload (e.g., EA) sampled More proficient operators subject to a lower sampling rate All operators should have some of their work sampled over the complete life cycle of the process 28/1/2011Fourth meeting of the TCG - Lubjana16

17. Statistical Expertise for Sound Decision Making Use of sampling Basic rules: Sampling rates may be increased towards the end of a process so that the quality of work does not suffer as staff lose interest in the process as it comes to an end Complex processes (e.g., coding occupation or industry) should be sampled at a higher rate than simpler processes (e.g., coding birthplace or religion) 28/1/2011Fourth meeting of the TCG - Lubjana17

18. Statistical Expertise for Sound Decision Making Use of sampling Basic rules: – Initial sampling units should be based on operational efficiency: If the basic workload is an EA, the sample should first be based on a percentage of EAs. – The sample can then be further refined to a percentage of households within those EAs… 28/1/2011Fourth meeting of the TCG - Lubjana18

19. Statistical Expertise for Sound Decision Making Method of operation Will depend on the process (data capture, coding…) : – A sample can be reprocessed by another operator, then comparison and inspection by a supervisor to determine the correct code – A sample can be directly controlled by a supervisor – A sample of forms captured by OCR can be captured by key entry and compared – A computer programme for editing and imputation can be controlled using tabulations 28/1/2011Fourth meeting of the TCG - Lubjana19

20. Statistical Expertise for Sound Decision Making Rejected units In general, rejected units of work are not reprocessed Except in cases where they do not meet the defined minimum standard Because the benefit is generally not justified by the cost: – Coding discrepancy rate 10%, sample rate 10%, correcting all the discrepancies would only reduce the overall discrepancy rate for that topic to 9 per cent. 28/1/2011Fourth meeting of the TCG - Lubjana20

21. Statistical Expertise for Sound Decision Making Setting the standard Measurable criteria for each part of the process where the output can be flagged as either “pass” or “fail” Based on results of previous censuses, tests, other surveys or international comparisons Trade off accuracy-timeliness-costs: avoid over- quality Important to prioritize: – Some variables are more important – Some sub-populations are more important Define limits: Correct / Acceptable 28/1/2011Fourth meeting of the TCG - Lubjana21

22. Statistical Expertise for Sound Decision Making Setting the standard Examples: – Receipt: 100% EAs received – Data capture (OCR): 99,9% for sex, date of birth 98% for other variables – Coding: 95% for occupation 99% for municipality 28/1/2011Fourth meeting of the TCG - Lubjana22

23. Statistical Expertise for Sound Decision Making Continuous Quality Improvement Core component of QA, different from QC Data processing phase lasts enough to be improved Four steps: – 1. Measure quality – 2. Identify the most important problems – 3. Identify the root causes of these important quality problems – 4. Implement corrective action 28/1/2011Fourth meeting of the TCG - Lubjana23

24. Statistical Expertise for Sound Decision Making Step 1: Measure Quality Regular reports: – At individual or team level – Every week or fortnight – Time series of indicators to show the trend: Rates of discrepancy % of work units accepted 28/1/2011Fourth meeting of the TCG - Lubjana24

25. Statistical Expertise for Sound Decision Making Ex. 1: Clerical coding 28/1/2011Fourth meeting of the TCG - Lubjana25

26. Statistical Expertise for Sound Decision Making Ex. 2: OCR data capture before/after correction 28/1/2011Fourth meeting of the TCG - Lubjana26

27. Statistical Expertise for Sound Decision Making Ex. 3: OCR data capture 28/1/2011Fourth meeting of the TCG - Lubjana27

28. Statistical Expertise for Sound Decision Making Step 2: Identify most important pb. Most frequent discrepancies Most problematic: – Error on the first digit is more problematic than on the last digit of a 4 digits classification Reports should provide information 28/1/2011Fourth meeting of the TCG - Lubjana28

29. Statistical Expertise for Sound Decision Making Step 3: Identify root causes Staff working in a process are in the best position to advise about how that process can be improved Case reporting forms to describe problems and provide suggestions Quality improvement team / facilitator 28/1/2011Fourth meeting of the TCG - Lubjana29

30. Statistical Expertise for Sound Decision Making Step 4: Implement corrective action Possible corrective actions: – Changes to procedures – Changes to the processing systems – Retraining or additional training – Reminders about particular procedures sent to staff – Changes to coding indexes 28/1/2011Fourth meeting of the TCG - Lubjana30

31. Statistical Expertise for Sound Decision Making Step 4: Implement corrective action – Before any corrective action is implemented, the implications must be carefully considered – Decision should be made at a high management level – Could be done through a quality management steering committee 28/1/2011Fourth meeting of the TCG - Lubjana31

32. Statistical Expertise for Sound Decision Making References UN Handbook on Census Management for Population and Housing Censuses 28/1/2011Fourth meeting of the TCG - Lubjana32 Eurostat Handbook on Data Quality Assessment Methods and Tools

33. Statistical Expertise for Sound Decision Making Thank you ! 28/1/2011Fourth meeting of the TCG - Lubjana33