1. Overview of Census Evaluation Methods

2. Why evaluate ? The census is a huge operation including many stages
It is not perfect and errors can and do occur at all stages of the census operation
Many countries have recognized the need to evaluate the overall quality of their census results and have employed various methods for evaluating census coverage as well as certain types of content error 2 2

3. Purpose of census evaluation
To provide users with some measures of quality of census data to help them interpret the results
To identify types and sources of error in order to assist the planning of future censuses
To serve as a basis for constructing the best estimate of census aggregates, such as total population, or to provide adjustment of census results

4. Planning Census Evaluation Programme
Census evaluation programme should be developed as part of the overall census programme and integrated with other census activities
Census errors can happen at all phases of census operation such as questionnaire design, mapping, enumeration, data capture, coding, editing, etc.
Evaluation of data accuracy may have two parts:
Preliminary evaluation will enable the identification of any problem areas that have not been previously detected in its process
More extensive evaluation should be undertaken on data items where problems have been identified
The results of the evaluation should be made available to census data users

5. Scope and methods of evaluation
Scope of evaluation of census data should be decided during the planning phase
Scope might be determined for example as:
Estimate coverage error at national level, regional level or provincial level
Analyze evidence of age misreporting
Compare census data with independent data sources (surveys, registers)
Methods of census evaluation should be determined according to resources and objectives

6. Institutional organization
Census evaluation team should be established
Team should be trained for the evaluation methods including demographic methods, other sources of data, etc.
Team should consist of staff having experience on different census topics such as demography, education, housing, labor force, etc.
Team should have knowledge on historical events and transition in population structure in the country
Team should collaborate with related institutions

7. Institutional organization
Equipment needed for census data evaluation including hardware and software should be assessed in the initial stage of planning
Cost of evaluation should be included in overall census budget

8. What are census errors? Coverage errors: Content errors:
Errors in the count of persons or housing units resulting from cases having been “missed” or counted erroneously
Content errors:
Errors in the characteristics of persons or housing units resulting from the interview operation (enumerators/respondents), coding, editing, etc. 8

9. Coverage errors Omissions
Missing housing units, households and/or persons during census enumeration
If the whole housing unit is missed, all households and persons living in the housing unit will also be missed
Major causes of omissions are:
Failure to cover whole land area of a country in creating EAs
Mistakes made by enumerators in canvassing assigned areas
Ambiguous definitions of EAs, unclear boundaries of EAs, faulty maps or coverage error during the pre-census listing exercise 9 9

10. Coverage errors Omissions contd.
In addition, omissions within EAs can result because all or some of the members of the household were not present at the time of enumeration
Proxy respondents can inadvertently or deliberately omit some members of a household 10

11. Coverage errors Duplications
Occur when persons, households or housing units are counted more than once
Reasons for duplications include:
Overlapping of enumerator’s assignments owing to errors done during pre-census listing and delineation
Failure by enumerators to clearly identify boundaries
In practice, the number of omissions usually exceeds the number of duplicates (net under-counts) 11

12. Coverage errors Erroneous Inclusions This includes:
Housing units, households and persons enumerated while they should have not been enumerated (e.g. babies born after the census reference date)
Housing units, households and persons enumerated in a wrong place 12

13. Coverage errors Gross error
This is the sum of duplications, erroneous inclusions and omissions
Net error
This is the difference between over-counts and under-counts
Net census under-count exists when number of omissions exceeds the number of duplicates and erroneous enumerations
Net census over-count is the opposite 13

14. Content errors
Every phase of census data collection and processing has the potential for introducing errors into the census results
The interviewing operation in which enumerators and respondents can make errors
Many other operations such as coding, editing where the personnel or the procedures cause errors which affect the census content
Content errors add bias and nonsampling variance components to the total mean square error of a census statistic 14

15. Methods for evaluation of census errors
Single Source of Data
Demographic analysis of the census
Interpenetration studies
Multiple Sources of Data
Non-matching studies
Demographic analysis using previous censuses
Comparison with administrative sources or existing surveys
Matching studies
Post Enumeration Surveys
Record checks 15 15

16. Single Source of Data Demographic Analysis of the Census
Average number of persons per household
Sex- and age- ratios
Tabulations...
For an overall assessment of quality:
an age pyramid is a standard method
stable population analysis can be undertaken as long as assumptions pertaining to constant fertility and mortality and no migration are met, for countries with declining mortality a quasi-stable model may be appropriate 16

17. Single Source of Data
Strengths and weaknesses
Methods that depend on a single data source provide less insight into the magnitude and types of errors in the census data
The advantage is that the methods using such sources do not require additional data to be collected
No need for sophisticated matching although this is also a limitation
It provides a general impression of quality of the census data 17 17

18. Single Source of Data Interpenetration studies
Method involves drawing subsamples, selected in an identical manner, from the census frame
Each subsample should be capable of providing valid estimates of population parameters
Assignment of personnel (i.e. enumerators, coders, data entry staff, etc.) is done randomly
The method helps to provide an appraisal of the quality of census information and procedures 18 18

19. Single Source - Interpenetration studies
Strengths and weaknesses
Gives good idea of different contribution of component errors to total census error
Helps to identify operational stages that contribute to census error, thus identifying procedural limitations in a census
Disadvantages include:
That it is an expensive operation demanding many field staff, intensive training and close supervision
Relatively complex in designing and implementation 19 19

20. Multiple Sources of Data – Non matching studies
Demographic Analysis
Comparison with data from administrative registers such as vital registration system
Demographic analysis based on successive censuses such as the cohort component method
More discussion of demographic methods is the focus of this workshop 20

21. Multiple Sources of Data – Non matching studies
Comparison with existing household surveys
In theory any probability sample of households or persons can be used to evaluate coverage and content error in a census if:
They have identical items with same concepts and definitions
They are independent from the census
They have been conducted close to the census date
There is a sufficient identification information to facilitate accurate matching 21 21

22. Multiple sources - Non-matching studies
Strengths and weaknesses
Review census results at aggregate rather than unit level i.e. provides only estimates of net census error
Evaluates very limited characteristics such as sex and age distributions
Advantage
They are relatively cheap compared to matching studies 22 22

23. Non matching methods - Demographic Analysis
Strengths and weaknesses
No additional data is needed to be collected to perform the analysis
Less costly
In statistical offices with sufficient numbers of demographers there is no need for additional staff to do the technical analysis
On the negative side these methods provide less insight into the different contributions of component errors to total error in the census
Quality of sources (Vital Statistics…) 23 23

24. Multiple Sources of Data – Matching studies
Record checks
Census records are matched with a sample of records from registration systems such as the vital registration system
The relevant respondents to the census questionnaire are traced to the time synchronized with the census
Sources include:
Previous census
Birth registrations
School enrolment
Citizen registration card
Immigration registers etc. 24

25. Multiple Sources of Data – Matching studies
Record checks contd.
Both coverage and content errors could be measured through the above comparisons
To evaluate coverage efficiently the following preconditions are essential:
A large proportion of census population should be covered in registration system
The census and registration system should be independent from each other
There should be sufficient information in records 25

26. Multiple Sources of Data – Matching studies
To evaluate content efficiently the following preconditions are essential:
The registers system should contain some relevant items covered in the census such as age, sex, education, relationship, marital status etc.
Definitions of items should be identical between the census and the registers
Countries that have used record checks include: Denmark, Finland, Norway, Sweden and Canada 26

27. Multiple Sources of Data – Matching studies
Strengths and weaknesses
It provide separate estimates of coverage and content error
Prospects of evaluating more characteristics compared to what can be done with non-matching studies
Challenges
Calls for high level technical skills including managerial
Matching is expensive 27 27

28. Multiple Sources of Data – Matching studies
Post Enumeration Survey (PES)
Consists of two separate coverage studies :
A survey conducted using a sample frame independent of the census. Persons from this survey are matched to the census to estimate the number of persons missed in the census
A survey conducted using a sample drawn from persons enumerated in the census. This sample is re-enumerated to determine if the sample person or unit was erroneously enumerated (inc. erroneously located) 28

29. Multiple Sources of Data – Matching studies
Post Enumeration Survey (PES) contd.
Results can be used to evaluate the reliability of some characteristics such as sex, age, marital status, relationship to reference person or head of the household.
For some countries the results of PES can be used to adjust some census results
Facilitates better interpretation of census results 29

30. Post enumeration survey
Advantages:
Its results can be used to independently evaluate census coverage and content error, including reliability of selected characteristics collected in a census
Incorporates matching of individuals or units between the census and PES
Its results are generally more reliable than those of the census i.e. it is justification for evaluation 30 30

31. Post enumeration survey
Challenges:
Requires highly skilled field and professional staff
Matching is complex
As it is supposed to be carried out immediately after the census at times there is lack of adequate funds and staff to implement the PES exercise 31 31

32. Thank You!