1. Session 15. Quality framework of cRVS
Civil Registration Process: Place, Time, Cost, Late Registration
UNITED NATIONS STATISTICS DIVISION
Workshop on Operation of Civil Registration, Vital Statistics and
Identity Management Systems for East Asian Countries
Hanoi, Vietnam, November 2017

2. Evaluation is essential
Vital Statistics System
Live births
Deaths
Fetal deaths
Marriages
Divorces
Annulments
Judicial
separations
Adoptions
Legitimation
Recognition
Health services
Certification of
cause of death
Authorized
institutions
Courts
Civil
Registration, including population registers
Principles:
Compulsory
Universal
Continuous
Confidentiality
Vital
Statistics
Compilation
Processing
Validation
Quality control
Dissemination
Complementary/
Interim sources
Population census
Surveys
Sample registration
areas
Additional
administrative
sources
Coronary
Police
Registries
Health records
National IDs’
Electoral lists
Passports …
Even if registration of an event is done very carefully, there is a wide set of people involved (health personnel, admin personnel, relatives informing of event, funeral homes) that can make mistakes, and these mistakes will accumulate and will be reflected in the quality of the vital statistics.
Data flow – plenty of actors involved since the occurrence of the event until the compilation of VS = plenty of opportunities to introduce errors or for data to be lost.
This is why evaluation is essential

3. Evaluation is essential
Adequately funded evaluation activities are essential
For improving systems that have deficiencies
For maintaining systems that function satisfactorily
Findings from data quality evaluation are useful to
identify and quantify data biases and correct them to derive more reliable vital statistics and indicators
identify systemic issues and make interventions to prevent weaknesses in data quality
Maintaining high standards of quality in CRVS should be a major and continuing concern.
For improving
For maintaining

4. Evaluation is essential
Strong mandate in Sustainable Development Agenda
Indicator : Percentage of children under 5 whose births have been registered
Indicator : Proportion of countries that… (b) have achieved 100 per cent birth registration and 80 per cent death registration
Other 18 indicators that use CRVS data as direct input
If these motivations were not enough, we also have a strong mandate from the Sustainable Development Goals
In particular indicators – registration of births among children under 5 years old
– statistical capacity of countries, it combines census conduction with vital events’ coverage
20 indicators in total that use CRVS as direct input (11 within Goal 3 on Health and Wellbeing)
Many more other goals that have tight dependency on the quality of CRVS because they will use it a input for population estimates, rates, ratios and other important figures.
1.3.1 Proportion of population covered by social protection floors/systems, by sex, distinguishing children, unemployed persons, older persons, persons with disabilities, pregnant women, newborns, work-injury victims and the poor and the vulnerable
1.5.1 Number of deaths, missing persons and persons affected by disaster per 100,000 people
Maternal deaths per 100,000 live births (maternal mortality ratio)
Proportion of births attended by skilled health personnel
Under-5 mortality rate (deaths per 1,000 live births)
Neonatal mortality rate (deaths per 1,000 live births)
Mortality of cardiovascular disease, cancer, diabetes or chronic respiratory disease
Suicide mortality rate
Number of road traffic fatal injury deaths within 30 days, per 100,000 population (age-standardized)
Adolescent birth rate (aged 10-14; aged 15-19) per 1,000 women in that age group
3.9.1 Mortality rate attributed to household and ambient air pollution
3.9.2 Mortality rate attributed to unsafe water, unsafe sanitation and lack of hygiene
3.9.3 Mortality rate attributed to unintentional poisoning
Proportion of women aged years who were married or in a union before age 15 and before age 18
Number of deaths, missing persons and persons affected by disaster per 100,000 people
Number of deaths, missing persons and persons affected by disaster per 100,000 people
Number of victims of intentional homicide per 100,000 population, by age group and sex
Conflict-related deaths per 100,000 population, by sex, age and cause
There are 2 indicators particularly on birth and death registration completeness:
Proportion of children under 5 years of age whose births have been registered with a civil authority, by age
Proportion of countries that (a) have conducted at least one population and housing census in the last 10 years; and (b) have achieved 100 per cent birth registration and 80 per cent death registration

5. Confidentiality and privacy
Evaluation exercises should be conducted with a degree of independence
So that an objective assessment can be achieved
Systemic issues affecting data quality should be discussed in terms of processes rather than specific institutions and/or personnel
Findings from evaluation should be discussed and utilised in a constructive environment.

6. Quality basic framework
Quality assurance
Quality assessment
Specific studies for specific questions
Coverage of registration of vital events
Accuracy of variables
Overall functioning of sub-systems
Can be ad hoc or regular exercises
Encompasses each stage of CRVS operations
Identifies bottle necks
There are two elements in the evaluation framework that are complimentary to each other: quality assurance and assessment.
Quality assurance deals with the process for producing information.
Quality assessment, on the other hand, has more to do with information that has already been produced (with the product, not the process).
Quality assurance:
encompasses each stage of the operations of civil registration and vital statistics systems (collection, transmission to electronic format, processing and dissemination)
bottle necks
Quality assessment:
entails specific studies that aim to answer specific questions. These questions could relate to:
the coverage of the registration of a vital event at the country level or in a smaller area;
the accuracy of one of the variables recorded in vital statistics;
or the overall status of civil registration and vital statistics systems.
Quality assessment can be conducted regularly or on an ad hoc basis.

7. “Comprehensive assessment”
Quality assurance
Ensure that:
All vital events are registered without duplication
All related information is recorded
Information is compiled, validated and processed
Vital statistics are released in timely manner
Focused on:
Processes
Functions
Protocols
WHO tools
“Rapid assessment”
“Comprehensive assessment”
Quality assurance:
to ensure that:
all vital events occurring are registered without duplication,
that all related information is accurately recorded,
and that the compilation and processing of recorded vital events result in the proper and timely production of vital statistics
E.g. The tools developed by the WHO, called “rapid assessment” or “comprehensive assessment” of CRVS refer mainly to quality assurance, given that they are focused on processes and protocols.
the registration authority must ensure that:
(a) all local registration areas carry out the required functions;
(b) every vital event has a record in the system; and
(c) all local offices transmit the records to a higher-level registration office, according to procedures

8. 2. Correctness or Accuracy
Standards
1. Completeness
2. Correctness or Accuracy 3.
Availability 4.
Timeliness
The quality of data should be measured according to the standards of completeness, correctness (or accuracy), availability and timeliness
So, any evaluation must address these four standards.

9. * Every vital event is registered
Standards
1. Completeness
* Every vital event is registered
* Statistical report is filed for every registered event
* Coverage error
2. Accuracy
Completeness:
Deviation is measured by “coverage error”
This (coverage error) is a major challenge facing CRVS and is usually uneven amongst different geographical areas, or population sub-groups. Rural areas and small settlements are the most affected.
4. Timeliness
3. Availability

10. Standards 2. Accuracy 1. Completeness * Every data item is filled
* Data items are accurately filled
* Content error
* Correctness or accuracy:
Means that there are no response errors or missing items
Deviation is measured by “content error”
Content error is most common in those questions that are not strictly necessary for the civil registration procedure. For ex, items regarding socioeconomic status, such as educational attainment
4. Timeliness
3. Availability

11. Standards 2. Accuracy 3. Availability 1. Completeness
* Data and statistics are available to users in a friendly format
* Difficult to satisfy, as demands have grown
* Availability:
Relates to disseminating information that is relevant to a range of different types of users, according to their needs and demands. This standard has become more difficult to fulfil, given that technological progress has boosted users’ exigencies.
4. Timeliness

12. Standards 2. Accuracy 4. Timeliness 3. Availability 1. Completeness
* CR: events are registered within time limit and statistical reports are filed according to schedule
* VS: prompt dissemination
3. Availability

13. Quality assessment methods
Direct methods
Matching of records
Indirect methods
Demographic analysis
The P&R distinguish 2 types of assessment methods: direct & indirect.

14. Quality assessment. Direct methods
Matching of records
Match registration records with records from an independent source
Matching of records
A direct method can provide useful information on the sources of underreporting (depending which list is used to match), and can also improve coverage by identifying unregistered vital events.

15. Quality assessment. Direct methods
Matching:
Birth registration with death registration
limited to infants deaths
can be carried out routinely
With administrative records
a variety of sources can be used
however, none is complete
useful to detect certain type of underreporting
Birth registration match with death registration: limited to infants deaths; matching adult death registration with their birth is extremely difficult.
can be carried out routinely
Admin. records: a variety of sources can be used, such as school enrolments, and hospital, vaccination, baptism and burial records. However, none of these sources is complete, so a particular matching exercise should not be used to estimate level of completeness. In any case, it can be useful to detect certain type of underreporting (depending on the source used to match registration records).

16. Quality assessment. Direct methods
Matching:
Lists from population censuses and surveys
compiled from questions on births and deaths
can lead to an estimate of completeness
national or sub-national level
Dual records system
a particular case of the lists
survey specifically to collect information on vital events
the two sources are confronted
Lists from population censuses and surveys: these lists are compiled from questions on births and deaths and then used to match against civil registration records. This type of exercise can indeed lead to an estimate of completeness either at national or sub-national level [question on births or deaths in household during the last 12 months]
Dual records system: is a particular case of the lists, where as a special survey is set up specifically to collect information on vital events. Unlike the other lists, it involves going to the field to collect info. Then, the two sources are confronted in a 2 × 2 contingency table showing the matching results of events from the Survey and from the civil registration system.

17. Quality assessment. Direct methods
Matching basic logic:
Civil Registration
Survey/Census
Result
Case 1 X
Matched
Case 2
Not in survey
Case 3
Not in CR …
Case n-1
Case n
Result
Count
Matched
1000
Not in survey
120
Not in CR
230
Dual records system:
For each case, we record whether it is contained in the CR system and in the special survey or list being used.
There will be three sets of cases: events recorded in both sources (matched), events only recorded in CR registration and events only recorded in the special survey.
We make counts of the totals in each of sets of cases. How many were successfully matched, how many are only in the CR and how many appear only in the survey.
The fourth set will be those cases which are in neither source. That is what we are trying to compute.
Missing in both ??
Case 4
Missing in both

18. Quality assessment. Direct methods
Matching basic logic:
Survey /Census
Civil Registration
Yes No
Total
Matched
Not in CR
M+NR
Not in survey
Missing in both
M+NS N
Chandrasekaran-Deming formula
Dual records system:
Applying the Chandrasekaran-Deming formula, which is the Maximum Likelihood Estimator of N, it is possible to compute the proportion of events not recorded in neither source.

19. Quality assessment. Direct methods
Matching basic logic:
Chandrasekaran-Deming formula
Survey/Census
Civil Registration
Yes No
Total
1000
230
1230
120
Missing in both
1120 N
147
Dual records system:
Applying the Chandrasekaran-Deming formula, it is possible to compute the proportion of events not recorded in neither source.
First we estimate the total number of cases
And use it to compute the blank cells we had in out working table, once we have all the margins, it is possible to compute the estimated number of missed records.
Finally, with the estimated figure of cases that were missed in both sources, we are able to estimate the undercoverage of the CR (and the VS if these are compiled from CR data)
Undercoverage of 1.96% = Coverage/completeness of 98.04%
257
=1377
Missing in both
?? = 27

20. Quality assessment. Indirect methods
Comparison of trends
Delayed registration
Questions on birth registration in surveys or censuses
Comparison with census data
If at least two censuses: balancing equation, Lexis diagram
If only one census: compare aggregates
Methods for incomplete data
Manual X
Tools for Demographic Estimation (online and print update of Manual X,
Demographic analysis
Indirect methods => Demographic techniques
Comparison of trends: comparing aggregated numbers of vital events across time (month to month, year to year, etc). In most cases, the total numbers should not differ greatly from one period to the next. The method is easy to apply for routine auditing and can be used by local registrars for monitoring. On the other hand, it is not possible to estimate the number of unregistered events by simply comparing trends.
Delayed registration: monitoring interval between date of occurrence and date of registration . The proportion of delayed registrations provides a rough measure of underreporting in previous time periods.
Questions in sample surveys on birth registration: typically implemented by UNICEF in their MICS surveys, asking whether children under 5 in the hh have birth certificates, or whether they are registered. This provides an estimate of birth registration coverage, although registration does not always translate into statistics.
Comparison with census data:
* if at least two censuses are available: using the balancing equation one can compare intercensal growth with intercensal births, deaths and net migration. Delta=B+I-D-E. Assuming census figures and migration figures are accurate (or if migration is negligible), the difference between both sides of the equation will be due to underregistration (or overregistration). This technique will yield only a rough estimate, as it will be impossible to separate out the underregistration of births and deaths.
* if at least two censuses are available: constructing a Lexis diagram with births and deaths by age from CR and population from census, expected deaths can be compared with registered deaths. This technique will result in an estimate of death underregistration.
* if only one census is available: comparison of aggregated numbers (of births and infant deaths) from census and civil register can be done.
* if only one census is available: it is also possible to evaluate the completeness of birth registration by comparing the number of under 1 year olds to the number of live births and infant deaths registered during the 12 months prior. The limitations to this approach are that there are other errors at play, in addition to underregistration, such as age wrongly stated and enumeration errors during the census.
Methods for incomplete data: with the methods described in both Manual X and its recent update, estimates of births and deaths will be produced which can be compared with registered events.

21. Direct or indirect ? Advantages Limitations Direct methods
More accurate assessment of registration completeness
May indicate sources of under or overregistration
Can be applied at any geographical level
Accuracy is affected by the choice of the second source of records
True independency of the second source is unlikely
Matching criteria difficult to find if there is no ID number
If manual: time consuming
If automated: computer algorithms can get too complex
Cost
Direct methods: it is advisable to verify a sample of matched records manually, to spot errors in matching.
Indirect methods: assumptions such as stable population (constant fertility and mortality), closed population (no migration), no error in age reporting (for both deaths and population), no selection bias in age reporting or missing data (for deaths).
In principle, direct methods provide information on quality of registration, in terms of coverage and accuracy. Indirect methods (with the exception of questioning in sample surveys about birth registration) work with statistical tabulations generated from civil registration. Therefore, most indirect methods assess quality of vital statistics.
Indirect methods
Prompt assessment of vital statistics completeness
Several can be applied at various geographical levels
Some have assumptions that may not hold
Some require reliable data from two censuses
Accuracy is affected by the degree of census completeness

22. Choosing the appropriate method depends on:
Direct or indirect ?
Objectives
Degree of precision
Timeliness
Type of event
Resources
Choosing the appropriate method depends on:
Final considerations. Whether direct or indirect, will depend on the needs of the analyst and the resources available. In some cases, a blend of direct and indirect methods might be appropriate.
Factors to take into account
Objectives: How the study findings will be applied may dictate the choice of method. Determining whether completeness or accuracy or both want to be evaluated, will help choose the assessment technique.
If the objective is to promote overall registration improvement or have routing monitoring, it may be sufficient to address coverage problems in general terms. In this case, indirect evaluation methods will suffice.
If the goal is to identify and eliminate specific coverage problems, direct methods are more appropriate.
Direct methods are appropriate if accuracy is being evaluated (such as cause of death).
Degree of precision: If reporting of vital events is grossly deficient, an estimate obtained through an indirect method usually will suffice.
But if the major problems have been resolved but significant minor problems still remain, direct methods may be the best.
Timeliness: time frame for results.
If the objective of the study is to verify whether a problem is developing, the results need to be made available as soon as possible. This calls for the use of an indirect method.
If the study is part of a long-term registration development plan, direct methods may be considered.
Type of event: a particular event, or a specific subset (infant or maternal deaths, e.g.) can call for specific methodologies.
Resources: level of funding, availability of skilled analysts and their level of expertise, types of data sources to be used and their quality. All plays an important role.
Cost of direct evaluation may be very high, particularly if field work is required

23. Thank You Спасибо 谢谢
Gracias Merci
ﺷﻜﺮﺍﹰ

24. Quality assessment. Direct methods
Practical example: Health services of the state of Queensland, Australia
Primary source: Perinatal Data Collection
Secondary source:
Birth registration
Linkage file:
file containing person identifiers from various admin. sources
Matching of records: this is an example of matching civil registration with other administrative records, in this case health records.
Period of study: Cases where the recorded date of birth of the baby was between 1 July 2010 and 30 June 2012
Primary source: The perinatal data contains information about all live births in Queensland and all stillbirths of at least 400g birth weight or 20 weeks gestation. The data is maintained and disseminated by the Health Statistics Branch, Queensland Health for national reporting purposes, for use in key performance indicators, to assist with service planning and for research into perinatal and obstetric care and outcomes.
Secondary source: The Queensland Department of Justice and Attorney-General maintains a register of all births (live, and after April 1989, still born of at least 20 weeks gestation or 400g birth weight)
Linkage file: A file containing person identifiers for various administrative data collections (including Perinatal Data Collection and birth registration data) managed by the Health Statistics Branch (HSB), Queensland Health.

25. Significant differences in linkage according to ethnic groups
Direct methods. Practical example: Health services of the state of Queensland, Australia
Some results
2.7% of Perinatal Data records could not be linked to Registration data.
Significant differences in linkage according to ethnic groups
Remote and very remote geographical areas also had high rates of under-registration
Indigenous mothers
15-18% undercoverage
Non-indigenous mothers
1.8% undercoverage
(2.7%) of PDC records could not be linked to the registration data.
There were significant differences in linkage between Indigenous mothers and non-Indigenous mothers (between 8 and 10 times more under-registration for indigenous mothers)
Remote and very remote geographical areas also had high rates of under-registration, however, this effect was only found for births to Indigenous mothers in these areas, with no obvious differences in under-registration by remoteness for non-Indigenous mothers.
There was a slight differential by the marital status of the mother.
Also younger non-Indigenous mothers were less likely to register births, while the proportion was constantly high across all age groups for Indigenous mothers.