IPUMS-International Integration Process Matt Sobek Minnesota Population Center sobek@pop.umn.edu

1 2 3 4 Input material Pre-processing Standardization Integration DATA Data files Batch samples Reformat data Donation Draw sample Confidentiality A Code clean-up Verify data Confidentiality B Harmonize codes Variable programming Constructed variables METADATA Data dictionary Enumeration forms Enum. instructions Sample information Translate to English Images to editable files Ipums data dictionary Tag enumeration text Document unharmonized variables Variable descriptions Sample design

Matt Sobek Minnesota Population Center sobek@pop.umn.edu End Matt Sobek Minnesota Population Center sobek@pop.umn.edu

Batch Samples In spring we identify the samples to integrate the following year. Samples are processed as a group – one per year. The entire batch of samples is processed through each stage before we proceed to the next step. There is little flexibility in the work process. If a sample is not available for processing during the earliest stages of integration, it cannot be included in the data release for that year.

Original Input Data Some examples of differing file formats: SPSS and SAS system files Redatam-format IMPS format Records that combine household and person characteristics Separate files for persons, households (and dwellings, buildings) Different types of records (mortality or migration) Separate files for different administrative units

Reformatting: Original Data File

Reformatting: Data File after Reformatting

Reformatting: Rectangular Sample (Person records only; household data duplicated on person records) geography housing person (head) person (child) person (spouse) geography housing person (head) person (child) geography housing person (head) person (spouse) person (child) (Brazil 1980)

Reformatting: Dwelling-Household-Person Sample (Separate dwelling and household records) dwelling household dwelling household person (head) person (spouse) person (child) person (head) person (spouse) person (child) dwelling household household person (head) person (child) person (head) person (child) dwelling household person (head) person (spouse) dwelling household person (head) person (spouse) (Chile 1992)

Reformatting: Merge Household and Person Files Household File serial 001 geog & housing serial 002 serial 003 serial 001 household serial 001 head spouse serial 002 household serial 002 head child Person File serial 001 head spouse serial 002 child serial 003 serial 003 household serial 003 head (Brazil 2000)

Reformatting: Persons not Organized in Households (Individuals only; not organized in households) geog person housing geog person housing household person household person household person household person household person (Mexico 1960)

Donation and Error Correction Data are tested for errors that affect structural integrity, such as merged households, unmatched person and household records, corrupted records, etc. Such errors often do not affect tabulations, but create inconsistencies across records within households that affect sophisticated analyses. Some problems can be resolved with custom programming. Other problems are resolved by donating (substituting) a donor household for the corrupted one. Households are divided into strata based on predictor variables. Donors are drawn from the same strata as the corrupted household, ensuring they share key characteristics. If a sample is drawn from the full census, a substitute donor record is used; if we are already starting with a sample, the donor record is duplicated. A flag indicates that a record was duplicated.

Drawing a Sample About one-third of IPUMS samples are drawn from full-count data. After reformatting, we draw a systematic sample of every Nth dwelling to yield the desired sample density – typically 10%. If the input data are not full-count (for example, they include only the long-form records), the sample design might have to account for differing sample densities between areas. Very large dwelling units (over 30 persons) are sampled at the individual level – not as intact units – in order to reduce sampling error. Every Nth individual is taken.

Confidentiality Measures: A Swap a small percentage of cases between geographic areas. Reorder households within geographic areas. Suppress low-level geographic variables. Suppress any variable deemed too sensitive by the National Statistical Office. Encrypt all versions of the data prior to the imposition of these confidentiality measures.

Code Clean-Up: Recoding Unharmonized Variables Recode the input variables to conform to some basic standards for treatment of missing values, etc. Recode stray values into a consolidated missing category as appropriate. Convert non-numeric characters to numeric. Most recoding is performed using a data translation matrix like the one below for Marital Status in 1984 Costa Rica. If the recoding requires more complex logic, use custom programming.

Verify Data: Unharmonized Variables Examine the marginal frequencies of every input variable. Analyze the data universe for each variable – the population at risk of having a response. Determine the theoretical universe from enumeration materials or other documentation, then empirically determine any discrepancies from that universe. Document the universe for each variable and any other observations.

Confidentiality Measures: B Recode geographic units to ensure small localities cannot be identified (typically those with fewer than 20,000 persons). For recent censuses: Identify cells that represent very small numbers of persons in the population. Code them to a residual category or combine them. Top- or bottom-code continuous variables that have a long tail that could identify small subpopulations. Suppress specific categories of variables as requested by the National Statistical Office.

Translation Matrix for Marital Status Harmonize Codes: Translation Matrix for Marital Status China 1982 Colombia 1973 Kenya 1989 Mexico 1970 U.S.A. 1990

Variable Programming Some variable manipulations are too complex to be handled using the translation matrix tables. Typically these involve continuous variables or recoding logic that refers to multiple variables. This programming is written in C++.

Constructed “Pointer” Variables (Simple household) Spouse’s Pernum Relate Age Sex Marst Chborn 1 head 46 male married n/a 2 spouse 44 female 3 aunt 77 widow 7 4 child 15 single 5 13 6 11 Location   2 1 Mother’s Father’s Pernum Relate Age Sex Marst Chborn 1 head 46 male married n/a 2 spouse 44 female 3 aunt 77 widow 7 4 child 15 single 5 13 6 11 Location   Location   2 1 2 1 2 1 (Colombia 1985)

Constructed “Pointer” Variables (Complex household) Spouse’s Mother’s Father’s Pernum Relationship Age Sex Marst Chborn 1 head 53 female separated 6 2 child 28 male single n/a 3 22 4 21 5 25 married child-in-law 7 grandchild 8 9 non-relative 32 10 11 Location   Location   Location   1 6 5 5 6 5 6 9 9 (Colombia 1985)

Original Data Dictionary – Kenya 1989

Original Data Dictionary – Romania 1992

Original Data Dictionary – China 1982

Original Data Dictionary – Mexico 1990

Enumeration Form: Original File

Enumeration Instructions: Original File (Mexico 1990)

Sample Information – from Statistical Office Sample information is difficult for the IPUMS project to collect. Often only limited information can be gleaned from available documentation. It is extremely helpful when countries collate the information themselves, as was done below by the Netherlands:

Translate Documents to English Many countries provide their census documentation in English. For those that do not, the IPUMS project hires translators from around the world. Often these are persons currently or formerly associated with National Statistical Offices. Some common languages are translated by staff in Minnesota.

Editable Enumeration Form – In English

IPUMS Data Dictionary

XML-Tagged Enumeration Form

Document Unharmonized Variables The enumeration form and instruction text provides most of the documentation for the unharmonized input variables. Other documentation is written as needed to clarify the interpretation of the variable for users. We also empirically determine the universe of persons or households with valid values for each variable.

Variable Description (Literacy)

Sample Design