IPUMS-International Integration Process Matt Sobek Minnesota Population Center

Input materialPre-processingStandardizationIntegration DATA Data filesReformat data Donation Draw sample Confidentiality A Code clean-up Verify data Confidentiality B Harmonize codes Variable programming Constructed variables METADATA Data dictionary Enumeration forms Enum. instructions Sample information Translate to English Images to editable files Ipums data dictionary Tag enumeration text Document unharmonized variables Variable descriptions Sample design

End Matt Sobek Minnesota Population Center

Original Input Data Some examples of differing file formats: SPSS and SAS system files Redatam-format IMPS format Records that combine household and person characteristics Separate files for persons, households (and dwellings, buildings) Different types of records (mortality or migration) Separate files for different administrative units

Reformatting: Original Data File

Reformatting: Data File after Reformatting

geographyhousing person (head) person (child) geographyhousingperson (head) geographyhousingperson (child) geographyhousingperson (child) geographyhousingperson (head) geographyhousingperson (spouse) geographyhousingperson (child) geographyhousingperson (child) geographyhousing person (head) person (spouse) person (child) (Brazil 1980) (Person records only; household data duplicated on person records) Reformatting: Rectangular Sample

serial 001head serial 001spouse serial 002head serial 002child serial 003head serial 001geog & housing serial 002geog & housing serial 003geog & housing serial 001household serial 001head serial 001spouse serial 003household serial 002household serial 002head serial 002child serial 003head Household File Person File (Brazil 2000) Reformatting: Merge Household and Person Files

geogpersonhousinggeogperson geogpersonhousinggeogperson geogpersonhousinggeogperson geogpersonhousinggeogperson geogpersonhousinggeogperson household person household (Mexico 1960) geogpersonhousinggeogperson geogpersonhousinggeogperson geogpersonhousinggeogperson geogpersonhousinggeogperson geogpersonhousinggeogperson (Individuals only; not organized in households) Reformatting: Persons not Organized in Households

Donation and Error Correction Data are tested for errors that affect structural integrity, such as merged households, unmatched person and household records, corrupted records, etc. Such errors often do not affect tabulations, but create inconsistencies across records within households that affect sophisticated analyses. Some problems can be resolved with custom programming. Other problems are resolved by donating (substituting) a donor household for the corrupted one. Households are divided into strata based on predictor variables. Donors are drawn from the same strata as the corrupted household, ensuring they share key characteristics. If a sample is drawn from the full census, a substitute donor record is used; if we are already starting with a sample, the donor record is duplicated. A flag indicates that a record was duplicated.

Drawing a Sample About one-third of IPUMS samples are drawn from full-count data. After reformatting, we draw a systematic sample of every Nth dwelling to yield the desired sample density – typically 10%. If the input data are not full-count (for example, they include only the long-form records), the sample design might have to account for differing sample densities between areas. Very large dwelling units (over 30 persons) are sampled at the individual level – not as intact units – in order to reduce sampling error. Every Nth individual is taken.

Confidentiality Measures: A Swap a small percentage of cases between geographic areas. Reorder households within geographic areas. Suppress low-level geographic variables. Suppress any variable deemed too sensitive by the National Statistical Office. Encrypt all versions of the data prior to the imposition of these confidentiality measures.

Code Clean-Up: Recoding Unharmonized Variables Recode the input variables to conform to some basic standards for treatment of missing values, etc. Recode stray values into a consolidated missing category as appropriate. Convert non-numeric characters to numeric. Most recoding is performed using a data translation matrix like the one below for Marital Status in 1984 Costa Rica. If the recoding requires more complex logic, use custom programming.

Verify Data: Unharmonized Variables Examine the marginal frequencies of every input variable. Analyze the data universe for each variable – the population at risk of having a response. Determine the theoretical universe from enumeration materials or other documentation, then empirically determine any discrepancies from that universe. Document the universe for each variable and any other observations.

Confidentiality Measures: B Recode geographic units to ensure small localities cannot be identified (typically those with fewer than 20,000 persons). For recent censuses: Identify cells that represent very small numbers of persons in the population. Code them to a residual category or combine them. Top- or bottom-code continuous variables that have a long tail that could identify small subpopulations. Suppress specific categories of variables as requested by the National Statistical Office.

China1982Colombia1973Kenya1989Mexico1970U.S.A.1990 Harmonize Codes: Translation Matrix for Marital Status

Variable Programming Some variable manipulations are too complex to be handled using the translation matrix tables. Typically these involve continuous variables or recoding logic that refers to multiple variables. This programming is written in C++.

PernumRelateAgeSexMarstChborn 1head46malemarriedn/a 2spouse44femalemarried3 3aunt77femalewidow7 4child15femalesingle0 5child13femalesinglen/a 6child11malesinglen/a PernumRelateAgeSexMarstChborn 1head46malemarriedn/a 2spouse44femalemarried3 3aunt77femalewidow7 4child15femalesingle0 5child13femalesinglen/a 6child11malesinglen/a Spouse’s Mother’sFather’s Location (Colombia 1985) (Simple household) Constructed “Pointer” Variables

PernumRelationshipAgeSexMarstChborn 1head53femaleseparated6 2child28malesinglen/a 3child22malesinglen/a 4child21malesinglen/a 5child25femalemarried2 6child-in-law28malemarriedn/a 7grandchild3malesinglen/a 8grandchild1malesinglen/a 9non-relative32femaleseparated2 10non-relative10malesinglen/a 11non-relative5femalesinglen/a Location Spouse’sFather’sMother’s (Complex household) (Colombia 1985) Constructed “Pointer” Variables

Original Data Dictionary – Kenya 1989

Original Data Dictionary – Romania 1992

Original Data Dictionary – China 1982

Original Data Dictionary – Mexico 1990

Enumeration Form: Original File

Enumeration Instructions: Original File (Mexico 1990)

Sample Information – from Statistical Office Sample information is difficult for the IPUMS project to collect. Often only limited information can be gleaned from available documentation. It is extremely helpful when countries collate the information themselves, as was done below by the Netherlands:

Translate Documents to English Many countries provide their census documentation in English. For those that do not, the IPUMS project hires translators from around the world. Often these are persons currently or formerly associated with National Statistical Offices. Some common languages are translated by staff in Minnesota.

Editable Enumeration Form – In English

IPUMS Data Dictionary

XML-Tagged Enumeration Form

Document Unharmonized Variables The enumeration form and instruction text provides most of the documentation for the unharmonized input variables. Other documentation is written as needed to clarify the interpretation of the variable for users. We also empirically determine the universe of persons or households with valid values for each variable.

Variable Description (Literacy)

Sample Design