Building Historical Social Science Infrastructure: Data Integration Projects of the Minnesota Population Center Robert McCaa and Steven Ruggles Minnesota Population Center

How to get data (once approved) (also SAS, STATA) 1. Access web-site study documentation 2. Make and submit extract 3. Get extract ready 3. Get extract ready 4. Retrieve extract 5. Decompress extract 6. Analyze using stat. package

Outline History of Public Use Census Microdata IPUMS IPUMS-International NAPP Differences among the projects –Data format –Harmonization –Administration, work processes, and legal constraints

History of U.S. Public Use Census Microdata The 1960 One-In-One-Thousand Public Use Sample The 1970 Public Use Samples DUALabs, Beresford, and the harmonized and expanded 1960 sample The new historical samples: Preston, Winsborough, Ruggles The 1980, 1990, and 2000 PUMS: incompatible

Table 1. Census files incorporated in the original version of IPUMS 1991: eight census years, four investigators, six performance sites, seven record layouts

IPUMS : SHRL Common format FORTRAN programs –Limitations: lost information, false cognates, poor documentation, expensive custom datasets IPUMS was an attempt to do it right –Single harmonized database, comprehensive integrated documentation, no lost information –Beta release 1993, full public release 1995 Internet dissemination –ftp in 1993, web-based interactive extraction in 1995

Table 2. Current and Planned IPUMS-USA Data Files

IPUMS-International After 1960, most censuses around the world were tabulated by computer McCaa decided that IPUMS model should be applied to other countries Began with a project for Colombia, then in 1999 NSF Infrastructure grant to add six more countries : three major new grants to increase database to 50+ countries

IPUMS-International Tasks Inventory and preservation of data and documentation Processing (standardizing format, correcting format errors, drawing samples, adding confidentiality protections, harmonizing codes, etc.) Documentation (especially comparability) Dissemination—obtain licenses that allow us to disseminate data for educational and scholarly usae, and set up secure web-based dissemination system

Table 3. Current IPUMS-International Samples

IPUMS-International, August 2005 dark green = disseminating medium green = harmonizing light green = negotiating Mollenweide projection

Table 4. Status of IPUMS-International Countries

North Atlantic Population Project IMAG 1999: LDS data for Britain, Canada, U.S. Minneapolis 2000: meetings to define scope of a harmonization project –Added Norway and Iceland –Adopted decentralized structure with coding work carried out at seven sites, coordination and programming at Minneapolis : preliminary datasets for all countries released : planned expansions (funding pending)

Table 5a. Phase I NAPP datasets

Table 5b. Phase II NAPP datasets

Differences Data Format Problems Harmonization Project administration and work process Ownership and dissemination restrictions

Merging the databases Current compatibility and incompatibilities Two formats Integration of web access tools

Thank you