Data Processing 1890-1964 Hollerith 1921 Hollerith Electric Tabulator 1890 IBM (Hollerith) System 360 1964

The price of storage has fallen by a factor of 12 million! Byte Magazine, July 1980 $1170 per MB, 2010 dollars Amazon.Com, January 2010 0.0095 cents per MB, 2010 dollars

The price of storage has fallen by a factor of 35 million! Byte Magazine, July 1980 $1,207 per MB, 2011 dollars Amazon.Com, September 2011 $0.000035 per MB, 2011 dollars

The storage in my new phone would cost (in 2011 dollars) $7,571 in 2001 $212,040 in 1991 $3,796,800 in 1981 $56,168,800 in 1971 $1,233,179,000 in 1961

Sustainable Digital Data Preservation and Access Network Partners (DataNet) 2007 National Science Foundation Solicitation $100 Million set aside for first phase DataNet Goal: provide reliable digital preservation, access, integration, and analysis capabilities for science and/or engineering data over a decades-long timeline.

TerraPop Goals A framework for global-scale spatiotemporal data Census microdata Government land-use statistics Land cover data from satellite imagery Historical climate records (temperature, precipitation, cloud cover)

TerraPop Goals Provide global-scale data on population, land use, land cover, and climate from the 19th century to the present Interoperable across time and space Accessible to researchers and the public

TerraPop Goals Create a sustainable organization that can guarantee preservation and access over multiple decades Organizational sustainability Financial sustainability Technological sustainability Organization will be a collaboration with major data archives and universities and will eventually assume responsibility for IPUMS

Terra Populus Terra Populus geology transportation paleontology criminology hazards Population Climate Terra Populus pollution Land Use Land Cover health economics politics biology hydrography/ water resources

Sustainability Organizational sustainability Financial sustainability Technological sustainability

Organizational Sustainability Hybrid approach: University based like a library International membership base ultimately responsible for governance

Financial Sustainability University endowment ($1 million) University will migrate technology “indefinitely, as long as it is needed” Three archival partners contribute to preservation Institutional member subscriptions Grants and contracts

Technological Sustainability Software and hardware continuously renewed Virtualization/private cloud/evolvable cyberinfrastructure Preservation standards (ICPSR, CIESIN, CESSDA/UKDA)

TerraPop Time Dimension

Khartoum, CBS-Sudan

Dhaka, Bangladesh Bureau of Statistics 18

Three Formats of Integrated Data Census microdata with attached characteristics describing land use, land cover, and climate for local areas Aggregate data and for administrative districts with tabulated population data and environmental characteristics Gridded data with characteristics of population and environment

Timeline September 2011: Funding begins February 2013: Functional prototype July 2015: Application for years 6-10

Reduced Scope 60% reduction in budget Not a research project: Infrastructure project

Reduced Scope

Additions Data inventory: Inventory and evaluate land-use, climate, and small-area census data for inclusion in the infrastructure. Data acquisition: Collect and preserve selected datasets relating to population and the environment. Boundary files: acquire or develop files, account for boundary changes

Organization Scientific Leadership Group Executive Committee Policy & Planning Committees Sustainability Preservation and Protection Data Acquisition Information Technology Committees User Interface Data Processing Scientific Web Communities Advisory Board

Scientific Leadership Group Steven Ruggles, Principal Investigator/Program Director. Victoria Interrante, Co-Principal Investigator Steven Manson, Co-Principal Investigator Jaideep Srivastava, Co-Principal Investigator Shashi Shekhar, Co-Principal Investigator Jonathon Foley, Director of the Institute on the Environment, University of Minnesota Wendy Pradt Lougee, University of Minnesota Librarian Robert Chen, Director, Center for International Earth Science Information Network, Columbia University George Alter, Director, Inter-University Consortium for Political and Social Research

Executive Committee Cathy Fitch, Executive Director Matt Sobek, Director of Microdata Integration Pete Clark, Director of Software Development Dave Van Riper, Director of Spatial Data TerraPop Project Manager

Sustainability Steve Ruggles Wendy Lougee George Alter Bob Chen Cathy Fitch

Preservation and Protection Steve Ruggles George Alter Nancy McGovern Bob Downs John Butler Matt Sobek Cathy Fitch

Data Acquisition Dave Van Riper Steve Manson Jon Foley Bob Chen Susana Adamo Bob Downs

User Interface Pete Clark Vicki Interrante Jaideep Srivastava Steve Manson Steve Ruggles

Data infrastructure Pete Clark Matt Sobek Steve Ruggles Jaideep Srivastava Shashi Shekhar

Scientific Web Communities Cathy Fitch Pete Clark John Butler Jaideep Srivastava Loren Terveen