United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Introduction to Census Archiving Session 3

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 What is data archiving?  Data archiving refers to the long-term storage of electronic data and its affiliated documentation (data collection instruments, study methods, descriptive information for variables and files)  Archiving is not merely making copies of data  Archiving/preservation is keeping digital data alive and accessible for long-term use  Involves adding value to and enhancing the usefulness of the data for secondary analysis  Therefore, archiving should be planned and undertaken with view point of potential long-term uses and users of the data

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Why archive the data? A: Responsibility of data producer  NSOs generate vast amounts of data which represent significant investment by country and have considerable value for present and future data uses  But, data dissemination (hence utilization) generally weakest part of census operation oDissemination of micro-data even more limited  Data producers have responsibility not only to collect and disseminate data but also to ensure their long-term usability through proper archiving and preservation  Archiving is not just safe-keeping of data, but to promote and disseminate statistics as widely and effectively as possible

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Why archive the data? B: Need to preserve the data for long-term usage  Without proper archiving valuable data could be lost over time due to loss, obsolescence of technology, or irreversible damage Note: challenges of maintaining digital content over successive generations of technology  Archiving makes it possible for researchers to conduct secondary analyses on the data and extension of original findings

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 What does archiving involve?  Archiving encompasses a broad range of activities beyond data collection - it covers the entire data lifecycle  Data archiving is a process and not an end state  Important, therefore, to develop a plan  Start process early (before data collection) and include preservation of accurate metadata in the plan  For the long-term, require formulating an archiving and preservation policy as part of overall strategy for data sharing and utilization

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Core attributes for a data archiving programme An institutional strategy for archiving should encompass:  Organizational infrastructure: defines how much data will be archived (and shared)  Technological infrastructure: defines how the data will be archived  Resources: defines how much it will cost to archive the data  Core attributes should be balanced and work in concert in order for the program to succeed (e.g., archiving is not just IT-related but needs also proper organizational arrangements and adequate resources)

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Core attributes for a data archiving programme (contd.) oUnderstand process and what’s required in terms of steps for developing an archiving system oStrategy should be developed taking into account needs and requirements of individual data producers but complying with standards of good practice within the digital preservation community

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Organizational infrastructure  Archiving programme exists within an organizational context and as such must fit the needs, priorities, and resources of that organization  Strong, well planned and sustainable organizational setting is crucial for long-term success of archiving programme  Policies and procedures oPolicies reflect commitments and decisions regarding data archiving and sharing oMission statement of NSO that encompasses data archiving and sharing oLegal frame work defines mandate and may cover data archiving and documentation, sharing and dissemination, particularly of micro-data oDevelopment of clear policy and strategy that explicitly address archiving commitments and decisions are important for success oWritten policy and procedures help to define responsibilities and requirements for archiving, sharing and access

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Technological infrastructure  Technology (technological and procedural suitability and system security) is a means to achieving archiving objectives, i.e., it’s an enabler o Equipment o Software o Hardware o Secure environment o Staffing  Technology enables organization to meet defined requirements for archiving – should be suitable for the purpose  Decisions on technological infrastructure center on whether to build, extend, collaborate, or out-source

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Technological infrastructure (contd.)  Technology is not static - organization to actively monitor technological developments and systematically consider potential implications for archiving programme (software and hardware)  The infrastructure should plan for technological changes and their attendant costs  As digital storage media (formats and physical storage) media will ultimately become absolete: store multiple copies on different storage media and different file formats  Need to develop appropriate data migration procedures  Important to maintain a flexible preservation system that evolves to meet demands of changing technology and new and increasing user expectations

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Resource framework  Organizational and technological infrastructures are not sustainable without an on-going commitment of resources  Organization should identify appropriate resources to develop and maintain archiving programme  Resources should cover start-up, on-going and contingency funding  Sustainable funding should be designated for cost of staff, equipment, software, and storage costs - to demonstrate the organization’s commitment to programme

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Major activities in archiving of micro-data  Documentation  Annonymization  Dissemination Discussed in brief with details provided in subsequent sessions

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Role of documentation  Documentation explains how the data were collected, their content and structure and any manipulation that may have taken place  Documentation is required in order to understand and interpret the data by providing a context  Without proper documentation, data are useless  Some items for documentation: Context of data collection Data collection methods Data validation, cleaning and quality assurance procedures Descriptions of variables Definitions of codes Definitions of scientific terminology Treatment of missing values  Also allows reuse of documents for future surveys  Metadata categorized according to international standards (ISO, DDI, Dublin Core)

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Anonymization  Dissemination of micro-data posses challenges for protection of data confidentiality  Appropriate procedures should be developed to measure disclosure risks  Develop appropriate procedures for anonymization of data so as to reduce risk of disclosure of confidential information

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Dissemination  Benefits of disseminating micro-data (broadening use of existing data and increasing return on data collection investments)  BUT there are also associated costs and risks (financial cost, exposure to criticism, disclosure of confidential information)  Benefits have to be weighed against ethical and legal obligations to keep respondent information confidential  Design dissemination strategy to provide access to micro-data while implementing procedures to guard against breach of data confidentiality  Dissemination strategy should include determination of types of outputs, access methods, procedures for terms of use of the data, as well as possible penalties for breaching terms of use, etc.

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Challenges to implementation of an archiving programme  Resource constraints  Lack of specific laws/regulations and policies for archiving  Lack of institutional commitment  Inadequate IT infrastructure  Lack of appropriate skilled personnel  Implementing data anonymization procedures  Lost data in original census databases  No general guidelines for choosing a suitable platform to archive census data  Lack of mechanism to cross check the quality of archived micro- data  Integration and harmonization of data from successive censuses

United Nations Regional Seminar on Census Data Archiving for Africa, Addis Ababa, Ethiopia, September, 2011 Thank You!