HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co Outline Open Archival Information System reference model Needs of data users, producers, and archivists

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co OAIS Information Packages Content Information Preservation Description Information Descriptive Information About Package 1 Package 1 Packaging Information Adapted from CCSDS B-1

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co OAIS Information Packages - Content Info. Data Object - the information to be preserved Representational Information - allows a user in the designated community to understand the data without consulting an expert  Structure / Syntax  Content / Semantics

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co OAIS Info. Packages - Preservation Description Provenance - documents the history of the object (e.g., Instrument descriptions, Processing history, sensor descriptions, Instrument & mode, Software interface specs, etc.) Reference - documents object identifiers and their generation mechanisms (e.g., Journal references, OID, Mission, Instrument, Data set Title, Parameters, etc.) Fixity - documents methods used to ensure there are no undocumented changes (Method, Algorithm, checksum values, etc.) Context - the relationship of the object to its environment (Calibration history, related data sets, funding history, etc.)

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co OAIS Info. Packages - Packaging Information Either logically or physically binds the Content Information and the Preservation Description Information into an object stored in a defined location Generally changes when migration occurs

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co OAIS Info. Package - Descriptive Information Information that allows users to find, assess, and retrieve/order information of interest (e.g., the catalog, indexes, etc.)

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co OAIS Information Packages CCSDS B-1

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co Data User Needs Searchable on their terms Amounts of data ranging from the miniscule to the entire contents of the archive and anything in between Data formatted their way With enough supporting information to be useful to them without having to consult an expert

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co Data Producer Needs An archive ready, willing, and able to accept their data in whatever format, on whatever media, with whatever packaging can best be accommodated by both parties High volume data producers need to be able to work with the archive to define automated interfaces

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co Archive Needs Data producers willing to help develop the information needed by users An good long-term archive format

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co What is a Good Long-Term Archive Format? Per a recent paper by Mike Folk and Bruce Barkstrom  Ease of archival storage  Ease of archival access  Usability  Data scholarship enablement  Support for data integrity  Maintainability and durability

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co What Makes a Good File Format? Per Eric Raymond, in “The Art of Unix Programming”, Addison-Wesley, 2004  Transparency  Interoperability  Extensibility  Storage economy He argues that the best general purpose file format is text He also argues that the only good justification for binary data is with very large data sets

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access Presented by R. Duerr at the HDF and HDF-EOS Workshop VIII, October 26-28, 2004 Aurora Co Disaster-Proofing Your Data If you can’t keep a data set as text, then at least keep the representational information in a human readable format (preferably right with the data itself)