HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term.

Slides:

Advertisements

Similar presentations

The Future of Scholarship in the Digital Age: The Role of Institutional Repositories Ann J. Wolpert Director of Libraries Massachusetts Institute of Technology.

Advertisements

Long-Term Preservation. Technical Approaches to Long-Term Preservation the challenge is to interpret formats a similar development: sound carriers From.

Animesh Bhattacharyya Librarian, Vivekananda Mahavidyalaya

Digital Preservation - Its all about the metadata right? “Metadata and Digital Preservation: How Much Do We Really Need?” SAA 2014 Panel Saturday, August.

From Analog to Digital: Changes in Preservation Gregor Trinkaus-Randall Digital Commonwealth Conference Worcester, MA March 25, 2010.

Future Access to the Scientific and Cultural Heritage – A shared Responsibility Birte Christensen-Dalsgaard State and University Library.

Architecture & Data Management of XML-Based Digital Video Library System Jacky C.K. Ma Michael R. Lyu.

Management is not a Natural Act Megan Winget - Co-Project Manager Managing the Digital University Desktop: Introduction and Preliminary Findings.

Knowledge Environments for Science and Engineering: Current Technical Developments James French, Information and Intelligent Systems Division, Computer.

The HDF Group Improving long-term preservation of EOS data by independently mapping HDF4 data objects Mike Folk, Ruth Aydt, Joe Lee, Binh-Minh.

Research Data Service at the IT Pro Forum HEIDI IMKER, DIRECTOR.

University of Illinois at Urbana-ChampaignHDF 1McGrath/Yang 2/27/02 Transitioning from HDF4 to HDF5 Robert E. McGrath Kent Yang.

System Design/Implementation and Support for Build 2 PDS Management Council Face-to-Face Mountain View, CA Nov 30 - Dec 1, 2011 Sean Hardman.

DSpace: Or How an Electronic Resource Management Enterprise at MIT Solved all the Problems of the Digital World. Butch Lazorchak INLS 235 Digital Library.

David Giaretta Associate Director (Development) Funders: DCC Development Digital Curation Centre a centre of expertise in data curation and preservation.

Database System Development Lifecycle © Pearson Education Limited 1995, 2005.

Overview of the Database Development Process

Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Mike Folks, The HDF Group Ruth Duerr, NSIDC 1.

Addressing Metadata in the MPEG-21 and PDF-A ISO Standards NISO Workshop: Metadata on the Cutting Edge May 2004 William G. LeFurgy U.S. Library of Congress.

September 23-24, 2014Dawn Conway, AMSR-E / AMSR2 TLSCF Lead Software Engineer AMSR-E / AMSR2 Team Lead Science Computing Facility TLSCF at UAH Dr. Roy.

University of Illinois at Urbana-ChampaignHDF 9/19/2000 McGrath 9/19/ Transition from HDF4 to HDF5: Issues Robert E. McGrath NCSA University of Illinois.

EARTH SCIENCE MARKUP LANGUAGE “Define Once Use Anywhere” INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE.

New Custodians and New Practices Digital Curation for Family History Materials IFLA-GENLOC Satellite Meeting,11 August 2011 Ross Harvey (Simmons College,

HDF-EOS Workshop VII, An XML Approach to HDF-EOS5 Files Jingli Yang 1, Bob Bane 1, Muhammad Rabi 1, Zhangshi Yin 1, Richard Ullman 1, Robert McGrath.

Exploring the Applicability of Scientific Data Management Tools and Techniques on the Records Management Requirements for the National Archives and Records.

Preserving Digital Collections for Future Scholarship Oya Y. Rieger Cornell University

Producción de Sistemas de Información Agosto-Diciembre 2007 Sesión # 8.

ESRI User Conference, August 8, 2006 Long-term archiving of geospatial data: the NGDA project Julie Sweetkind-Singer John Banning Stanford University.

December 1, 2005HDF & HDF-EOS Workshop IX P eter Cao, NCSA December 1, 2005 Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration.

HDF Converting between HDF4 and HDF5 MuQun Yang, Robert E. McGrath, Mike Folk National Center for Supercomputing Applications University of Illinois,

EARTH SCIENCE MARKUP LANGUAGE Why do you need it? How can it help you? INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE.

Integrating a Statewide Web Gateway With Digital Collections ______________________ Eric Weig and Beth Kraemer University of Kentucky and KCVL.

Data Preservation Creating trustworthy archives. Digital Preservation does not happen by accident  To preserve digital information, we need to take careful,

DAITSS: Dark Archive in the Sunshine State Priscilla Caplan, Florida Center for Library Automation DCC Workshop on Long-term Curation within Digital Repositories.

Johnson Museum Online 15,800 works on paper 6,700 objects in Asian collection high resolution, medium resolution, and thumbnail Luna.

Support for NPP/NPOESS by The HDF Group Mike Folk The HDF Group HDF and HDF-EOS Workshop XII October 17, 2008 Oct HDF and HDF-EOS Workshop XII1.

Archival Information Packages for NASA HDF-EOS Data R. Duerr, Kent Yang, Azhar Sikander.

Towards Long-Term Archiving of NASA HDF-EOS and HDF Data Data Maps and the Use of Mark-Up Language Ruth Duerr, Mike Folk, Muqun Yang, Chris Lynnes, Peter.

Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Ruth Duerr, NSIDC Christopher Lynnes, GES DISC The HDF Group Oct HDF and.

Creating Archive Information Packages for Data Sets: Early Experiments with Digital Library Standards Ruth Duerr, NSIDC MiQun Yang, THG Azhar Sikander,

1 24 September BREAKOUT :30 1)Review of Metadata Standards Directory (DCC version and GitHub) 2)Introduction of Metadata Standards Catalog.

Long-Term Knowledge Retention Joshua Lubell Manufacturing Systems Integration Division, NIST FIRM’s Forum at FOSE March 20, 2007.

March 17, 2006CIP Status Meeting March 17, 2006 Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration Project Report at CIP AG Meeting.

National Geospatial Digital Archive Greg Janée University of California at Santa Barbara.

Recent Developments in CLARIN-NL Jan Odijk P11 LREC, Istanbul, May 23,

Research Data Services from the ASU Libraries Mary Whelan GIS Data Manager.

- 1 - HDF5, HDF-EOS and Geospatial Data Archives HDF and HDF-EOS Workshop VII September 24, 2003.

Discovery and Access of Historic Literature from the IPYs (DAHLI) Rescuing Records and Publications of Early IPY Ventures Ruth Duerr and Allaina Howard.

Space Research Software Pedro Vicente Space Research Software, Inc. HDF-EOS Workshop VI Dec. 4-5, 2002 San Francisco, CA HDF.

19 May 2006 NCSA CIP Status Meeting 1 HDF5/SRB Integration Peter Cao & Mike Folk, NCSA Mike Wan & Reagan Moore, SDSC.

Metadata “Data about data” Describes various aspects of a digital file or group of files Identifies the parts of a digital object and documents their content,

July 20, Update on the HDF5 standardization effort Elena Pourmal, Mike Folk The HDF Group July 20, 2006 SPG meeting, Palisades, NY.

HDF and HDF-EOS Workshop VII September 24, 2003 HDF5, HDF-EOS and Geospatial Data Archives Don Keefer Illinois State Geological Survey Mike Folk Univ.

NetCDF and Scientific Data Durability Russ Rew, UCAR Unidata ESIP Federation Summer Meeting

The Importance of Standards in Digital Preservation Tina Norris Kayla Payne Jennifer

Data Management Plans: Elements of a Data Management Plan Ruth Duerr National Snow and Ice Data Center Version 1.0 Review Date.

Trials and Tribulations of a Small Archive Presented at the THIC Conference, NCAR, Boulder CO June 30, 2004 Presented at the THIC Meeting at the National.

HDF and HDF-EOS: Implications for Long-Term Archiving and Data Access.

Aligning Digital Preservation Policies with Community Standards Nancy McGovern Digital Preservation Officer.

MODIS Data at NSIDC MODIS Science Team Meeting - Nov. 2, 2006.

Institutional Repositories July 2007 DIGITAL CURATION creating, managing and preserving digital objects Dr D Peters DISA Digital Innovation South.

Infrastructure Breakout What capacities should we build now to manage data and migrate it over the future generations of technologies, standards, formats,

Digital Preservation: The State of the Art November 1999 Update Marc Fresko CONSOLIDATING THE EUROPEAN LIBRARY SPACE - 18 NOVEMBER 1999 © Copyright Applerace.

Using the DMPTool for data management plans Kathleen Fear February 27, 2014.

Writing a Data Management Plan with the DMPTool Kathleen Fear January 15, 2015.

Preservation Functionality in a Digital Archive Erik Oltmans Koninklijke Bibliotheek Raymond J. van Diessen IBM Business Consulting Services Hilde van.

A Semi-Automated Digital Preservation System based on Semantic Web Services Jane Hunter Sharmin Choudhury DSTC PTY LTD, Brisbane, Australia Slides by Ananta.

Transition from HDF4 to HDF5: Issues

Working Group 4 Facilities and Technologies

Presented April 7, 2005 at the 2005 AAG meeting, Denver, CO

Presentation transcript:

HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term

HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, Outline The Long Term Problem A Few Ideas Discussion

HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, The Digital Preservation Challenge “digital objects require constant and perpetual maintenance, and they depend on elaborate systems of hardware, software, data and information models, and standards that are upgraded or replaced every few years” NSF and Library of Congress, August 2003

HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, What is a Good Long-Term Archive Format? Per a recent paper by Mike Folk and Bruce Barkstrom  Ease of archival storage  Ease of archival access  Usability  Data scholarship enablement  Support for data integrity  Maintainability and durability

HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, What Makes a Good File Format? Per Eric Raymond, in “The Art of Unix Programming”, Addison-Wesley, 2004  Transparency  Interoperability  Extensibility  Storage economy He argues that the best general purpose file format is text He also argues that the only good justification for binary data is with very large data sets

HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, Disaster-Proofing Your Data If you can’t keep a data set as text, then at least keep the representational information in a human readable format (preferably right with the data itself)

HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, HDF and HDF-EOS as an Archive Format Neither are optimized for archival  Not a text-based format  Does not enforce inclusion of semantic metadata  Meant to be used with the HDF* libraries

HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, Towards the Future? Discussions with Don Sawyer, Lou Reich, Mike Folk and others at NCSA A Few Ideas  Specifying an HDF-archive format a’la PDF-A  Text wrapper/encoding for HDF  Tools to translate to/from an archive format Mechanisms?  CODATA  International workshop  Proposals

HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, Discussion Any suggestions for moving forward? Questions? Concerns?

HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented at the HDF & HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec.2, 2005 by R. Duerr HDF for the Ages: Keeping HDF Data Accessible and Usable for the Long Term Presented by R. Duerr, HDF &HDF-EOS Workshop IX, San Francisco, Nov. 30-Dec. 2, Contact Info For questions about data preservation, etc. contact me at For information about NSIDC data products, programs, etc. see or contact our user services at