Improving long-term preservation of EOS data by independently mapping HDF4 data objects The HDF Group.

Slides:



Advertisements
Similar presentations
The HDF Group Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Ruth Duerr, NSIDC Christopher Lynnes, GES DISC Mike.
Advertisements

The HDF Group Support for NPP/NPOESS by The HDF Group Mike Folk, Elena Pourmal The HDF Group HDF/HDF-EOS Workshop XIV September 30, 2010.
The HDF Group HDF/HDF-EOS Workshop XIV1 Easy Remote Access via OPeNDAP Kent Yang and Joe Lee The HDF Group The 14 th HDF/HDF-EOS Workshop.
The HDF Group HDF Group Support for NPP/JPSS Mike Folk, Elena Pourmal, Larry Knox, Albert Cheng The HDF Group Briefing to ESDIS October.
The HDF Group HDF Group Support for NPP/JPSS Mike Folk, Elena Pourmal, Larry Knox, Albert Cheng The HDF Group The 15 th HDF and HDF-EOS.
HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.)
The HDF Group HDF4 Mapping Status and Proposal Mike Folk, Ruth Aydt The HDF Group Oct. 17, 2012 Apr , 2012HDF/HDF-EOS Workshop XV.
US GPO AIP Independence Test CS 496A – Senior Design Team members: Antonio Castillo, Johnny Ng, Aram Weintraub, Tin-Shuk Wong Faculty advisor: Dr. Russ.
The HDF Group Improving long-term preservation of EOS data by independently mapping HDF4 data objects Mike Folk, Ruth Aydt, Joe Lee, Binh-Minh.
University of Illinois at Urbana-ChampaignHDF 1McGrath/Yang 2/27/02 Transitioning from HDF4 to HDF5 Robert E. McGrath Kent Yang.
Software Documentation Written By: Ian Sommerville Presentation By: Stephen Lopez-Couto.
Support EOS: Review and Discussions Kent Yang and Joe Lee The HDF Group October 16, 2012 Oct. 16, 2012Annual HDF Briefing to ESDIS1.
HDF 1 NCSA HDF XML Activities Robert E. McGrath Mike Folk National Center for Supercomputing Applications.
The HDF Group HDF/HDF-EOS Workshop XIV1 Easy Access of NASA HDF data via OPeNDAP Kent Yang and Joe Lee The HDF Group September 28,2010.
An Overview of Selected ISO Standards Applicable to Digital Archives Science Archives in the 21st Century 25 April 2007 Donald Sawyer - NASA/GSFC/NSSDC.
The HDF Group July 8, 2014HDF 2014 ESIP Summer Meeting HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann The.
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Mike Folks, The HDF Group Ruth Duerr, NSIDC 1.
The HDF Group HDF4 Mapping Project Update Apr , 2012HDF/HDF-EOS Workshop XV1 Ruth Aydt
EARTH SCIENCE MARKUP LANGUAGE “Define Once Use Anywhere” INFORMATION TECHNOLOGY AND SYSTEMS CENTER UNIVERSITY OF ALABAMA IN HUNTSVILLE.
-Nikhil Bhatia 28 th October What is RUP? Central Elements of RUP Project Lifecycle Phases Six Engineering Disciplines Three Supporting Disciplines.
HDF-EOS Workshop VII, An XML Approach to HDF-EOS5 Files Jingli Yang 1, Bob Bane 1, Muhammad Rabi 1, Zhangshi Yin 1, Richard Ullman 1, Robert McGrath.
EOSDIS User survey follow-up Mike Folk, Kent Yang, Elena Pourmal The HDF Group Oct. 17, 2012 Annual HDF Briefing to ESDIS1.
HDF Windows Support MuQun Yang, Xuan Bai, Elena Pourmal, Barbara Jones, Pedro Vincent, Robert E. McGrath National Center for Supercomputing Applications.
Important ESDIS 2009 tasks review Kent Yang, Mike Folk The HDF Group April 1st, /1/20151Annual briefing to ESDIS.
DM_PPT_NP_v01 SESIP_0715_AJ HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann Gerd Heber, John Readey, Joel Plutchak The HDF Group HDF.
 To explain the importance of software configuration management (CM)  To describe key CM activities namely CM planning, change management, version management.
Usability Issues Documentation J. Apostolakis for Geant4 16 January 2009.
Configuration Management (CM)
Meta Tagging / Metadata Lindsay Berard Assisted by: Li Li.
Why do I want to know about HDF and HDF- EOS? Hierarchical Data Format for the Earth Observing System (HDF-EOS) is NASA's primary format for standard data.
HDF Converting between HDF4 and HDF5 MuQun Yang, Robert E. McGrath, Mike Folk National Center for Supercomputing Applications University of Illinois,
Tools for Interoperability between HDF and NetCDF Mike Folk and MuQun Yang The HDF Group The HDF Group provides the following tools for the NASA HDF and.
1/14/200925th IIPS Conference 1 Challenges to Archive and Access NASA HDF-EOS Data in the long Term MuQun Yang (The HDF Group) Choonghwan Lee (The HDF.
Support for NPP/NPOESS by The HDF Group Mike Folk The HDF Group HDF and HDF-EOS Workshop XII October 17, 2008 Oct HDF and HDF-EOS Workshop XII1.
Archival Information Packages for NASA HDF-EOS Data R. Duerr, Kent Yang, Azhar Sikander.
11/7/2007HDF and HDF-EOS Workshop XI, Landover, MD1 HDF5 Software Process MuQun Yang, Quincey Koziol, Elena Pourmal The HDF Group.
The HDF Group November 3-5, 2009 HDF-OPeNDAP Project Update HDF/HDF-EOS Workshop XIII1 Joe Lee and Kent Yang The HDF Group James Gallagher.
Towards Long-Term Archiving of NASA HDF-EOS and HDF Data Data Maps and the Use of Mark-Up Language Ruth Duerr, Mike Folk, Muqun Yang, Chris Lynnes, Peter.
Ensuring Long Term Access to Remotely Sensed HDF4 Data with Layout Maps Ruth Duerr, NSIDC Christopher Lynnes, GES DISC The HDF Group Oct HDF and.
Creating Archive Information Packages for Data Sets: Early Experiments with Digital Library Standards Ruth Duerr, NSIDC MiQun Yang, THG Azhar Sikander,
HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII10/17/2008.
The FCLA Digital Archive Joint Meeting of CSUL Committees, 2005.
HDF OPeNDAP Project Update MuQun Yang and Hyo-Kyung Lee The HDF Group March 31, Annual briefing to ESDIS10/31/2015.
The HDF Group HDF/HDF-EOS Workshop XV1 Tools to Improve the Usability of NASA HDF Data Kent Yang and Joe Lee The HDF Group April 17, 2012.
- 1 - HDF5, HDF-EOS and Geospatial Data Archives HDF and HDF-EOS Workshop VII September 24, 2003.
The HDF Group Support for NPP/NPOESS by The HDF Group Mike Folk, Elena Pourmal, Peter Cao The HDF Group November 5, 2009 November 3-5,
OAIS Rathachai Chawuthai Information Management CSIM / AIT Issued document 1.0.
HDF Windows Support MuQun Yang, Xuan Bai, Elena Pourmal, Barbara Jones, Pedro Vincent, Robert E. McGrath National Center for Supercomputing Applications.
HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1 HDF and HDF-EOS Workshop XII10/17/2008.
Using a Friendly OPeNDAP Client Library to Access HDF5 Data MuQun Yang and Hyo-Kyung Lee (The HDF Group) 1 25th IIPS Conference01/14/2009.
HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII, Aurora,
19 May 2006 NCSA CIP Status Meeting 1 HDF5/SRB Integration Peter Cao & Mike Folk, NCSA Mike Wan & Reagan Moore, SDSC.
July 20, Update on the HDF5 standardization effort Elena Pourmal, Mike Folk The HDF Group July 20, 2006 SPG meeting, Palisades, NY.
HDF and HDF-EOS Workshop VII September 24, 2003 HDF5, HDF-EOS and Geospatial Data Archives Don Keefer Illinois State Geological Survey Mike Folk Univ.
10/16/2012Annual HDF briefing1 HDF OPeNDAP support Kent Yang, Joe Lee, Mike Folk The HDF Group Oct. 16, 2012.
HDF-EOS Workshop IV September 19-21, 2000 Richard E. Ullman ESDIS Information Architect NASA/ GSFC, Code 423.
11/8/2007HDF and HDF-EOS Workshop XI, Landover, MD1 Software to access HDF5 Datasets via OPeNDAP MuQun Yang, Hyo-Kyung Lee The HDF Group.
Improving long-term preservation EOS data by independently mapping HDF4 data objects Mike Folk, Ruth Aydt, Peter Cao, Kent Yang Ruth Duerr Christopher.
The HDF Group November 3-5, 2009HDF/HDF-EOS Workshop XIII1 The New HDF-EOS Web Site - How it can help you Kent Yang, Joe Lee The HDF Group.
Support for NPP/NPOESS by The HDF Group Mike Folk, Elena Pourmal The HDF Group Annual HDF Briefing to ESDIS March 31, 2009 March Annual HDF Briefing.
HDF5 OPeNDAP Project Update and Demo MuQun Yang and Hyo-Kyung Lee (The HDF Group) James Gallagher (OPeNDAP, Inc.) 1HDF and HDF-EOS Workshop XII, Aurora,
Joint Meeting of CSUL Committees,
HDF and HDF-EOS Workshop XII
Mike Folk, Peter Cao, Kent Yang Ruth Duerr Christopher Lynnes
Kent Yang, Mike Folk The HDF Group March 31, 2009
Software Documentation
Access HDF5 Datasets via OPeNDAP’s Data Access Protocol (DAP)
HDF Support for NASA Data Producers
PREMIS Tools and Services
HDF5 Tools Updates and Discussions
Presentation transcript:

Improving long-term preservation of EOS data by independently mapping HDF4 data objects The HDF Group

Mapping project team members The HDF Group Ruth Aydt Mike Folk Joe Lee Elena Pourmal Binh-Minh Ribler Muqun {Kent} Yang NASA Ruth Duerr & Luis Lopez(NSIDC) Chris Lynnes (GES DISC) April Annual HDF Briefing to NASA2 Raytheon Evelyn Nakamura many others

Recap Problem The complex byte layout of HDF files makes long- term readability of HDF data dependent on long- term availability of HDF software. Solution Create a map of the layout of data objects in an HDF file, allowing a simple reader to be written to access the data. Implement tools to create layout maps for EOS data products. Deploy tools at select EOS data centers. April Annual HDF Briefing to NASA3

April Annual HDF Briefing to NASA4

HDF4 mapping workflow HDF4 File HDF4 Map File (XML document) h4mapwriter linked with HDF4 library h4mapwriter linked with HDF4 library Reader program Object Data Groups, Data Objects, Structural and Application Metadata; Locations of Object Data April Annual HDF Briefing to NASA5

PHASE 1 BUILD A PROTOTYPE (COMPLETED IN 2009) April Annual HDF Briefing to NASA6

PHASE 2 PRODUCTIZE HDF4 MAPPING SCHEMA AND TOOLS FOR DEPLOYMENT April Annual HDF Briefing to NASA7

Phase 2 tasks April Annual HDF Briefing to NASA8 A.Investigate integration of mapping schema with existing standards B.Determine HDF-EOS 2 requirements C.Redesign and expand the XML schema D.Implement production quality map writer E.Develop demo map reader F.Deploy tools at select NASA data centers

April Annual HDF Briefing to NASA9

TASK A INVESTIGATE INTEGRATION OF MAPPING SCHEMA WITH EXISTING STANDARDS April Annual HDF Briefing to NASA10

Investigate existing standards Investigated: METS, PREMIS, ESML, NcML, and CSML Concluded: Existing standards have different purposes than mapping schema None meet all needs of mapping project Develop new schema tailored to project goals Harmonize with PREMIS Leverage terminology and approaches from all Status: Need to write report Need to include some PREMIS-like data such as HDF4 file size and possibly MD checksum April Annual HDF Briefing to NASA11

TASK B DETERMINE HDF- EOS2 REQUIREMENTS April Annual HDF Briefing to NASA12

Background An HDF-EOS2 file is an HDF4 file, so one can create an HDF4 mapping file to archive the HDF-EOS2 file. However, for some HDF-EOS2 files, it may be extremely difficult to retrieve correct geo- location information from the mapping files. For those files, special HDF-EOS2 mapping files may be needed. April Annual HDF Briefing to NASA13

Categorize HDF-EOS2 data products Created a data pool from NASA data centers GES DISC, NSIDC, LAADS, LP DAAC LaRC, PO.DAAC, GHRC, OBPG Analyzed data and reported options for adding HDF-EOS2 contents to the mapping file Conclusion: No special mapping for HDF- EOS2 needs to be done However, the study uncovered some important shortcomings in certain HDF-EOS products April Annual HDF Briefing to NASA14

Status and Plans Status: Complete Detailed descriptions of sample data: Documents and reports at wiki: Plans We plan to recommend a future task in which these issues are made known to the project April Annual HDF Briefing to NASA15

TASK C REDESIGN SCHEMA April Annual HDF Briefing to NASA16

Design priorities and assumptions Mapping files Provide complete access to user-supplied content in NASA’s EOS binary HDF4 files Have enough information to stand on their own Be as simple as possible Mapping schema Describe the Mapping files Used for validation and documentation May not be available to target user April Annual HDF Briefing to NASA17

Status and Plans Status All HDF4 objects found in EOS products are now handled by the Mapping schema. Plans Complete schema elements for HDF4 file description information File size, MD checksum (?), HDF4 library version stamp (?) Finalize schema documentation Address any additional HDF4 objects found during remainder of project, either by updating schema and map writer, or with follow-on proposal if substantial amount of effort required. April Annual HDF Briefing to NASA18

TASK D IMPLEMENT MAP WRITER April Annual HDF Briefing to NASA19

Map Writer Requirements Retrieve information needed from HDF4 file Write out corresponding XML file Quality requirements Completeness Don’t miss any objects in file Report on objects or features not handled by the writer Accuracy – don’t give wrong information Readability – p rovide adequate instructions in the file April Annual HDF Briefing to NASA20

Activities 1.Implement functions to facilitate map creation Develop writer requirements based on new XML schema and additional deployment needs Implement new functions as needed Include functions in library as appropriate 2.Implement writer: h4mapwriter Interpret map requirements according to schema Implement writer Package for deployment Support deployment April Annual HDF Briefing to NASA21

Status and Plans Status 1.Implement functions to facilitate map creation All functions implemented 2.Implement writer Handles all objects Available as alpha-2 release Being tested by GES DISC, NSIDC, Raytheon Plans 1.Functions to facilitate map creation Include in future HDF4 releases 2.Writer Finish HDF4 file description elements Complete testing and documentation Support deployment, fix bugs and add features as needed April Annual HDF Briefing to NASA22

TASK E IMPLEMENT DEMO READER April Annual HDF Briefing to NASA23

Demo Reader Requirements Multiplatform command line tool Easy to use clear arguments and output Must validate that objects in the mapping file are actually in the HDF4 file Developed in a well-supported high level language (python) Well documented Available as open source April Annual HDF Briefing to NASA24

Demo reader activities 1.Develop requirements, based on new schema and identification of additional deployment needs. 2.Design reader, based on requirements, and from a review of the prototype design. 3.Implement and document reader. 4.Test reader on EOS file “zoo” 5.Deposit reader, documentation, and tests in open source repository, probably SourceForge. April Annual HDF Briefing to NASA25

Demo Reader Status Status Support provided so far for Vdata, SDS, Group, and Attribute Current source code available at Documentation at Plans Add raster image (RIS) and palette support April Annual HDF Briefing to NASA26

TASK G DEPLOY April Annual HDF Briefing to NASA27

Task G: Deploy Begin in April 2011, complete in June The HDF Group Provide h4mapwriter map generation tool Maintain tool during deployment and validation Assist GES DISC, NSIDC, and Raytheon with deployment and validation Raytheon Validate HDF4 map software in anticipation of future deployment GES DISC and NSIDC: see next slide April Annual HDF Briefing to NASA28

What about GES DISC and NSIDC? Activities (formerly): GES DISC Incorporate into the existing archive ingest system Manage the retrofit into existing metadata files NSIDC Support implementation in NSIDC’s ECS system Other ESDCs Encouraged to join in But deployment to other centers expected subsequent to the project. Ruth Duerr’s observation: The task for NSIDC is to assist in the ECS implementation at NSIDC, which won't take place until 2012 Task G only includes the work up to the handoff to ECS Thus, what NSIDC does needs to extend after the period of performance of this award is over How do we resolve that issue? April Annual HDF Briefing to NASA29

BEYOND JULY 15 April Annual HDF Briefing to NASA30

Future work NSIDC assist in the ECS deployment at NSIDC GES DISC: ? The HDF Group Monitor deployment activities by Raytheon and others to identify Unsupported objects and tags occurring in products Software defects Feature requests As needed, fix defects, add features, and add support for new objects and tags Address performance issues Add h4mapwriter tool and supporting API to regular HDF4 testing regime Perform other services in support of the software as needed All Perform post mortem and identify lessons learned Write paper summarizing the project Investigate HDF5 mapping April Annual HDF Briefing to NASA31

The End

Acknowledgements This work was supported by cooperative agreement number NNX08AO77A from the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author[s] and do not necessarily reflect the views of the National Aeronautics and Space Administration. April Annual HDF Briefing to NASA33