GSICS EP Meeting 2015 – GSICS Archiving Strategy 1 GSICS Archiving Strategy Peter Miu / Boulder, USA CMA, CNES, EUMETSAT, ISRO, IMD, JAXA, JMA, KMA, NASA, NIST, NOAA, ROSHYDROMET, USGS, WMO GSICS Data Preservation Strategy

GSICS EP Meeting 2015 – GSICS Archiving Strategy Overview  Clarification: Archive vs Preservation  Identifying what GSICS Products to preserve.  Data Preservation Strategy.  Requirements and Summary. 2

GSICS EP Meeting 2015 – GSICS Archiving Strategy Data Archiving vs Data Preservation  The term data archiving not only implies storage but additional services such as:  data discovery – services for users to detect the data (making the data explorable from multiple sources).  data identification – services for users to clearly understand what the data is (e.g. meta-data, landing pages, etc.).  data search – services for users to find the data.  data migration – obsolescence mitigation services to ensure the components of the archive maintain a reasonably level of integrity e.g. media change. These services make data archiving expensive.  Data preservation is less of a formal endeavour to ensure that digital information of continuing value remains accessible and usable. It only imply reliable access to the data. For GSICS products, a data preservation strategy shall apply. 3

GSICS EP Meeting 2015 – GSICS Archiving Strategy Identifying what to Preserve  Do we need to preserve everything? 4

GSICS EP Meeting 2015 – GSICS Archiving Strategy GEOLEOIR NRTC Products Analysis  EUMETSAT GEOLEOIR Near Real Time Correction (NRTC) Product Analysis:  One product has one correction applicable to support real time product generation.  Products have a shelf life after which time they are no longer useful.  For an ATBD, EUMETSAT can regenerate all NRTC products if needed. Do we need to preserve NRTC GSICS products? Recommendation: No. 5

GSICS EP Meeting 2015 – GSICS Archiving Strategy GEOLEOIR RAC Products Analysis  EUMETSAT GEOLEOIR Re-Analysis Corrections (RAC) Product Analysis:  RAC is a combined product replaced daily with a new correction.  RAC products are expected to be available indefinitely i.e. no shelf life.  A RAC product will continue to grow until an ATBD update. A new MAJOR VERSION of the RAC combined product file is then created for the same time period as the version it replaces. The old version of the RAC product is moved but still accessible for reference.  All versions of a RAC product should be available for reference.  EUMETSAT can recreate any ATBD version of a RAC product if needed. Do we need to preserve RAC GSIC products? Recommendation: Yes. 6

GSICS EP Meeting 2015 – GSICS Archiving Strategy Data Preservation Strategy (1)  For the latest RAC products’ version, product regeneration is done through:  By design as a RAC product is generated daily with a new correction which replaces the existing RAC file. In addition to this, RAC products are expected to be replicated :  Across all collaboration servers.  Through System administration of the servers where cluster technology is used to ensure service always available and high availability disk arrays.  In addition to this, backups of disk are periodically taken to simplify data recovery. Worst case scenario is the RAC product can be regenerated by the product generation software. 7

GSICS EP Meeting 2015 – GSICS Archiving Strategy Data Preservation Strategy (2)  For the Old RAC products’ versions, product regeneration is done through the replication as described for the latest version:  Resulting from the mirroring across all collaboration servers.  Through System administration of the servers where cluster technology is used to ensure service always available and high availability disk arrays.  In addition to this, backups of disk are periodically taken to simplify data recovery. Worst case scenario is the RAC product of any version is regenerated by the product generation software. 8

GSICS EP Meeting 2015 – GSICS Archiving Strategy Requirements and Summary  To implement GSICS Product Preservation, the following Data Management Systems are required to be in place:  Mirroring on the collaboration servers for all GSICS products.  Collaboration Server Administrators need to ensure service and backup strategies are in place.  GSICS products generation software should be developed to be able to regenerate an version of a GSICS product. The last point is a mandatory requirement in the development of the GSICS product generation software. 9

GSICS EP Meeting 2015 – GSICS Archiving Strategy 10 End of Presentation: Thank you for your attention