GDWG Agenda Item Repository for GSICS Work - Clarification on Archiving Masaya Takahashi, JMA CMA, CNES, EUMETSAT, ISRO, IMD, JAXA, JMA, KMA, NASA, NIST, NOAA, ROSHYDROMET, USGS, WMO

Overview and Purpose Overview Purpose of this presentation Existing datasets for the ATBDs 1) SRF netCDFs for all the GRWG activities Now archived on the NOAA THREDDS and the GISCS wiki 2) Source dataset (lunar observation netCDFs) for GRWG VIS/NIR sub-group Now archived on the EUMETSAT internal Linux server Possible datasets for the ATBDs 3) Intermediate dataset (Deep Convective Cloud) for GRWG VIS/NIR sub-group To share those data online is very important not only for maintaining the data but also for accelerating/encouraging the GRWG activities Purpose of this presentation To introduce the GRWG requirements To discuss the proper way to archive those datasets

GRWG Requirements for archiving the dataset for the ATBDs – 1) SRF netCDFs Spectral Response Function (SRF) Instrument specific (fundamental) information -> necessary for GRWG activities File size: small in general (~MB), not so frequently updated Each GPRC has their official SRF(s) A variety of formats (.csv, .txt, .xls, .hdf, etc.) In 2013, GDWG (A. Jelenak) converted official SRFs to netCDFs for user convenience GRWG VIS/NIR sub-group is the first user (after minor modifications by M. Takahashi) More and more GRWG members will use the dataset for the ATBDs in the near future How to store the data at present? On the NOAA THREDDS server as “Source Data” GSICS THREDDS servers are NOT the data archiving system, even though that is not clarified On the GSICS wiki (https://gsics.nesdis.noaa.gov/wiki/Development/Srf4Giro) Commas/plus signs: automatically removed from filenames E.g., W_CN-CMA-NSMC,VIS+IR+SRF,FY2D+SVISSR_C_BABJ.nc -> W_CN-CMA-NSMCVISIRSRFFY2DSVISSR_C_BABJ.nc

GRWG Requirements for archiving the dataset for the ATBDs – 2) Lunar observation netCDFs Source dataset for the ATBD (lunar calibration, GRWG VIS/NIRS sub-group) To collaborate with the lunar calibration community, the lunar calibration workshop (LCW, Dec 2014@EUMETSAT) decided to share some datasets File size: depends on the instrument (e.g., ~5MB for MTSAT-2, ~20MB for Himawari-8) # of files: depends on the instrument (e.g., ~5/month for MTAST-2, ~500/month for Himawari-8) NO need to share all the data, so we can adjust data volume depending to the storage capacity Password protection would be required to limit the access within the community Action assigned at the LCW GLCWS_14.7: GSICS Data Working Group to identify a suitable mechanism for sharing the lunar calibration dataset (Main: M.Takahashi) How to store the data at present? On the EUMETSAT Linux server Only EUMETSAT people can access the data

GDWG Proposed Solution for issued on the data archiving Group discussion to proposed the following solution S1: GDWG to define GSICS THREDDS servers as the archiving system S2: GDWG to recommend to Use GSICS wiki S3: GDWG to find online storage services such as Dropbox

GDWG discussion on the Pros and Cons of the presented solutions S1: GDWG to define GSICS THREDDS servers as the archiving system pros: good to store the netCDF files cons: internal issues in EUMETSAT/NOAA/CMA? S2: GDWG to recommend to use the GSICS wiki pros: easy to put small / few files cons: difficult to store big volume data, problem on the automatic change of the netCDF file name S3: GDWG to find online storage services such as Dropbox pros: easy-to-use cons: THREDDS/OpenDAP is not supported?, the service might not free, security issue? I would like to recommend to invest resources to implementing S1.

GDWG discussion on the details on the proposed solution (to be updated in the discussion) The GDWG proposed … Resources Needed: Man Days Effort: Identify a system (service) and evaluate its suitable for GSICS dataset archiving – 1 week Specify how the system is to be used and the tasks to be implemented – 2 weeks Implementation – 1 week. HW & SW Procurements: unknown Input Data Acquisition Effort: not applicable Proposed Time Scale: To be available as soon as possible for collaboration. Latest implementation is for the next joint meeting Deliverables: Data archiving server (or service) Dataset on the server Developer’s documentation for using the server Priority of the Task: High – by the next GSCIS annual meeting

Agenda item summary; assign action identifier, make EP recommendations and propose a lead for the action (to be updated in the discussion) Action Identifier: GDWG_15.XX: XX to … Substantial effort if required by the GDWG, GDWG chair should inform this activity to GSICS Executive Panel, and ask for feedback regarding: Should GDWG resources be used to investigate how to archive the dataset for GSICS activities (not products) and provide solutions to GRWG. The GDWG estimates 4 weeks of resources is needed to support this action. Identify the Working Group Member Taking the Lead on this Action: XXXX The EP is invited to comment on this if the action is to be performed.

End of Presentation: Thank you for your attention WMO GSICS Portal http://gsics.wmo.int GSICS Coordination Centre - http://www.star.nesdis.noaa.gov/smcd/GCC/index.php GSICS Product Catalog - http://www.star.nesdis.noaa.gov/smcd/GCC/ProductCatalog.php EUMETSAT’s Data and Management Server http://gsics.eumetsat.int