1. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India 1 GRWG Requirements Summary on Data, Products, and Tools - for GDWG Understanding Masaya Takahashi, JMA CMA, CNES, EUMETSAT, ISRO, IMD, JAXA, JMA, KMA, NASA, NIST, NOAA, ROSHYDROMET, USGS, WMO

2. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Overview and Purpose  Overview of GRWG requirements 1.Change of existing netCDF variables for GEO-LEO-IR 2.Handling of multiple references/calibration-methods in one GSICS Correction 3.Sharing/Archiving source codes, datasets for the ATBDs 4.Developing tools for GRWG activities 5.Clarifying the GSICS data versioning  Some of the requirements are related  Updating GSCIS THREDDS directory structure  Updating GSICS bias plotting tool  Purpose of this presentation  To introduce the GRWG requirements and help GDWG members to understand the issues to be discussed in day-3/4 breakout sessions 2

3. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Overview and Purpose  Overview of GRWG requirements 1.Change of existing netCDF variables for GEO-LEO-IR 2.Handling of multiple references/calibration-methods in one GSICS Correction 3.Sharing/Archiving source codes, datasets for the ATBDs 4.Developing tools for GRWG activities 5.Clarifying the GSICS data versioning 3

4. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Requirements 1. Change of existing netCDF variables for GEO-LEO-IR  Discussed in July 2014 web meeting on ”Inter-Calibration Products to Primary GSICS References””Inter-Calibration Products to Primary GSICS References  The following variables to be modified: a) Calibration coefficients (slope, slope_se, offset, offset_se, covariance) Many variables are required in the processing (e.g. Implementation of GSICS correction, calculation of uncertainty) Based on linear regression = no expandability for non-linear regression (might be needed in future) b) Validity period Impossible to specify “valid hour range” in the current variable “valid hour range” is required for some products, in order to Boost the confidence of the product, Avoid the incorrect use of the product  Global attributes New global attributes such as “doc_url”, “doc_doi”, and “instrument_wmo_code”  Once this change is accepted, this will also be applied to forthcoming GSICS products 4

5. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India GDWG discussion on the calibration coefficients – comparison of the current/proposed variables 5 float offset(date, chan) ; offset:valid_max = 200.f ; offset:valid_min = -200.f ; offset:_FillValue = "-999999" ; offset:long_name = "Regression Offset" ; offset:units = "mW m-2 sr-1(cm-1)-1" ; float offset_se(date, chan) ; offset_se:valid_max = 200.f ; offset_se:valid_min = -200.f ; offset_se:_FillValue = "-999999" ; offset_se:long_name = "Standard Error of Regression Offset" ; offset_se:units = "mW m-2 sr-1(cm-1)-1" ; float slope(date, chan) ; slope:valid_max = 2.f ; slope:valid_min = -2.f ; slope:_FillValue = "-999999" ; slope:long_name = "Regression Slope" ; slope:units = "1" ; float slope_se(date, chan) ; slope_se:valid_max = 2.f ; slope_se:valid_min = -2.f ; slope_se:_FillValue = "-999999" ; slope_se:long_name = "Standard Error of Regression Slope" ; slope_se:units = "1" ; float covariance(date, chan) ; covariance:valid_max = 200.f ; covariance:valid_min = -200.f ; covariance:_FillValue = "-999999" ; covariance:long_name = "Regression Coefficients Covariance" ; covariance:units = "mW m-2 sr-1(cm-1)-1" ; Current GRWG proposal float coeff(ncoeff, date, chan) ; offset:valid_max = 200.f ; offset:valid_min = -200.f ; offset:_FillValue = "-999999" ; offset:long_name = "Regression coefficients" ; offset:units = "mW m-2 sr-1(cm-1)-1" ; float covar(ncoeff, ncoeff, date, chan) ; covariance:valid_max = 200.f ; covariance:valid_min = -200.f ; covariance:_FillValue = "-999999" ; covariance:long_name = "Regression coefficients covariance matrix" ; covariance:units = "mW m-2 sr-1(cm-1)-1" ; GRWG proposal is a general form of the regression

6. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India GDWG discussion on the calibration coefficients – comparison of the current/proposed variables 6 Y = slope * X + offset Y = a * X 2 + b * X + c Covariance matrix (need for the uncertainty calculation) Existing variables cannot support quadratic (polynomial) regression equation, but the proposed one can! float coeff(ncoeff, date, chan) float covar(ncoeff, ncoeff, date, chan) ncoeff = 2 (i.e., [offset, slope] ) covar(2, 2, date, chan) Linear regression (existing GEO-LEO-IR products adopt this way) Quadratic regression (future GSICS products might use this regression) Is the proposal acceptable? No problem on the plotting tool / THREDDS?

7. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India GDWG discussion on the validity period – problem on the current variable 7 Current variable: validity_period double validity_period(date, validity) ; validity_period:long_name = "correction validity period" ; validity_period:standard_name = "time" ; validity_period:calendar = "gregorian" ; validity_period:units = "seconds since 1970-01-01T00:00:00Z" ; validity_period[783][2] [0], 1.2926304E9, 1.295136E9 [1], 1.2927168E9, 1.2952224E9 [2], 1.2928032E9, 1.2953088E9... “2”, this means start/end time for each result Example  Problem: Impossible to specify “valid hour range” in that variable Necessary for instruments which have diurnal calibration biases (e.g., NOAA/JMA GEO-LEO-IR products) Should be specified for all instruments (In definition, GSICS Correction is derived from only night-time observation)  Possible solutions Add new attribute regarding validity period to each variable Add new variable to specify “validity hours” Expand existing validity_period to contain hour range

8. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Overview and Purpose  Overview of GRWG requirements 1.Change of existing netCDF variables for GEO-LEO-IR 2.Handling of multiple references/calibration-methods in one GSICS Correction 3.Sharing/Archiving source codes, datasets for the ATBDs 4.Developing tools for GRWG activities 5.Clarifying the GSICS data versioning 8

9. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Requirements for updating the current file naming / netCDF conventions for the new GSICS products  GEO-LEO-VISNIR (DCC will be the first method for this algorithm type)  This algorithm type contains “multiple methods’ results” in one file  M. Takahashi and S. Wagner showed the proposal in 2014  Addition of the THREDDS directories is necessary  GRWG requirements: GDWG to investigate the method to implement multiple calibration methods’ results in one netCDF file  GSICS Prime Correction for GEO-LEO-IR  Change of file naming: slightly different file naming from existing GEO-LEO-IR  This product contains “multiple references’ results” in one file  M. Takahashi and T. Hewison showed the proposal in 2014  Change of THREDDS directory structure is necessary  GRWG requirements: GDWG to discuss whether the file naming is acceptable or not  One GSCIS Correction file for one target instrument  Long term plan, to be discussed in this meeting (?)  The same issue as GEO-LEO-VISNIR product 9

10. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India How to implement multiple methods/references results in one file 10  S1: Use netCDF-4 “grouping” functionality (enhanced format) Combined result (DCC+Moon+…) Global attributes, Dimensions, Variables Method1 (Deep Convective Cloud) Global attributes, Dimensions, Variables Method2 (Moon) Global attributes, Dimensions, Variables SEVIRI vs. Aqua/MODIS VISNIR product... Method3 (Desert) Global attributes, Dimensions, Variables schematic image of usage of “grouping”  S2: Use different global attributes / dimensions / variables for each method float coeff(nmethod, ncoeff, date, chan) ; offset:_FillValue = "-999999" ; offset:long_name = “Inter-calibration coefficients" ; offset:units = “W m-2 sr-1 um-1" ; double date(nmethod, date) ; standard_name: time; calendar: gregorian; units: seconds since 1970-01-01T00:00:00Z; Common variables can be merged To put method-specific variables, additional information (e.g., prefix) is necessary for variable name float dcc_mon_mode_dc(date, chan) ; float dcc_ref_mode_dc(date, chan) ; float moon_mon_irr_obs(date, chan) ; float moon_mon_irr_rolo(date, chan) ; … Prefix to distinguish the inter-calibration method

11. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Is multiple file naming for the same algorithm type acceptable? 11 1) Existing GEO-LEO-IR products One reference in one file (e.g., MSG3/SEVIRI-MetOpA/IASI, MSG3/SEVIRI-MetOpB/IASI) Need to specify the “Reference Instrument Name” in a filename e.g., W_XX-EUMETSAT-Darmstadt,SATCAL+RAC+GEOLEOIR,MSG2+SEVIRI-MetOpA+IASI_C_EUMG_20110101000000_preop_01.nc 2) GSICS Prime Correction (at that time, this is for GEO-LEO-IR) Multiple reference (blended results) in one file No need to specify the “Reference Instrument Name” in a filename e.g., W_XX-EUMETSAT-Darmstadt,SATCAL+RAC+GEOLEOIR,MSG2+SEVIRI_C_EUMG_20110101000000_preop_01.nc Each GPRC will continue producing 1) even if 2) is produced Any possible issues on the proposal from the GDWG point of view? If the proposal is NOT accepted, dummy reference instrument name would be added, but redundant! e.g., W_XX-EUMETSAT-Darmstadt,SATCAL+RAC+GEOLEOIR,MSG2+SEVIRI- PrimaryRef_C_EUMG_20110101000000_preop_01.nc

12. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India GRWG proposal on updating the THREDDS directory structure for forthcoming GSICS products 12  GRWG proposal is available on: https://gsics.nesdis.noaa.gov/wiki/Development/DataServerConf MSG2 SEVIRI calibrated against Aqua MODIS W_XX-EUMETSAT-Darmstadt,SATCAL+NRTC+GEOLEOVISNIR,MSG2+SEVIRI_C_EUMG_20140203000000_preop_01.nc New directory for the Prime Correction forthcoming GEO-LEO-IR Prime Correction forthcoming GEO-LEO-VISNIR Prime Correction New directory for individual references (= existing GEO-LEO-IR GSICS Corrections) forthcoming GEO-LEO-VISNIR GSICS Correction The same structures as those for NRTC

13. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Overview and Purpose  Overview of GRWG requirements 1.Change of existing netCDF variables for GEO-LEO-IR 2.Handling of multiple references/calibration-methods in one GSICS Correction 3.Sharing/Archiving source codes, datasets for the ATBDs 4.Developing tools for GRWG activities 5.Clarifying the GSICS data versioning 13

14. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Existing tools/datasets for the ATBDs  GRWG: GIRO (GSICS Implementation of the ROLO model, written by C)  Reference model for lunar calibration GIRO input/output handling utilities are also provided by EUMETSAT (IDL)  Lunar observation netCDFs (source dataset for the GIRO) Now archived on the EUMETSAT internal Linux server  GIRO source codes and lunar observation dataset to be shared within the lunar calibration community (GRWG: discuss the policy, GDWG: discuss the way to share them) GLCWS_14.7: GSICS Data Working Group to identify a suitable mechanism for sharing the lunar calibration dataset (Main: M.Takahashi)  GDWG  GSICS plotting tool by EUMETSAT (Java and GWT)  SRF netCDF converter by NOAA (python) For all GRWG activities, SRF netCDFs are archived on the NOAA THREDDS and GSICS wiki  netCDF metadata checker by NOAA (Java and shellscript)  To share those data online is very important not only for maintaining the data but also for accelerating/encouraging the GRWG activities 14

15. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India GDWG discussion on the repository for codes/tools/datasets for the ATBDs 15  S1: GDWG to define GSICS THREDDS servers as the archiving system  S2: GDWG to recommend to use the GSICS wiki  S3: GDWG to find online storage services such as Dropbox Repository for tools/datasets: Current GSICS THREDDS servers are NOT considered as the data archiving system  S1: GPRC to provide a code to members for development.  S2: GCC to be the central repository for GSICS developed Code.  S3: Investigate a “free” tool for code configuration Repository for source codes:

16. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Overview and Purpose  Overview of GRWG requirements 1.Change of existing netCDF variables for GEO-LEO-IR 2.Handling of multiple references/calibration-methods in one GSICS Correction 3.Sharing/Archiving source codes, datasets for the ATBDs 4.Developing tools for GRWG activities 5.Clarifying the GSICS data versioning 16

17. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Requirements for the GIRO output checker  GIRO (GSICS Implementation of the ROLO model)  Input/Output files: follow the GSICS filenaming/netCDF convention  Outcome from the original GIRO is only considered as “legal” However, someone needs to modify the source codes to implement the GIRO to his/her agency’s computer system Needs for the checker which validates the consistency of the GIRO output with the original one 17 GLCWS_14.10 The GSICS Data Working Group is asked to develop a procedure for verifying users’ implementation  GRWG requirements: to compare the GIRO outputs by users with reference data  Reference GIRO outputs: to be executed under certain conditions (32/64 bit Linux, Mac OS X, …)  Checking items: values of all the variables (and dimensions, attributes), global attributes should also be checked (from the GSICS point of view)  Output of the checker: printout / output checking results (text messages are enough?)

18. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Overview and Purpose  Overview of GRWG requirements 1.Change of existing netCDF variables for GEO-LEO-IR 2.Handling of multiple references/calibration-methods in one GSICS Correction 3.Sharing/Archiving source codes, datasets for the ATBDs 4.Developing tools for GRWG activities 5.Clarifying the GSICS data versioning 18

19. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India Requirements for updating the GSCIS data versioning  Existing version controlling of GSICS deliverables https://gsics.nesdis.noaa.gov/wiki/Development/NetcdfConvention#Data_Versio ning https://gsics.nesdis.noaa.gov/wiki/Development/NetcdfConvention#Data_Versio ning  Data version identifier: "vX.Y.Z” (X, Y, and Z are integer numbers)  X: major version number, Y: minor version number, Z: revision number  Actually used for GEO-LEO-IR products Should be applied to GSICS tools/datasets (e.g., GIRO)?  However, current definitions are not enough  Ambiguities in the case of bug-fixes  No information on the use of major version “0”  Unclear how to update version numbers  GRWG Requirements  Bug fixes (w/ and w/o big impact on the outcome) should be assigned to either one of the three versions.  Procedure to update versions should be clarified. Our definition is the same as the right figure (from Wikipedia)?  Usage (definition) of major version 0 should be clarified. 19 http://en.wikipedia.org/wiki/Software_versioning

20. 2015 GSCIS annual meeting, 16-20 March, 2015, New Delhi, India 20  Any questions / requirements / suggestions / proposals?  If you are interested in the GDWG topic(s), please feel free to join the GDWG breakout session!