GIRO Input/Output data convention Masaya Takahashi (JMA), Bartolomeo Viticchie (EUMETSAT) Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

GIRO Input/Output files We decided to use netCDF for both input and output files Inputs: lunar observation and instrument SRF files Output: lunar calibration file Why? From the project framework point of view GIRO is a GSICS Implementation of the ROLO model GSICS adopts (classic) netCDF format for the inter-calibration products Lunar calibration: one of the future inter-calibration method for visible and near infrared channels GIRO will be used for the GSICS inter-calibration purpose From coding point of view Common format data: easy-to-handle (read, write) Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

GSICS file-naming/netCDF convention GSICS netCDF convention https://gsics.nesdis.noaa.gov/wiki/Development/NetcdfConvention Follows the netCDF CF (Climate and Forecast) Convention widely used in the netCDF community (http://cfconventions.org/) GSICS file naming convention https://gsics.nesdis.noaa.gov/wiki/Development/FilenameConvention  Follows the rules given in the General File Naming Conventions section of the W.M.O. Manual on The Global Telecommunication System Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

Input-1: lunar observation netCDF File name (e.g., MTSAT-2/Imager) W_JP-JMA-MSC,VISNIR+SUBSET+MOON,MTSAT2+Imager_C_RJTD_20130701062451_01.nc satellite and instrument name lunar observation time as yyyyMMddhhmmss Single file for each lunar occurrence, including all-channel information Variable units: must be the same as those in a template Recommended to use classic netCDF4 format Could be used as “GSICS source dataset” in the future Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

global attributes: lunar observation netCDF No impact on the GIRO results, but very important for the dataset to be shared within GSICS (+ α) community in future NAME CONTENT Conventions CF-1.6 Metadata_Conv entions Unidata Dataset Discovery v1.0 standard_name_ vocabulary CF Standard Name Table (Version 19, 22 March 2012) project e.g. Global Space-based Inter-Calibration System http://gsics.wmo.int title e.g. MSG-2+SEVIRI lunar imagtette file summary e.g. Lunar observation file keywords “free content” references TBD institution e.g. EUMETSAT licence “terms of use” creator_name e.g., EUMETSAT - European Organisation for the Exploitation of Meteorological Satellites NAME CONTENT creator_email e.g. ops@eumetsat.int creator_url e.g. http://www.eumetsat.int instrument e.g. MSG2 SEVIRI instrument_wmo_code e.g. (56, 207) data_source “free content” e.g., Level 1.0 date_created YYYY-MM-DDTHH:MM:SSZ date_modified history “free content” id File name wmo_data_category 101 for Imager data(satellite) wmo_international_dat a_subcategory 0 for Multi-purpose VIS/IR imagery processing_level e.g. “v1.0.0” doc_url URL of document doc_doi DOI of document (if exists) Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

dimensions/variables: lunar observation netCDF NAME VALUE DESCRIPTION chan e.g. 4 for SEVIRI Number of channels chan_strlen e.g. 6 for SEVIRI Maximum length of variable: chan date 1 Number of date sat_ref_strlen 6 Maximum length of variable: sat_pos_ref (i.e. “ITRF93”) sat_xyz 3 X, Y, and Z axis of satellite position col, row e.g. 499 for SEVIRI Number of columns and rows of the imagette Compulsory Imagette (optional) Additional (optional) variables NAME [dimension] LONG NAME UNITS TYPE date[date] time of lunar observation seconds since 1970- 01-01T00:00:00Z  double precision floating point channel_name[chan,chan_strlen] channel identifier character ,e.g. “VIS006” sat_pos[sat_xyz] satellite position km sat_pos_ref[sat_ref_strlen] satellite position x y z character, either “J2000” or “ITRF93” irr_obs[chan] observed moon irradiance W m-2 µm-1 rad_obs_imgt[row,col,chan] observed moon radiance imagette W sr-1 m-2 µm-1 dc_obs_imgt[row,col,chan] observed moon digital counts imagette 1 32-bit integer array moon_pix_thld[chan] digital counts threshold for moon mask 32-bit integer pix_solid_ang[chan] pixel solid angle sr ovrsamp_fa[chan] oversampling factor dc_obs[chan] integrated digital counts of moon observation dc_obs_offset[chan] averaged digital counts offset of deep space moon_pix_num[chan] number of moon pixels Possible to handle floating point counts using attributes: scale_factor and add_offset Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

Input-2: instrument SRF netCDF File name (e.g., Meteosat-10/SEVIRI): W_XX-EUMETSAT-Darmstadt,VIS+IR+SRF,MSG3+SEVIRI_C_EUMG.nc Prepared by GSICS Data management Working Group (GDWG, mainly by M.Takahashi) using the SRFs provided by each participant Some participants generate the file by themselves due to a practical reason Some files provided by GSICS members are available on the GSICS Wiki https://gsics.nesdis.noaa.gov/wiki/Development/Srf4Giro Potential dataset as the “GSICS Standard SRF” (to be discussed in the near future) Originally, generated by GDWG in 2013 for the purpose of GSCIS product developments http://gsics.nesdis.noaa.gov/thredds/instrument-srf.html “This file was produced in support of GSICS activities and thus is not meant for public use although the data in the file is in the public domain.” (from docs on the NOAA GSICS Data server) Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

Output: lunar calibration netCDF File name (e.g., Meteosat-7/MVIRI): W_XX-EUMETSAT-Darmstadt,SATCAL+ROLOVISNIR,MET7+MVIRI_C_EUMG_201307010634_01.nc Single file contains all-time, all-channel information follows the GSICS approach to generate inter-calibration products (e.g., GEO-LEO-IR Re-Analysis Correction) Contents Contents of a lunar observation file (except imagette) + ROLO reference irradiance, the Moon phase angle, positions of the Moon, the Sun, etc. Starting point for developing a GSICS inter-calibration product Concept: applicable to the future GSICS GEO-LEO VIS/NIR products Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

global attributes: lunar calibration netCDF Details will be discussed/updated in the future NAME CONTENT Conventions CF-1.6 Metadata_Conve ntions Unidata Dataset Discovery v1.0 standard_name_v ocabulary CF Standard Name Table (Version 19, 22 March 2012) project e.g., Global Space-based Inter-Calibration System http://gsics.wmo.int title e.g. GSICS MSG-2+SEVIRI vs ROLO1.0 lunar calibration file summary Lunar calibration file keywords “free content” references TBD institution “institution name” (e.g. EUMETSAT) licence “terms of use” creator_name e.g., EUMETSAT - European Organisation for the Exploitation of Meteorological Satellites creator_email ops@eumetsat.int creator_url http://www.eumetsat.int NAME CONTENT instrument e.g., MSG2 SEVIRI instrument_wmo_code (56, 207) date_created YYYY-MM-DDTHH:MM:SSZ date_modified history “free content” id file name wmo_data_category 30 for Calibration dataset (satellite) wmo_international_dat a_subcategory TBD (e.g. 5 for Re-Analysis Correction) processing_level e.g. “v1.0.0” atbd_url ATBD URL atbd_doi ATBD DOI time_coverage_start time_coverage_end phase_angle_min 2 [degree] phase_angle_max 92 [degree] reference_model e.g. ROLO version 3.11g Differences from the lunar observation netCDF Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

dimensions: lunar calibration netCDF NAME VALUE DESCRIPTION chan e.g. 4 for SEVIRI Number of channels chan_strlen e.g. 6 for SEVIRI Maximum length of variable: chan date 1 Number of date sat_ref_strlen 6 Maximum length of variable: sat_pos_ref (i.e. “ITRF93”) sat_xyz 3 X, Y, and Z axis of satellite position lambda_rolo 32 Number of ROLO spectrum lambda_apollo 221 Number of APOLLO spectrum obs_strlen Length of lunar observation NetCDF file name Differences from lunar observation netCDF Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

variables: lunar calibration netCDF 3 options (Basic, Partial: Basic+, Complete: Partial+) will be implemented NAME LONG NAME UNITS TYPE date[date] time of lunar observation seconds since 1970-01-01T00:00:00Z double precision floating point channel_name[chan,chan_strlen] channel identifier character, e.g. “VIS006” central_wavelength[chan] nominal channel central wavelength µm irr_obs[date,chan] observed moon irradiance W m-2 µm-1 irr_rolo[date,chan] rolo irradiance dc_obs[date,chan] integrated digital counts of lunar observation 1 32-bit integer dc_obs_offset[date, chan] averaged digital counts offset of deep space moon_pix_num[date,chan] number of moon pixels pix_solid_ang[date,chan] pixel solid angle sr ovrsamp_fa[date,chan] oversampling factor moon_pix_thld[date,chan] digital counts threshold for moon mask sat_selen_lat[date ] satellite selenographic latitude degrees sat_selen_lon[date] satellite selenographic longitude dist_sat_moon[date] satellite moon distance km sun_selen_lat[date] sun selenographic latitude sun_selen_lon[date] sun selenographic longitude dist_sun_moon[date] sun moon distance AU phase_ang[date] moon phase angle sat_pos_ref[sat_ref_strlen] satellite position x y z character, either “J2000” or “ITRF93” sat_pos[date, sat_xyz] satellite position sun_pos[date, sat_xyz] sun position moon_pos[date, sat_xyz] moon position Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

variables: lunar calibration netCDF 3 options: Basic, Partial: Basic+, Complete: Partial+ NAME LONG NAME UNITS TYPE rolo_spectr[date, chan, lambda_rolo] rolo reflectance spectrum at rolo wavelength 1 double precision floating point rolo_lambdas[lambda_rolo] rolo wavelength µm rolo_smooth_spectr[date, chan, lambda_apollo] rolo reflectance spectrum at apollo wavelength apollo_lambdas[lambda_apollo] apollo wavelength m double precision moon_obs_file[date, obs_strlen] lunar observation file name character Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

Open issue: handling of lunar observation time At present, GIRO does not use this “elapsed seconds” because… NO clear definition whether “leap seconds” should be considered or not Possible solutions a) NO leap seconds in variable: date (= “UNIX (epoch) TIME”) Easy conversion of elapsed seconds to other time format GIRO takes into account of leap seconds in JPL SPICE library b) variable: date contains exact “seconds since 1970-01-01T0:00:00Z” Each data provider has to check leap seconds Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

Summary All input/output files for the GIRO: netCDF Lunar observation file (input) Instrument SRF file (input) Lunar calibration file (output) Open issues Handling of the lunar observation time, separated lunar observation files for different channels, etc. We really appreciate your cooperation for providing SRF data and preparation of the lunar observation dataset! That enables us to share lots of lunar calibration results using the common reference: GIRO. Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

Thanks for your attention Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

General information about netCDF http://www.unidata.ucar.edu/software/netcdf/docs/ easy-to-understand User’s guide and tutorials Utilities: many CUI/GUI apps, ncdump/ncgen are quite useful http://www.unidata.ucar.edu/software/netcdf/docs/html_guide/netcdf_utilities_guide.html ncdump: convert NetCDF file to text form (CDL) ncgen: generates a NetCDF file or a C/FORTRAN code that creates a netCDF dataset from CDL “NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. NetCDF was developed and is maintained at Unidata. ...” (from Unidata website above) Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

Open issue: usage of separated files for different channels Single file for each lunar occurrence, including all-channel information Two possible solutions in order to use separate files: 1) To keep the same file names, but put in different directories Easy to implement 2) Change file names E.g., MSG2/SEVIRI L1.5 netCDFs on the EUMETSAT THREDDS server: W_XX-EUMETSAT-Darmstadt,VIS+IR+IMAGERY,MSG2+SEVIRI_C_EUMG_20141124222009.nc W_XX-EUMETSAT-Darmstadt,HRV+IMAGERY,MSG3+SEVIRI_C_EUMG_20140409104510.nc GIRO has to be updated to support this file naming Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

NetCDF-4 variable compression function GMS-5/VISSR lunar observation file w/ imagettes 4.2MB File size (KB) 379 364 337 316 296 278 273 264 261 (KB) 2 3 4 5 6 7 8 9 compression level w/o compression Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany