1. GIRO Input/Output data convention
Masaya Takahashi (JMA),
Bartolomeo Viticchie (EUMETSAT)
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

2. GIRO Input/Output files
We decided to use netCDF for both input and output files
Inputs: lunar observation and instrument SRF files
Output: lunar calibration file
Why?
From the project framework point of view
GIRO is a GSICS Implementation of the ROLO model
GSICS adopts (classic) netCDF format for the inter-calibration products
Lunar calibration: one of the future inter-calibration method for visible and near infrared channels
GIRO will be used for the GSICS inter-calibration purpose
From coding point of view
Common format data: easy-to-handle (read, write)
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

3. GSICS file-naming/netCDF convention
GSICS netCDF convention
Follows the netCDF CF (Climate and Forecast) Convention widely used in the netCDF community (
GSICS file naming convention
 Follows the rules given in the General File Naming Conventions section of the
W.M.O. Manual on The Global Telecommunication System
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

4. Input-1: lunar observation netCDF
File name (e.g., MTSAT-2/Imager)
W_JP-JMA-MSC,VISNIR+SUBSET+MOON,MTSAT2+Imager_C_RJTD_ _01.nc
satellite and instrument name
lunar observation time as yyyyMMddhhmmss
Single file for each lunar occurrence, including all-channel information
Variable units: must be the same as those in a template
Recommended to use classic netCDF4 format
Could be used as “GSICS source dataset” in the future
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

5. global attributes: lunar observation netCDF
No impact on the GIRO results, but very important for the dataset to be shared within GSICS (+ α) community in future
NAME
CONTENT
Conventions
CF-1.6
Metadata_Conv entions
Unidata Dataset Discovery v1.0
standard_name_ vocabulary
CF Standard Name Table (Version 19, 22 March )
project
e.g. Global Space-based Inter-Calibration System
title
e.g. MSG-2+SEVIRI lunar imagtette file
summary
e.g. Lunar observation file
keywords
“free content”
references
TBD
institution
e.g. EUMETSAT
licence
“terms of use”
creator_name
e.g., EUMETSAT - European Organisation for the Exploitation of Meteorological Satellites
NAME
CONTENT
creator_
e.g.
creator_url
e.g.
instrument
e.g. MSG2 SEVIRI
instrument_wmo_code
e.g. (56, 207)
data_source
“free content” e.g., Level 1.0
date_created
YYYY-MM-DDTHH:MM:SSZ
date_modified
history
“free content” id
File name
wmo_data_category
101 for Imager data(satellite)
wmo_international_dat a_subcategory
0 for Multi-purpose VIS/IR imagery
processing_level
e.g. “v1.0.0”
doc_url
URL of document
doc_doi
DOI of document (if exists)
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

6. dimensions/variables: lunar observation netCDF
NAME
VALUE
DESCRIPTION
chan
e.g. 4 for SEVIRI
Number of channels
chan_strlen
e.g. 6 for SEVIRI
Maximum length of variable: chan
date 1
Number of date
sat_ref_strlen 6
Maximum length of variable: sat_pos_ref (i.e. “ITRF93”)
sat_xyz 3
X, Y, and Z axis of satellite position
col, row
e.g. 499 for SEVIRI
Number of columns and rows of the imagette
Compulsory
Imagette (optional)
Additional (optional)
variables
NAME [dimension]
LONG NAME
UNITS
TYPE
date[date]
time of lunar observation
seconds since T00:00:00Z 
double precision floating point
channel_name[chan,chan_strlen]
channel identifier
character ,e.g. “VIS006”
sat_pos[sat_xyz]
satellite position km
sat_pos_ref[sat_ref_strlen]
satellite position x y z
character, either “J2000” or “ITRF93”
irr_obs[chan]
observed moon irradiance
W m-2 µm-1
rad_obs_imgt[row,col,chan]
observed moon radiance imagette
W sr-1 m-2 µm-1
dc_obs_imgt[row,col,chan]
observed moon digital counts imagette 1
32-bit integer array
moon_pix_thld[chan]
digital counts threshold for moon mask
32-bit integer
pix_solid_ang[chan]
pixel solid angle sr
ovrsamp_fa[chan]
oversampling factor
dc_obs[chan]
integrated digital counts of moon observation
dc_obs_offset[chan]
averaged digital counts offset of deep space
moon_pix_num[chan]
number of moon pixels
Possible to handle floating point counts using attributes: scale_factor and add_offset
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

7. Input-2: instrument SRF netCDF
File name (e.g., Meteosat-10/SEVIRI):
W_XX-EUMETSAT-Darmstadt,VIS+IR+SRF,MSG3+SEVIRI_C_EUMG.nc
Prepared by GSICS Data management Working Group (GDWG, mainly by M.Takahashi) using the SRFs provided by each participant
Some participants generate the file by themselves due to a practical reason
Some files provided by GSICS members are available on the GSICS Wiki
Potential dataset as the “GSICS Standard SRF” (to be discussed in the near future)
Originally, generated by GDWG in 2013 for the purpose of GSCIS product developments
“This file was produced in support of GSICS activities and thus is not meant for public use although the data in the file is in the public domain.” (from docs on the NOAA GSICS Data server)
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

8. Output: lunar calibration netCDF
File name (e.g., Meteosat-7/MVIRI):
W_XX-EUMETSAT-Darmstadt,SATCAL+ROLOVISNIR,MET7+MVIRI_C_EUMG_ _01.nc
Single file contains all-time, all-channel information
follows the GSICS approach to generate inter-calibration products (e.g., GEO-LEO-IR Re-Analysis Correction)
Contents
Contents of a lunar observation file (except imagette) + ROLO reference irradiance, the Moon phase angle, positions of the Moon, the Sun, etc.
Starting point for developing a GSICS inter-calibration product
Concept: applicable to the future GSICS GEO-LEO VIS/NIR products
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

9. global attributes: lunar calibration netCDF
Details will be discussed/updated in the future
NAME
CONTENT
Conventions
CF-1.6
Metadata_Conve ntions
Unidata Dataset Discovery v1.0
standard_name_v ocabulary
CF Standard Name Table (Version 19, 22 March 2012)
project
e.g., Global Space-based Inter-Calibration System
title
e.g. GSICS MSG-2+SEVIRI vs ROLO1.0 lunar calibration file
summary
Lunar calibration file
keywords
“free content”
references
TBD
institution
“institution name” (e.g. EUMETSAT)
licence
“terms of use”
creator_name
e.g., EUMETSAT - European Organisation for the Exploitation of Meteorological Satellites
creator_
creator_url
NAME
CONTENT
instrument
e.g., MSG2 SEVIRI
instrument_wmo_code
(56, 207)
date_created
YYYY-MM-DDTHH:MM:SSZ
date_modified
history
“free content” id
file name
wmo_data_category
30 for Calibration dataset (satellite)
wmo_international_dat a_subcategory
TBD (e.g. 5 for Re-Analysis Correction)
processing_level
e.g. “v1.0.0”
atbd_url
ATBD URL
atbd_doi
ATBD DOI
time_coverage_start
time_coverage_end
phase_angle_min
2 [degree]
phase_angle_max
92 [degree]
reference_model
e.g. ROLO version 3.11g
Differences from the lunar observation netCDF
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

10. dimensions: lunar calibration netCDF
NAME
VALUE
DESCRIPTION
chan
e.g. 4 for SEVIRI
Number of channels
chan_strlen
e.g. 6 for SEVIRI
Maximum length of variable: chan
date 1
Number of date
sat_ref_strlen 6
Maximum length of variable: sat_pos_ref (i.e. “ITRF93”)
sat_xyz 3
X, Y, and Z axis of satellite position
lambda_rolo 32
Number of ROLO spectrum
lambda_apollo
221
Number of APOLLO spectrum
obs_strlen
Length of lunar observation NetCDF file name
Differences from lunar observation netCDF
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

11. variables: lunar calibration netCDF
3 options (Basic, Partial: Basic+, Complete: Partial+) will be implemented
NAME
LONG NAME
UNITS
TYPE
date[date]
time of lunar observation
seconds since T00:00:00Z
double precision floating point
channel_name[chan,chan_strlen]
channel identifier
character, e.g. “VIS006”
central_wavelength[chan]
nominal channel central wavelength µm
irr_obs[date,chan]
observed moon irradiance
W m-2 µm-1
irr_rolo[date,chan]
rolo irradiance
dc_obs[date,chan]
integrated digital counts of lunar observation 1
32-bit integer
dc_obs_offset[date, chan]
averaged digital counts offset of deep space
moon_pix_num[date,chan]
number of moon pixels
pix_solid_ang[date,chan]
pixel solid angle sr
ovrsamp_fa[date,chan]
oversampling factor
moon_pix_thld[date,chan]
digital counts threshold for moon mask
sat_selen_lat[date ]
satellite selenographic latitude
degrees
sat_selen_lon[date]
satellite selenographic longitude
dist_sat_moon[date]
satellite moon distance km
sun_selen_lat[date]
sun selenographic latitude
sun_selen_lon[date]
sun selenographic longitude
dist_sun_moon[date]
sun moon distance AU
phase_ang[date]
moon phase angle
sat_pos_ref[sat_ref_strlen]
satellite position x y z
character, either “J2000” or “ITRF93”
sat_pos[date, sat_xyz]
satellite position
sun_pos[date, sat_xyz]
sun position
moon_pos[date, sat_xyz]
moon position
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

12. variables: lunar calibration netCDF
3 options: Basic, Partial: Basic+, Complete: Partial+
NAME
LONG NAME
UNITS
TYPE
rolo_spectr[date, chan, lambda_rolo]
rolo reflectance spectrum at rolo wavelength 1
double precision floating point
rolo_lambdas[lambda_rolo]
rolo wavelength µm
rolo_smooth_spectr[date, chan, lambda_apollo]
rolo reflectance spectrum at apollo wavelength
apollo_lambdas[lambda_apollo]
apollo wavelength m
double precision
moon_obs_file[date, obs_strlen]
lunar observation file name
character
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

13. Open issue: handling of lunar observation time
At present, GIRO does not use this “elapsed seconds” because…
NO clear definition whether “leap seconds” should be considered or not
Possible solutions
a) NO leap seconds in variable: date (= “UNIX (epoch) TIME”)
Easy conversion of elapsed seconds to other time format
GIRO takes into account of leap seconds in JPL SPICE library
b) variable: date contains exact “seconds since T0:00:00Z”
Each data provider has to check leap seconds
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

14. Summary All input/output files for the GIRO: netCDF
Lunar observation file (input)
Instrument SRF file (input)
Lunar calibration file (output)
Open issues
Handling of the lunar observation time, separated lunar observation files for different channels, etc.
We really appreciate your cooperation for providing SRF data and preparation of the lunar observation dataset!
That enables us to share lots of lunar calibration results using the common reference: GIRO.
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

15. Thanks for your attention
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

16. General information about netCDF
easy-to-understand User’s guide and tutorials
Utilities: many CUI/GUI apps, ncdump/ncgen are quite useful
ncdump: convert NetCDF file to text form (CDL)
ncgen: generates a NetCDF file or a C/FORTRAN code that creates a netCDF dataset from CDL
“NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. NetCDF was developed and is maintained at Unidata. ...”
(from Unidata website above)
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

17. Open issue: usage of separated files for different channels
Single file for each lunar occurrence, including all-channel information
Two possible solutions in order to use separate files:
1) To keep the same file names, but put in different directories
Easy to implement
2) Change file names
E.g., MSG2/SEVIRI L1.5 netCDFs on the EUMETSAT THREDDS server:
W_XX-EUMETSAT-Darmstadt,VIS+IR+IMAGERY,MSG2+SEVIRI_C_EUMG_ nc
W_XX-EUMETSAT-Darmstadt,HRV+IMAGERY,MSG3+SEVIRI_C_EUMG_ nc
GIRO has to be updated to support this file naming
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany

18. NetCDF-4 variable compression function
GMS-5/VISSR lunar observation file w/ imagettes
4.2MB
File size (KB)
(KB)
compression level
w/o
compression
Lunar Calibration Workshop, 1-4 December, 2014, Darmstadt, Germany