1. GHRSST-PP METRICS G. Corlett and I. Barton Friday 13 th June, 0900

2. Observation: ● The GHRSST Metrics presentation is always made on the last day of the meeting. ● Because metrics excite GHRSST Science Team Members more than any other topic, it’s place in the agenda is used by our leader as a strategy to ensure that no participant leaves the meeting before Friday.

3. Presentation content ● The Metrics document presented at GHRSST-8 has been consolidated into a new shorter document (Version 2.2) and is included in the documents provided for GHRSST-9. ● I know that everybody has read the document, but I will go through it briefly to remind you of its content. ● This is an interactive presentation so please comment if you have an issue with any of the content matter!!

4. Desired outcomes of the session ● Each RDAC and GDAC to agree their metrics. ● We need to establish a common template for reports that will provide a common format for all the DACs. This should be done off-line. ● Each RDAC and GDAC should implement their agreed metrics analysis and then report monthly to GHRSST PO or display analysis results on www.ghrsst-pp.orgwww.ghrsst-pp.org Suggested timelines (Action Items) ● Metric agreement by end July 2008 ● Metric analysis reports start by October 2008 (ACTION ITEM) ACTION ITEMS: 1. RDACs and GDACs to return completed metric parameters to Barton/Corlett by end July. 2. GDAC to provide a template for metric reporting ???

5. GHRSST Metrics Version 2.2 ● All data quality, coverage, and delivery time metrics are expressed as a percentage with a higher percentage being good. ● Thresholds for data delivery/availability times, biases, and rms values can be adjusted with experience and expectations once the data flows begin. ● Thresholds should be set in a way that helps with data quality and delivery, i.e. thresholds should not be set so that the metric is always 100%.

6. ● Metrics for data usage (downloads) and user-numbers are monthly sum totals. ● All metrics are to be reported each calendar month. ● Eventually metrics will need to be set for comparison of L4 fields against an ensemble average, or against single or selected sensors, or against model outputs. Guidelines for these metrics have been supplied by Jim Cummings, and these will form a basis for L4 data inter-comparisons before L4 comparison metrics can be set within the broader GMPE project.

7. Suggested Metrics are included in the Version 2.2 WORD Document included in the GHRSST-9 document list. RDAC and GDAC operators should check that their recommended metrics for data quality and supply are sensible

8. MeasureTargetNumeric metric OwnerDocumentation M1RDAC L2P delivery/availability time: Polar Orbiting Satellites L2P AVHRR GAC/LAC L2P AVHRR HRPT L2P MODIS-TERRA L2P MODIS-AQUA L2P AATSR L2P AMSRE L2P TMI L2P Orbit files available within N hours of data acquisition (orbit start time) N = 6 N = 3 N = 6 N = 3 N = 8 N = 12 Percent of times the target was met during the month. RDACRDAC L2P targets & metrics for delivery timeliness – ancillary data metrics can take a hit in order to meet end-user timeliness requirements M2RDAC L2P delivery/availability time: Geostationary Satellites L2P SEVIRI L2P GOES-E L2P GOES-W L2P MTSAT Full disc and sector files available within N hours of data acquisition N = 3 Percent of times the target was met during the month. RDAC

9. M3.1RDAC L2P acquisition of orbits Polar Orbiting Satellites L2P AVHRR GAC/LAC L2P AVHRR HRPT L2P MODIS-TERRA L2P MODIS-AQUA L2P AATSR L2P AMSRE L2P TMI Percent of available orbits/passes collected per month. Target for all is 98% 98 Percent of available orbits/passes collected per month. RDACRDACs should maintain a catalogue of missing orbits & data locations. M3.2RDAC L2P data coverage: Polar Orbiting Satellites L2P AVHRR GAC/LAC L2P AVHRR HRPT L2P MODIS-TERRA L2P MODIS-AQUA L2P AATSR L2P AMSRE L2P TMI Percent of orbits with >90% coverage/month Percent of times targets met/per month RDAC

10. M4.1RDAC L2P full-disk data acquisition: Geostationary Satellites L2P SEVIRI L2P GOES-E L2P GOES-W L2P MTSAT Percent of available scenes collected per month. Target is 98% Percent of available scenes collected per month. RDAC M4.2RDAC L2P data coverage: Geostationary Satellites L2P SEVIRI L2P GOES-E L2P GOES-W L2P MTSAT Percent of full- disk scenes collected with >90% coverage. Target is 98% Percent of full-disk scenes collected with >90% coverage. RDAC

11. M5.1RDAC L2P data quality: Polar Orbiting Satellites L2P bias against real- time in situ data L2P AVHRR GAC/LAC L2P AVHRR HRPT L2P MODIS-TERRA L2P MODIS-AQUA L2P AATSR L2P AMSRE L2P TMI Bias target (K) 1.2 0.8 0.6 1.2 Percent of times the bias is less (better) than the bias target. RDACEach RDAC will need to specify the timeliness and proximity of the in situ data used in their match-ups. Each will do this in their own way and using their own data – there will be a loss of independence and duplication of match- ups – but this will be unavoidable. M5.2RDAC L2P data quality: Polar Orbiting Satellites L2P rms against real- time in situ data L2P AVHRR GAC/LAC L2P AVHRR HRPT L2P MODIS-TERRA L2P MODIS-AQUA L2P AATSR L2P AMSRE L2P TMI Rms target (K) 1.2 0.8 0.6 1.2 Percent of times the rms is less (better) than the rms target. RDACEach RDAC will need to specify the timeliness and proximity of the in situ data used in their match-ups. Each will do this in their own way and using their own data – there will be a loss of independence and duplication of match- ups – but this will be unavoidable.

12. M6.1RDAC L2P data quality: Geostationary Satellites L2P bias against real- time in situ data L2P SEVIRI L2P GOES-E L2P GOES-W L2P MTSAT Bias target (K) 0.8 Percent of times the bias is less (better) than the bias target. RDACEach RDAC will need to specify the timeliness and proximity of the in situ data used in their match-ups. Each will do this in their own way and using their own data – there will be a loss of independence and duplication of match- ups – but this will be unavoidable. M6.2RDAC L2P data quality: Geostationary Satellites L2P rms against real- time in situ data L2P SEVIRI L2P GOES-E L2P GOES-W L2P MTSAT Rms target (K) 0.8 Percent of times the rms is less (better) than the rms target. RDACEach RDAC will need to specify the timeliness and proximity of the in situ data used in their match-ups. Each will do this in their own way and using their own data – there will be a loss of independence and duplication of match- ups – but this will be unavoidable. M7RDAC L2P ancillary data: Polar Orbiting Satellites L2P wind speed L2P AOD L2P SSI L2P Sea ice L2P DV estimate Percent of files with ancillary data 95 75 Percent of files with ancillary data per month RDACA loose metric for DV as it is not yet clear how this should be estimated in real- time.

13. M8RDAC L2P ancillary data: Geostationary Satellites L2P wind speed L2P AOD L2P SSI L2P Sea ice L2P DV estimate Percent of files with ancillary data 95 75 Percent of files with ancillary data per month RDACA loose metric for DV as it is not yet clear how this should be estimated in real-time. M9GDAC L2P data availability time L2P data from RDAC L4 data from RDAC HR-DDS from RDAC Data available at the GDAC within N minutes of completion of reception from an RDAC. N = 60 N = 5 Percent of data files made available with delays less than the target of N minutes. GDACAlso need a metric for L4 DDS files produced at the GDAC M10GDAC and RDAC: Number of data users: JPLGDAC EURGDAC EURDAC USRDAC JPRDAC AURDAC Target number of unique users 40 Number of unique users accessing GHRSST-PP data per month RDACs

14. M11RDAC: Number of data downloads: JPLGDAC EURGDAC EURDAC USRDAC JPRDAC AURDAC Target number of data downloads 400 Number of GHRSST-PP data downloads per month. RDACs M12.1Population of MMR: Details of all GHRSST files need to be deposited in the MMR within set target times. JPLGDAC EUGDAC EURGDAC USRDAC JPRDAC AURDAC All MMR files required within 1 hour of availability at the GDAC and RDAC. Target % 98 Percent of metadata files lodged within 1 hour during the month GDACs and RDACs M12.2Population of MMR: Details of all GHRSST files need to be deposited in the MMR each month. JPLGDAC EUGDAC EURGDAC USRDAC JPRDAC AURDAC All metadata files must be deposited in the MMR. Thus target is 100% 100 Percent of all metadata files lodged in MMR per month GDACs and RDACs

15. M13Population of DDS repository: JPLGDAC EUGDAC EURDAC USRDAC JPRDAC AURDAC DDS data files should be lodged with the DDS repository within 2 hours of L2P and L4 data availability. Target percentages are all 95% Percent of files produced within the target time per month. GDACs and RDACs M14GDAC and RDAC Data enquiries: JPLGDAC EUGDAC EURDAC USRDAC JPRDAC AURDAC Number of user enquires logged List of user feedback and number of problems addressed GDACs and RDACs M15User support: timely notification of system problems. JPLGDAC EUGDAC EURDAC USRDAC JPRDAC AURDAC A notice should be posted within 1 hour of problem identification when a problem in the R/GTS system affects service provision Percentage of notifications posted within 1 hour GDACs and RDACs

16. THE END

18. Your location: Home / http://www.ghrsst-pp.org/.htmlScience Team /Homehttp://www.ghrsst-pp.org/.htmlScience Team The GHRSST PP Multi Product Ensemble (GMPE) TAG The GHRSST-PP Multi-Product Ensemble (GMPE) Technical Advisory Group (GMPE-TAG) The GMPE-TAG is responsible for the coordination of all aspects of the GMPE experiments and real-time intercomparison systems maintained by producer/providers. It is chaired by Ian Barton (ian.barton@csiro.au).ian.barton@csiro.au GMPE activities  The current TAG-Team is listed herehere  The GMPE Terms of ReferenceGMPE Terms of Reference  Information regarding GODAE Metrics documentation can be found here.here  Information on L4 analysis intercomparison issues can be found in  a L4 SST inter-comparison discussion document written by Jim Cummings.L4 SST inter-comparison discussion document  The Group on Earth Observations (GEO) Action DA-06-03Group on Earth Observations (GEO)Action DA-06-03  (Use of Ensemble techniques) Operational daily ensemble of global SST Operational daily intercomparisons of various global SST products of the Met Office Operational SSTMet Office and Sea Ice analysis (OSTIA) together with animations, ensemble mean, standard deviation and number of observations can be found here. here

19. GMPE-TAG Members Ian Barton, CSIRO, Australia (Chair) John Stark, MetOffice, UK Chelle Gentemann, RSS, USA Helen Beggs, BoM, Australia Dave Poulter, NOCS, UK Eileen Maturi, NOAA/OSPD, USA Bruce McKenzie, NAVOCEANO, USA Jim Cummings, NRL, USA Jean-Fracios Piolle/ Emmanuelle Autret, IFREMER, France Tsurane Kuragano, JMA, Japan Craig Donlon, MetOffice, UK.

20. GMPE-TAG Terms of Reference The following terms of reference for the GMPE-TAG have been developed: 1.To co-ordinate SST multi-product ensemble activities within the GHRSST-PP in collaboration with other bodies and agencies as required (e.g., GEO) 2. To encourage the early population of the JPL GDAC with Level-4 (L4) SST data files. This includes both satellite and analysed SST products from RDACs, meteorological services, and other research and operational agencies. 3. To develop methods and metrics for the inter-comparison of the different SST products available on the GDAC. 4. When sufficient data products are available at the GDAC, to select a period for intense inter-comparisons and to assist agencies in the implementation of the inter-comparisons. 5. To develop techniques for producing a GMPE (GHRSST Multi-Product Ensemble) and to encourage its use by operational and research agencies.