Agenda Item 7.1: Aggregation Rules 2. Workshop of TT-WDQMS 12.12. – 14.12.17 @ ECMWF

Outcome of Meetings Meeting at WMO on Nov. 6th 2017 and discussion during the Project Steering Meeting of OSCAR/Surface about the strategic role of WDQMS – OSCAR/Surface – Country Profile DB It was considered to be important to clarify the purpose and scope of each application (for WDQMS not only for the Pilot but for a potential final solution) To define the users to be addressed with each of the applications To clarify the flow of information between the systems in order to avoid redundancy and guarantee consistency. Additional discussion on a more general level about the need to agree on the most useful granularity (or aggregation) of information in the three components, considering that they should not duplicate information unnecessarily and also address the different needs of different users (WMO SG vs an NMHSs network manager).

WDQMS, OSCAR and CPDB in the context of monitoring (strategic level) Primary purpose Describe how well WIGOS is providing required observations, and report findings, incidents (Quality monitoring & incident mgt. of global/national assets) Understanding observational data, GAP analysis, support WDQMS, support CPDB, use as national DB (Technical doc/analysis of global/national assets) To provide national perspective on capacities of Members, contact information, country KPIs, institutional context. Hi-level doc/analysis of global/national assets Users Members, Regional WIGOS centers, users of data Network planners/managers, global/regional/national levels, data users (scientists, eg. climatologists), international organizations SG, donors, WMO staff, WMO regional offices , international organizations Variable scope Individual observed variable (P, T, U, more if available) (P, T, U, radiation, chemical composition, hydrology, …) Overall KPI of observing systems Temporal aggregation 6-hourly or else highest resolution available Monthly (actual as well as historical) Monthly or yearly? Spatial aggregation Individual location of observation (station level) Country level KPI = key performance indicator

WDQMS, OSCAR and CPDB in the context of monitoring (strategic level) Imports Displays Declared status, schedules (present) «real» status (weekly) Contacts, organizations Declared status (monthly) «real» status (monthly) KPIs on metadata completeness KPIs on Focal Point Activity Exports «real» status (daily) Declared status, schedules (present, past) KPIs on metadata completeness Focal Point Activity KPIs for GOS, GAW, GCW, … Contacts, organizations (?) Access Restricted to Focal Points Public KPI = key performance indicator Agreement: monthly aggregated values for each calendar month and each variable in OSCAR/Surface (not only actual but also historical)

Purpose & Users of WDQMS

Guidance on Quality Monitoring, Evaluation and Incident Management Proceduresfor Regional WIGOS Centres (RWC) WIGOS Evaluation Function: takes the Quality Monitoring outputs from all the contributing WIGOS Monitoring Centres, .. generates routine daily performance reports based on at least two performance indicators, which are at the moment; a comparison with the status of WIGOS as described in WIS/OSCAR; trends in network performance over a suitable period (for GOS elements monthly rolling averages are proposed).  New element “Schedule of International Exchange” in WIGOS Metadata Standard  shall also be added to OSCAR

Aggregation within WDQMS: Daily aggregation across Monitoring centers So far .. ECMWF JMA NCEP EUCOS ….. Monitoring Results Monitoring Results Monitoring Results Monitoring Results Monitoring Results e.g. 6 out of 8 5 out of 8 7 out of 8 6 out of 8 …. To be done .. Aggregation 1 value / variable / station / day

Daily aggregations WDQMS Challenge: results of the different WIGOS Monitoring Centres might differ. how to deal with this, if RWC shall only initiate an Incident Management Process if the majority of WIGOS Monitoring Centres show similar results Does not only concern Data availability, but also Timeliness and Accuracy Aggregated value needed for OSCAR/Surface and probably also CPDB  user requirements not really clear. Rules: Data Availability: evtl. already implemented by WDQMS Tool and if so, how? Proposal: value considered to be reported if at least one of the monitoring centres is using it. Timeliness and Accuracy: ?? Average over all centers? Information comparable?

Outputs of WDQMS Data availability: total number of meteorological bulletins (TAC/BUFR) received during a defined period (e.g. 24 hours) compared to the required number of bulletins as determined by the observing schedule New element “Schedule of International Exchange” in WIGOS Metadata Standard  shall also be added to OSCAR Timeliness: delay between the nominal observation time of a particular observation message type issued at site of a Member and reception time at users´ database of this message received via GTS. Accuracy: Combining trueness and precision as outlined in ISO5725 (mainly derived from “Observation minus Background” (O-B) NWP results from Global NWP Centres for particular parameters such as air pressure, air temperature, wind and relative humidity observations. The Web-tool should allow the statistics to be filtered, for example by country or to display only those stations which exceed targets on data availability, timeliness or measurement uncertainty (bias, standard deviation, mean vector difference, root mean square vector difference).  requirements not very clear yet (in particular for CPDB)

Purpose & Users of OSCAR/Surface

Aggregation for OSCAR/Surface Request by ICG-WIGOS: «3.1.4 ICG-WIGOS requested that “quantitative monitoring information from the WDQMS to become part of the station report in OSCAR/Surface”, which will provide valuable additional information.” What do OSCAR/Surface Users really need (Task 9)?  different attempts to find out but still not really clear Output of Meetings (6/11/17 and 20/11/17): OSCAR/Surface wants: an aggregated value per observation for each calendar month Not only actual but also historical

Aggregation per variable: Display? In discussions how and where to implement this information it showed, that It isn’t clear, if this aggregated value has to be calculated for each observation in relation to the program to which it contributes .. The user might apply different criteria to check the needed availability Display of reporting status: Do we use categories and if so  related to Thresholds of Requirements (would have to be set specifically depending on priority, region etc)

Generic vs specific information Another question: where to put the information?  Observations:  not a specific setting but the observation in general or  Deployments  specific setting of how it is carried out .. or here.. or? Here …

What information is needed in OSCAR/Surface? Is only a performance status required per variable or is there any need for a «station»-wise point of view?  aggregation across variables necessary?  if no guidance: do people not do it anyway? Regularly reporting Occasionally reporting Not reporting

Possible Solutions Challenges: We still don’t know the concrete requirements about how this information is going to be used within OSCAR/Surface We can decide it’s not needed to come up with a guideline how to aggregate across variables but people might nonetheless start to do it  guideline how to do would be helpful Challenge of categorisation (e.g. «reporting» = % of reported values)  applied categorisation can differ from country to country depending on used criteria. Relation to thresholds of requirements? Proposals: Aggregation across «Station»:  define order of variables which should be used to determine performance status instead of implementing a complex aggregation algorithm. Not definitely specified user needs:  create table (see next slide) to deliver basis in order that they can calculate in OSCAR/Surface their required actions itself.

Data delivered to OSCAR/Surface Proposal to create table for each variable and day Month of the year Facility Variable 1.11.17 * # exptected values as actually used in the monitoring systems (dynamic meta data) Results created by Monitoring system Delivery: once a month Analytics in OSCAR/Surface:  Guidelines should be provided by WDQMS total availability Monthly variations Date or day of month # exptected values* # reported values 1.12.17 8 6 2.12.17 7 3.12.17 …. …

Purpose & Users of Country Profile DB

Aggregations for CPDB Opinions differ as to whether it makes sense for CPDB to show the list of stations of each country in the CPDB or rather display summary statistics and a KPI of the observing capability of a country  link to OSCAR to show the list of stations and KPIs for each individual one. It needs to be clarified what are the exact needs who knows them? WDQMS shouldn’t have to worry about it but provide the necessary basic information (analogous to OSCAR/Surface) and give guidance how to calculate e.g. yearly values or KPIs Proposal: Aggregations calculated based on same information given to OSCAR/Surface This discussion needs to be continued and agreement be found, such that all 3 components can flourish side by side and have their specific raison d'être but that there is consistency of the information provided.