1. WG5: STATUS of activities
WG5, COSMO SMC, 1-2 February 2017

2. Anastasia Bundel (1), Flora Gofa (2) (PP leaders),
PP INSPECT status
Anastasia Bundel (1), Flora Gofa (2) (PP leaders),
Dmitry Alferov (1), Elena Astakhova (1), Petra Baumann (4), Dimitra Boukouvala (2), Ulrich Damrath (3), Pierre Eckert (4), Alexander Kirsanov (1), Xavier Lapillonne (4), Joanna Linkowska (5), Chiara Marsigli (6), Andrea Montani (6), Anatoly Muraviev (1), Elena Oberto (7), Maria Stefania Tesini (6), Naima Vela (7), Andrzej Wyszogrodzki (5), André Walser (4), and Mikhail Zaichenko (1)
(1) RHM (2) HNMS, (3) DWD, (4) MCH, (5) IMGW-PIB, (6) ARPA-SIMC, (7) ARPA-PT
PP INSPECT status, COSMO SMC, 1-2 February 2017

3. PP INSPECT
INSPECT PP tries to summarize the COSMO experience of applying spatial verification methods to forecast systems of very-high-resolution (1-3 km) compared to high-resolution models
INSPECT runs in parallel to MesoVICT (several INSPECT tasks involve reruns of COSMO high and very-high-resolution models for MesoVICT test cases)
Same as MesoVICT, INSPECT will focus on the ensembles and variables besides precipitation
In addition to targeting the goals of MesoVICT, INSPECT is expected to provide COSMO users more choice of verification domains and reference data - newer and longer periods, two complex terrains (the Alps and the Caucasus)
Finally, INSPECT will try to provide criteria for deciding which methods are best suited to particular applications

4. Status highlights
The PP INSPECT is extended until August 2017 due to delays in some tasks, with overall FTE amount almost unchanged
Delayed tasks in question are extended with unused FTEs shifted to later periods
PP INSPECT status, COSMO SMC, 1-2 February 2017

5. 51 ensemble runs; fc+96h; 3-hourly post-processing; TL639L91 (~40 km)
Tasks involving reruns of MesoVICT test cases: completed ongoing pending
MCH: COSMO-1 reruns for ALL MesoVICT cases are DONE and available at WG5 repository
ARPA-SIMC: ECMWF-IFS reruns for cases 1 and 2 Update of the status is expected
COSMO-Ru2-EPS: rerun of 10 members using clusterisation procedure for the 1st MesoVICT case, rerun of complete ensemble (51 member) is under preparation
MCH: COSMO-E rerun is completed
Thus, all necessary data are or will be available soon, but with
some delays. Thus, the application of spatial methods to the
ensembles (INSPECT priority) is also delayed for some tasks.
51 ensemble runs; fc+96h; 3-hourly post-processing; TL639L91 (~40 km)
Case 1 (Core Case):
initial times: , , ,
Class=it
Expver=b091
Case 2
initial times: , , ,
Expver=b092
PP INSPECT status, COSMO SMC, 1-2 February 2017

6. Tasks involving development of routines for running spatial methods
RHM, HNMS, INGW-PIB: examples of scripts for neighborhood methods, CRA, SAL, and MODE are uploaded at the WG5 repository
ARPA-PT: Works are ongoing to add two new developments in VAST: inclusion of time dimension and the possibility to operate with other variables besides precipitation, primarily TCC (to be concluded in Jan 2017)
PP INSPECT status, COSMO SMC, 1-2 February 2017

7. The policy on Additional Verification Tools and verification data and formats
HNMS, RHM, IMGW-PIB: The policy on Additional Verification Tools and verification data and formats is prepared and approved by the STC
7.7 Policy on common verification data formats
The STC reviews the document on policy on verification data formats prepared by WG5. The aim of the policy is to support COSMO policy on Additional Verification Tools (AVTs) and needs of PP INSPECT.
P. Steiner notes that the recommendations in the document need to be more specific and the recommendation that the future development of COSMO data processing software should aim “..to fulfil the requirements of verification applications” does not sound clear. M. Ziemianski notes that the document enumerates main data formats (for forecasts as well as point and gridded observations), important in the context of verification needs, and tool to handle them. D. Majewski comments that there may be new needs and the document should not be too restrictive; if necessary, new external convertors can be used.
The STC approves the document “Requirements for model output and observation data format for Verification Tools” and thanks the authors for its preparation.
PP INSPECT status, COSMO SMC, 1-2 February 2017

8. Tasks involving the application of spatial methods to deterministic forecasts
The routines developed earlier are applied to MesoVICT cases -> this should facilitate the comparison of different approaches and summarizing the results
However, same as for MesoVICT, it was found that the setup of experiments was not completely identical (different precipitation accumulation periods, time slots, etc.), and we should bear this in mind making conclusions about different methods
PP INSPECT status, COSMO SMC, 1-2 February 2017

9. Status
Wind verification using DIST for both wind speed and direction (Maria-Stefania)
HNMS, IMGW-PIB: comparisons of traditional and spatial scores for extreme events from MesoVICT cases (SAL, MODE, neighborhood)
RHM: CRA experiments using R SpatialVx for the Sochi region and MesoVICT cases
PP INSPECT status, COSMO SMC, 1-2 February 2017

10. Tasks involving the application of spatial methods to ensemble forecasts
A progress is made, although some of these tasks are delayed (RHM ensemble reruns)
SAL for COSMO-LEPS
DIST method applied to the COSMO-LEPS ensemble for MesoVICT cases using two sets of observations
PP INSPECT status, COSMO SMC, 1-2 February 2017

11. SAL for COSMO-LEPS 21/6 12 UTC 22/06 12UTC
S> 0 too large or too flat objects are produced.
A<0 underestimation, More red points
PP INSPECT status, COSMO SMC, 1-2 February 2017

12. Objective verification of COSMO-LEPS
Main features:
variable: 6h cumulated precip (0-6, ..., UTC);
period: all mesoVICT cases (Jun – Sep 2007);
region: 43-50N, 2-18E (D-PHASE area);
method: NGP, BILIN, BOXES of different sizes;
obs: JDC or VERA;
fcst ranges: 0-6h, 6-12h, ..., h;
thresholds: 1, 5, 10, 15, 25, 50 mm/6h;
system: COSMO-LEPS;
scores: ROC area, RPS, RPSS, Outliers, ...

13. Verification with boxes of the distributions (DIST)
The verification can be made in terms of:
Average value
Maximum value
50th percentile (Median)
75th, 90th, 95th percentiles
in a box
Operationally, we consider two measures of precipitation:
the average volume of water deployed over a specific region;
the rainfall peaks occurring within the same region.
Station observation
Grid point forecast
A.Montani

14. Probabilistic prediction of tp: ROC area (boxes)
tp_06h > 1mm
Consider the event: average 6-hour precipitation exceeding 1 mm within boxes of increasing size
0.5 x 0.5
1.0 x 1.0
More and more similar skill for the 2 verification networks as the box size increases.
Slightly higher skill when COSMO-LEPS is verified against VERA gridded analysis.
The skill increases with increasing box size.
1.5 x 1.5
A.Montani; The COSMO-LEPS system.

15. Follow-up of MesoVICT
The 2nd MesoVICT international meeting was held in Bologna, Italy, on September 2016 under the umbrella of ARPA-SIMC.
During the same days/place, the INSPECT meeting also took place
PP INSPECT status, COSMO SMC, 1-2 February 2017

16. Need for geometric cases for complex terrain
Main outcomes of the MesoVICT meeting in Bologna and intersection with INSPECT
Most spatial methods applied so far indicate that the results are sensitive to accum. period, domain size, number of objects, and model resolution
Need for geometric cases for complex terrain
Include time space (Maria Stefania, VAST)
Develop method ranking idea (some ideas of Anatoly Muraviev)
Include uncertainty in observations (Andrea, others??)
inference (statistical testing)
testing SpatialVx (RHM, IMGW-PIB)
information exchange of basic settings (! Necessary for comparison of methods)
Overview results paper, special AMS collection? (Also the need of publications for INSPECT final report)
PP INSPECT status, COSMO SMC, 1-2 February 2017

17. Task 5: Guidelines for relative usefulness of various spatial methods in decision-making
Goal: Using the knowledge gained from INTERP, ICP and MesoVICT projects and the long time series of spatial verification scores, to try to identify the relative usefulness of each spatial method for precipitation and other weather parameters. To propose a kind of Guidelines for using spatial methods within the COSMO community
Most of subtasks in Task 5 are delayed.
To simplify the preparation of the Task 5 deliverables, a questionnaire was sent to participants, the answers are expected in mid-March
PP INSPECT status, COSMO SMC, 1-2 February 2017

18. Questionnaire sent to INSPECT participants
Method applied (related to an INSPECT Task) and objectives
Short description of the dataset (forecast-observation data), adaptation required, software for the method application
Main findings (plots and explanation)
Characteristics of the method applied:
efficiency in calculation time
ability to deal with different density of observations
stability against observation errors
ability to assess the added value of high-resolution models
ability to address specific issues of interest (e.g. location errors, intensity errors, performance at different scales) etc.
Other
Possible comparison with other methods applied by the user on the same dataset
Other comments (optional)
PP INSPECT status, COSMO SMC, 1-2 February 2017

19. Common Plot Reports 2016-2017 FTE attribution (since sep 2014)
STC was in favor of the proposition for FTE attribution for common plots
STC suggests to restrict the task to the plots on the common area, which are the ones bringing a benefit since really comparable. STC requests for a deeper analysis in the report
STC suggests that FTE should be required to perform some additional explanation of results which will help to detect and improve outdated installations and correct model setups.
STC decides to keep conditional verification in the task, but requests that the conditions should be decided every year new by the WG3a/b.
WG5, COSMO SMC, 1-2 February 2017

20. operational coarse (-7km) res models
DWD (ICON-EU)
IMGW
RHM
DWD (ICON-EU)
MCH
COMET
NMA
HNMS
WG5, COSMO SMC, 1-2 February 2017

21. Standard Verification on Common Area
WG5, COSMO SMC, 1-2 February 2017

22. CP HRES scenario 2 WG5, COSMO SMC, 1-2 February 2017 Scenario 2 model
resolution
COSMO-IT2
0.02
COSMO-1
0.01
COSMO-DE
0.025
COSMO-ME5
0.045
COSMO-GR4
0.04
COSMO-PL
WG5, COSMO SMC, 1-2 February 2017

23. operational COSMO EPS models
IMGW (0.025)
LEPS (0.0625)
DWD (0.025)
COMET (0.09)
MCH (0.02)
WG5, COSMO SMC, 1-2 February 2017

24. Common Plot Reports
Responsible for CP reports and analysis A.Kirsanov (RHM)
Guidelines report for current year was prepared by WGc
Keep the coarser resolution comparison (~5-7km) for one year (2mT, 2mTd, 10mWsp, MSLP, 6h-24h Preci)
Add 12UTC run
Add high res model comparison on a common area with restricted model participation
Motivation that models can predict extreme values associated with dangerous weather (rare binary events). Extremal dependence scores were added on seasonal reports (EDI – SEDI for preci)
Operational EPS verification is postponed to be discussed next year
Conditional verification not included
Webpage is updated with first two seasons : JJA2016, SON2016

25. Issues to be considered
Purpose of CP statistics and reports; Current use
Maintain VERSUS as common software for the score preparation (Verification tools strategy – agenda item)
Apply other more suitable scores or methods in CP activity (spatial, upper air, ECMWF headline scores)
Conditional verification: Can this be transferred to individual testing; Definition of a PT with WG3 support ? (agenda item)
Communication of CP-derived information to COSMO management and other WGs (Conditional Verification). Perform several individual CV tests and organize meeting with WG3 to be presented and discussed
WG5 COSMO SMC, 1-2 February 2017

26. Maintenance phase since Sep. 2016 - VERSUS SCA : Antonio Vocino
VERSUS: Common Verification Software
EPS score discrepancy (CRPS) testing is pending
Problems with bugs revealed during Common Plots activity and NWP test suite – slow response is creating problems
Maintenance phase since Sep VERSUS SCA : Antonio Vocino
“The Long-Term Maintenance services are planned to be assured by C.O.Met (the new Operational Meteorological Center Of the Italian Air Force) for 2016/2017 COSMO year , conditionally to the available resources”
WG5, COSMO SMC, 1-2 February 2017

27. WG5 repository http://www.cosmo-model.org/rep/repository/wg5
New WG5 repository was established ion new COSMO server (T. Andreadis). It is aimed to host all relevant material to Common Plots presentation (guidelines, BUFR meansurements), PP INSPECT material, AVT (additional verification software) as described in Policy for verification input format, etc.
WG5, COSMO SMC, 1-2 February 2017

28. ADDITIONAL SLIDES

29. Common Verification Plots Conditional Verification experiments

30. Common Verification Plots for Common Area
Conditional Verification
2mT verification with the following criteria (one condition):
Soil water content ≥4 (condition based on forecasts)
Soil water content <2 (condition based on forecasts)
Wind Speed verification with the following criteria (one condition):
Roughness length <0.2 m(condition based on forecasts)
Roughness length >1m (condition based on forecasts)
Need to report the sample size or percentage of cases in each category
Easier to draw conclusions when difference from unconditioned errors on parameters are plotted

31. Cooperation with WG3a (2014-2015): Processing of verification feedback on model development
Scope: Direct and Indirect effects of the option of COSMO model that considers the interaction between the turbulence scheme and the wind tendency due to the presence of subgrid scale variability (LTKESSO) will be evaluated.
2mT, Wind speed and MSLP verification with the following criteria (one condition):
subgrid scale orography variance (SSO_STDH) ≤25m (condition based on forecasts)
subgrid scale orography variance (SSO_STDH) ≥100m (condition based on forecasts)
17th COSMO General Meeting, Wroclaw - WG5 parallel session, : Common Plot reports

32. Conditional Verification experiments: 2015-2016
With Communication with WG3b Coordinator
Study: Impact of soil type and vegetation height on the performance of various weather parameters in the lower atmosphere
Reasoning: The soil-vegetation representation in the model involves the fluxes of energy and water at the surface and determines the exchange of heat, moisture and momentum between the surface and the atmosphere. This has consequently an impact on near surface weather parameters (temperature, dewpoint, wind)
Scope: Evaluate the relevant effect in bias of modeled weather elements due to the variability on terrain characteristics
17th COSMO General Meeting, Wroclaw - WG5 parallel session, : Common Plot reports

33. Task description Score Production
Score Production
Preparation of input data and calculation of seasonal statistics over a common area according to the guidelines derived on an annual basis from WG5 ( /verification.priv/common/guidelines.pdf ) for each participating model. This Task includes conditional verification tests performed over this area. IFS and ICON driving model statistics have also been added.
Reporting
Processing of data from all models for each parameter and conditional verification test in appropriate format
R scripting for production of graphs (cross model representation)
Preparation of final annual report
Commenting of significant errors or discrepancies between models
Preparation of web graphics
Long term trend calculations
WG5, COSMO SMC, 1-2 February 2017

34. Main Issues to be considered
Purpose of CP statistics and reports, current use
Main focus of operational verification in each service
Maintain VERSUS are common software for the score preparation
Conditional verification: Can this be transferred to individual testing; Definition of a PT with WG3 support
Apply other more suitable scores or methods in CP activity (spatial, upper air, ECMWF headline scores)
Availability and number of SYNOP reports for each sub-area (scenario): necessary to perform analysis (volunteer?)
Communication of CP-derived information to COSMO management and other WGs (Conditional Verification) – Preparation of short report to be distributed to WG3a,b
Currently only 00UTC run is tested
WG5, COSMO SMC, 1-2 February 2017

35. Decisions from GM2016:
A. Keep present common coarse domain for an additional year either with fewer models and/or adding 4-5km (COMET,HNMS) to the list
Pros: keep long term trend
Cons: restrict participation and usability of feedback
B. Substitute with HRES verification over multiple domains and varying resoltutin
Pros: test model performance over a variety of geographical characteristics (how many and which?), participation of more services
Cons: increase work load for score extraction, increase work load of CP analysis, loss of long term trend
C. Add HRES (one domain) to current test (A)
Pros: test model performance over a variety of geographical characteristics (how many???), participation of more services, keep long term trend
Cons: increase further work load for score extraction, increase work load of CP analysis, loss long term trend
D. Expand operational verification for CP to EPS
WG5, COSMO SMC, 1-2 February 2017

36. Verification grid (e.g. 1.0x1.0)
OBSERVATION MASK
Verification grid (e.g. 1.0x1.0)

37. Probabilistic prediction of tp: ROC area (ngp vs bilin)
Area under the curve in the HIT rate vs FAR diagram; the higher, the better …
Valuable forecast systems have ROC area values > 0.6.
Consider two events: 6-hour precipitation exceeding 1 mm and 10 mm.
tp_06h > 1mm
tp_06h > 10mm
NGP vs BILIN: similar COSMO-LEPS forecast skill using either VERA or JDC obs for verification network.
Average precipitation in BOXES: controversial impact of increasing the box size on the verification scores.
1mm: similar performance of the system with respect to the 2 verification networks.
10 mm: higher skill when COSMO-LEPS is verified against VERA gridded analysis.
Almost no impact of the verification technique (ngp ~ bilin) for both thresholds.
A.Montani; The COSMO-LEPS system.

38. Verification networks and methodologies
COSMO-LEPS was verified against the following networks/methodologies
for all mesoVICT cases:
Networks/methodo- logies
Nearest grid point
Bilinear interpolation
Boxes:
0.5x0.5, 1.0x1.0, 1.5x1.5
VERA gridded analysis
done
done, done, done
JDC obs
A.Montani; The COSMO-LEPS system.