1. PP INSPECT report
Dmitry Alferov (1), Elena Astakhova (1), Petra Baumann (4), Dimitra Boukouvala (2), Anastasia Bundel (1), Ulrich Damrath (3), Pierre Eckert (4), Flora Gofa (2), Alexander Kirsanov (1), Xavier Lapillonne (4), Joanna Linkowska (5), Chiara Marsigli (6), Andrea Montani (6), Anatoly Muraviev (1), Elena Oberto (7), Maria Stefania Tesini (6), Naima Vela (7), Andrzej Wyszogrodzki (5), and Mikhail Zaichenko (1), André Walser (4)
(1) RHM (2) HNMS, (3) DWD, (4) MCH, (5) IMGW-PIB, (6) ARPA-SIMC, (7) ARPA-PT
COSMO GM, September 2017, Jerusalem, Israel

2. √ √ √ √ √ √ √ √ √ Spatial methods Many√ Filtering methods
Displacement methods
Many√
Neighborhood
(Ebert, 2008)
Scale Decomposition
DIST method
Features-based
Contiguous Rain Area (CRA)
(Ebert and McBride, 2000)
Method for Object-based
Diagnostic Evaluation (MODE)
(Davis et al., 2006)
 Structure, Amplitude, Location
(SAL) (Wernli et al., 2008)
Field deformation
√ √ √ √
√ √
√ √ √
Now almost all the categories of spatial methods are applied by INSPECT participants. Flora Gofa: “It gave us a chance to learn methods that before we just knew by name”.
Thus, the main benefit of INSPECT “that the wide range of spatial verification methods available will become commonly used within the COSMO community” is achieved.

3. Status highlights
The PP INSPECT is extended until the end of 2017 due to delays in some tasks, with 0.3 FTE shifted and 0.2 FTE added
Research tasks are mostly completed except for some tasks involving ensembles.
The reports describing the properties of each spatial method are written by participants. The report summarizing them is under preparation.

4. Reruns for MesoVICT test cases
MCH: COSMO-1 reruns for ALL MesoVICT cases are done and available at WG5 repository (Petra Baumann)
ECMWF-IFS reruns (51 member) for cases 1 and 2 (8 initial dates) (Andrea Montani),
COSMO-E reruns (21 member) for cases 1 and 2 (8 initial dates) (André Walser),
COSMO-Ru2-EPS (51 member) for case 1 (1 initial date) and case 2 (is running now) (Dmitry Alferov)
All reruns are interpolated to VERA grid by Manfred Dorninger (Austria) -> easy to use and compare!

5. Tasks involving development of routines for neighborhood, CRA, SAL, and MODE
For the most part, the software is based on free R SpatialVx package (developed by E. Gilleland). For SAL (D.Boucoucala) and Neighborhood (J.Linkowska) comparisons are made with alternative packages -> bug fixing of SpatialVx
VAST development by N. Vela (ARPA-PT): inclusion of time dimension and variables besides pecipitation, namely, cloud cover and wind speed

6. Neighborhood applications. Analysis of long time series (DWD, MCH)
Relax the requirement for an exact match by evaluating forecasts in the local neighborhood of the observations

7. Uli Damrath’s method to aggregate the scores
Getting Equitable Threat Score (ETS) for upscaling and Fractions skill score (FSS) as monthly values from fuzzy verification
No averaging over daily values but calculation of scores from the contingency tables of the whole month
Calculation of running means of the results over one year
Presentation of mean values and mean averages

8. Comparison of COSMO-EU to COSMO-DE – upscaling Equitable Threat Score (ETS)
Threshold W i n d o w s I z e
1.625
0.825
0.425
0.225
0.125
0.075
0.025
Mesh width of
COSMO-EU
Precipitation amount

9. Compact visualization of total precipitation FSS: to focus on the useful scale for a given lead-time and threshold (MCH)
FSS: Fractions Skill Score
P is the event fraction in the neighborhood.
Threshold
Lead time
0.1 mm/h
1mm/h
2 mm/h
01-12
2.2 km
19.8 km
59.4 km
13-24
33.0 km
99.0 km

10. Intensity-scale: filtering method (F. Gofa, HNMS)
Error image is expressed as the sum of components on different spatial scales
by performing a 2-dimentional discrete Haar wavelet decomposition.
The spatial scale refers to the spatial scale of the error and not that of the
precipitation features as it happens in the neighborhood methods.
MesoVICT case 1: , Intensity Scale Skill
COSMO2
COSMO1
Small scales have skill close to zero, while large scales exhibit large skill.
COSMO1 ISS graphs exhibit that errors due to displacements of small
spatial scale features are more important compared to those of COSMO-2

11. Case study 2007.09.25.06, 6h precipitation, threshold>=5mm
Object-based methods by IMGW-PIB
MODE, CRA, SAL
Case study , 6h precipitation, threshold>=5mm
VERA
COSMO 2
unmatched object (false)
Selected feature pairings based on total interest
obs feature mod feature total interest

12. SAL: Maria Stefania Tesini and Daniele D'Alessandro
A single parameter to evaluate the structure, amplitude, or location error in forecast is not enough when the precipitation field complexity is too high
Precipitation intensity is overestimated in Germany and underestimated in France, but the amplitude A is close to zero

13. Object matching for EPS, MesoVICT case 1 (A.Bundel)
Probability of each observed object is found and
the ensemble skill can be estimated using the BSS, for example
COSMO-E ensemble,
first 6 of 21 members,
precip threshold >0.5 mm/1h
Probabilities of each of 5 observed
objects:
1/21 20/21 10/21 19/21 14/21
Difficulty of such an approach: we have estimate only probabilities of objects
CRA errors In spatial shift, precip volume, and fine-scale pattern can be

14. SAL for EPS: ways to display large amount of ensemble data (Dimitra Boucouvala)
20/6
20/6 14

15. Probability threshold =1
SAL for EPS: introducing observation uncertainty (Dimitra Boucouvala)
Objects comparison for probability of precipitation >= 2mm
Probability threshold =1
Observations
LEPS
Preci >= 2 mm
Preci >= 2 mm for all 16 members
3 h Precipitation 21/6 12 UTC
S=1, A=0.38, L=0.3

16. Andrea Montani: Sensitivity of COSMO-LEPS forecast skill to the verification network: application to MesoVICT cases

17. New ensemble precipitation observation product by MeteoSwiss
Available for the past data (e.g. for Mesovict cases)
Available for Swiss + whole alpine domain for daily accumulation

18. Main results
Several ways of compact visualization of long time series of neighborhood scores are proposed (DWD, MCH)
The object-based SAL (Structure-Amplitude-Location) method is easier to implement as it doesn’t require pair-wise matching of observed and forecast objects. However, it can give misleading results as it estimates characteristics in the whole domain, e.g. if precipitation intensity is overestimated in one place and underestimated in another, the amplitude value will be close to zero.
The object-based MODE and CRA methods provide more information as the matched pairs of observed and forecast objects are compared. However, it’s difficult to find a best universal matching function, in particular for high-resolution fields with objects of complex shape.
The DIST, SAL and CRA methods are applied to ensembles (ARPAE-SIMC, HNMS, RHM)
The experiments are started on introducing observation uncertainty into the spatial methods (ARPAE-SIMC, HNMS, MCH)
The main benefit of INSPECT is achieved: the most wide-spread spatial methods enter into the everyday practice within the COSMO Community

19. Thank you for your attention !