PP INSPECT report Dmitry Alferov (1), Elena Astakhova (1), Petra Baumann (4), Dimitra Boukouvala (2), Anastasia Bundel (1), Ulrich Damrath (3), Pierre Eckert (4), Flora Gofa (2), Alexander Kirsanov (1), Xavier Lapillonne (4), Joanna Linkowska (5), Chiara Marsigli (6), Andrea Montani (6), Anatoly Muraviev (1), Elena Oberto (7), Maria Stefania Tesini (6), Naima Vela (7), Andrzej Wyszogrodzki (5), and Mikhail Zaichenko (1), André Walser (4) (1) RHM (a.bundel@gmail.com), (2) HNMS, (3) DWD, (4) MCH, (5) IMGW-PIB, (6) ARPA-SIMC, (7) ARPA-PT COSMO GM, 11-15 September 2017, Jerusalem, Israel

√ √ √ √ √ √ √ √ √ Spatial methods Many√ Filtering methods Displacement methods Many√ Neighborhood (Ebert, 2008) Scale Decomposition DIST method Features-based Contiguous Rain Area (CRA) (Ebert and McBride, 2000) Method for Object-based Diagnostic Evaluation (MODE) (Davis et al., 2006)  Structure, Amplitude, Location (SAL) (Wernli et al., 2008) Field deformation √ √ √ √ √ √ √ √ √ Now almost all the categories of spatial methods are applied by INSPECT participants. Flora Gofa: “It gave us a chance to learn methods that before we just knew by name”. Thus, the main benefit of INSPECT “that the wide range of spatial verification methods available will become commonly used within the COSMO community” is achieved.

Status highlights The PP INSPECT is extended until the end of 2017 due to delays in some tasks, with 0.3 FTE shifted and 0.2 FTE added Research tasks are mostly completed except for some tasks involving ensembles. The reports describing the properties of each spatial method are written by participants. The report summarizing them is under preparation.

Reruns for MesoVICT test cases MCH: COSMO-1 reruns for ALL MesoVICT cases are done and available at WG5 repository (Petra Baumann) ECMWF-IFS reruns (51 member) for cases 1 and 2 (8 initial dates) (Andrea Montani), COSMO-E reruns (21 member) for cases 1 and 2 (8 initial dates) (André Walser), COSMO-Ru2-EPS (51 member) for case 1 (1 initial date) and case 2 (is running now) (Dmitry Alferov) All reruns are interpolated to VERA grid by Manfred Dorninger (Austria) -> easy to use and compare!

Tasks involving development of routines for neighborhood, CRA, SAL, and MODE For the most part, the software is based on free R SpatialVx package (developed by E. Gilleland). For SAL (D.Boucoucala) and Neighborhood (J.Linkowska) comparisons are made with alternative packages -> bug fixing of SpatialVx VAST development by N. Vela (ARPA-PT): inclusion of time dimension and variables besides pecipitation, namely, cloud cover and wind speed

Neighborhood applications. Analysis of long time series (DWD, MCH) Relax the requirement for an exact match by evaluating forecasts in the local neighborhood of the observations

Uli Damrath’s method to aggregate the scores Getting Equitable Threat Score (ETS) for upscaling and Fractions skill score (FSS) as monthly values from fuzzy verification No averaging over daily values but calculation of scores from the contingency tables of the whole month Calculation of running means of the results over one year Presentation of mean values and mean averages

Comparison of COSMO-EU to COSMO-DE – upscaling Equitable Threat Score (ETS) Threshold 0.1 0.2 0.5 1 2 5 10 20 50 W i n d o w s I z e 1.625 0.825 0.425 0.225 0.125 0.075 0.025 Mesh width of COSMO-EU Precipitation amount

Compact visualization of total precipitation FSS: to focus on the useful scale for a given lead-time and threshold (MCH) FSS: Fractions Skill Score P is the event fraction in the neighborhood. Threshold Lead time 0.1 mm/h 1mm/h 2 mm/h 01-12 2.2 km 19.8 km 59.4 km 13-24 33.0 km 99.0 km

Intensity-scale: filtering method (F. Gofa, HNMS) Error image is expressed as the sum of components on different spatial scales by performing a 2-dimentional discrete Haar wavelet decomposition. The spatial scale refers to the spatial scale of the error and not that of the precipitation features as it happens in the neighborhood methods. MesoVICT case 1: 20070621-15, Intensity Scale Skill COSMO2 COSMO1 Small scales have skill close to zero, while large scales exhibit large skill. COSMO1 ISS graphs exhibit that errors due to displacements of small spatial scale features are more important compared to those of COSMO-2

Case study 2007.09.25.06, 6h precipitation, threshold>=5mm Object-based methods by IMGW-PIB MODE, CRA, SAL Case study 2007.09.25.06, 6h precipitation, threshold>=5mm VERA COSMO 2 unmatched object (false) Selected feature pairings based on total interest obs feature mod feature total interest 1 1 0.898

SAL: Maria Stefania Tesini and Daniele D'Alessandro A single parameter to evaluate the structure, amplitude, or location error in forecast is not enough when the precipitation field complexity is too high Precipitation intensity is overestimated in Germany and underestimated in France, but the amplitude A is close to zero

Object matching for EPS, MesoVICT case 1 (A.Bundel) Probability of each observed object is found and the ensemble skill can be estimated using the BSS, for example 2007062021 COSMO-E ensemble, first 6 of 21 members, precip threshold >0.5 mm/1h Probabilities of each of 5 observed objects: 1/21 20/21 10/21 19/21 14/21 Difficulty of such an approach: we have estimate only probabilities of objects CRA errors In spatial shift, precip volume, and fine-scale pattern can be

SAL for EPS: ways to display large amount of ensemble data (Dimitra Boucouvala) 20/6 20/6 14

Probability threshold =1 SAL for EPS: introducing observation uncertainty (Dimitra Boucouvala) Objects comparison for probability of precipitation >= 2mm Probability threshold =1 Observations LEPS Preci >= 2 mm Preci >= 2 mm for all 16 members 3 h Precipitation 21/6 12 UTC S=1, A=0.38, L=0.3

Andrea Montani: Sensitivity of COSMO-LEPS forecast skill to the verification network: application to MesoVICT cases

New ensemble precipitation observation product by MeteoSwiss Available for the past data (e.g. for Mesovict cases) Available for Swiss + whole alpine domain for daily accumulation

Main results Several ways of compact visualization of long time series of neighborhood scores are proposed (DWD, MCH) The object-based SAL (Structure-Amplitude-Location) method is easier to implement as it doesn’t require pair-wise matching of observed and forecast objects. However, it can give misleading results as it estimates characteristics in the whole domain, e.g. if precipitation intensity is overestimated in one place and underestimated in another, the amplitude value will be close to zero. The object-based MODE and CRA methods provide more information as the matched pairs of observed and forecast objects are compared. However, it’s difficult to find a best universal matching function, in particular for high-resolution fields with objects of complex shape. The DIST, SAL and CRA methods are applied to ensembles (ARPAE-SIMC, HNMS, RHM) The experiments are started on introducing observation uncertainty into the spatial methods (ARPAE-SIMC, HNMS, MCH) The main benefit of INSPECT is achieved: the most wide-spread spatial methods enter into the everyday practice within the COSMO Community

Thank you for your attention !