1. Assessment of the statistical properties of COSMO-I7 QPF as a methodology to evaluate its predictable spatial scales and optimize the operational use for weather forecasting Carlo Cacciamani, Maria Stefania Tesini Federico Grazzini (many thanks to Chiara Marsigli) 9th COSMO General Meeting - WG5 parallel session, Athens 18-21 Sep. 2007

2. Open questions: How can we use HR-LAM QPF ? Does COSMO-LM-7 (and 2.8) furnish a real added value (with respect to coarser GCMs or LAMs) to the forecasters (example: for the emission of alerts in case of intense precipitation occurrence) ?

3. QPF of HR LAMs: Problems we know… Precipitation has a large space-time variability. HR-LAMs have problems to reproduce mesoscale details (Zepeda Arce et all., 2000). In detail: HR Lams are able to forecast QPF maxima but may be wrong in space-time localisation (Theis et all, 2005; Ebert & Mc Bride, 2000; Bernadet et all., 2000). A HR Lams gives much more realistic simulation of QPF than a GCM, but may have the same a low quality (double penalty). On the contrary a GCM may have a better quality but be completely un- realistic

4. Some examples

5. COSMO LM-I28 6h forecast Observed rain rate from SPC radar REALISTIC performance but error in time and space precipitation localisation !

6. The added value of limited area models patterns are more realistic…….. Totally non-realistic but better score ! Realistic but bad QPF scores !

7. Very high CAPE ~3000 J/Kg Unrealistic updraft, 4/5 grid points with w at 3500 m (model level 24) greater than 2 m/s Example of big QPF errors ! A case of upscaling of convection

8. Problem: TIME INCONSISTENCY and error in amplitude ! Precipitation in one point: PORRETTA (Emilia-Romagna Appennine) x Large time shift !

9. Very (very !) often, when QPF is large, the difference between QPF of the nearest grid points to a selected observation is larger than the QPF (or the obs.prec.) itself ! Problem: Spatial INCONSISTENCY

10. Below a defined spatial scale (what ?), linked to the LAM resolution, QPF cannot be reproduced in a deterministic way. How can we manage QPF (and HR QPF) ? In order to increase details it is necessary to increase LAM resolution... Alternative/pragmatic approach: forget the “dream” to forecast QPF details but use a “fuzzy-probabilistic approach” (i.e. use of quantities inferred by a “pdf” of QPF: median, time-space averages, maxima, percentiles, probability of occurrence of large amplitude events...); What can we learn from all that ? From Athens COSMO meeting in 2002

11. In box of different size (what is the best size ?) alert warning areas (Emilia-Romagna) First simple approach: averaging QPF

12. TS increases making spatial averaging ! x

13. Benefit of precipitation aggregation: use of BOX LAMI 7Km grid-points distribution over verification boxes of 0.5°x0.5° observation distribution over verification boxes of 0.5°x0.5°

14. Contingency tables and scores

15. Verification of 24h accumulated precipitation aggregated over boxes of 0.5°x0.5°. Sensitivity to precip. threshold and variable: QPF averaged value vs QPF maximum value 20 mm/24h 5mm/24h N° obs~ 950 N° obs~ 300 Mean value in the box N° obs~ 1300 N° obs~ 550 Maximum value in the box 5mm/24h 20 mm/24h The gain of HRLam with respect to GCMs is greater for high thresholds and for precipitation maxima Mean value in the boxMaximum value in the box

16. Soglia 10Soglia 20 N° obs~ 750N° obs~ 380 Soglia 10Soglia 20 N° obs~ 400 N° obs~ 150 Much better results increasing time averaging ! Verification of 6h and 24h cumulated precipitation aggregated over boxes of 0.4°x0.4° (sensitivity to time aggregation)

17. Sensitivity to box size and precipitation threshold Positive impact of larger box is more visible at higher precipitation thresholds

18. Sensitivity to box size and precipitation threshold Positive impact of larger box is more visible at higher precipitation thresholds

19. Sensitivity to box size and precipitation threshold Positive impact of larger box is more visible at higher precipitation thresholds

20. Best result box = 0.5 deg ? (7 * 7 grid points …) Sensitivity to box size and precipitation threshold

21. Best result box = 0.5 deg ? (7 * 7 grid points …)

22. Some preliminary conclusions QPF spatial averaging over box or alert areas produces a more usable QPF field for applications. Space-time localisation errors are minimised Box or alert areas with size of 5-6 times the grid resolution gives the best results Positive impact of larger box is more visible at higher precipitation thresholds The gain of HRLam with respect to GCMs is greater for high thresholds and for precipitation maxima Better results increasing time averaging (problems with 6 hours accumulation period, much better with 24 hours cumulated period !

23. Application to the entire italian territory Study of the QPF pdf

24. Motivation The predicted state of interest has a substantial amount of intrinsic uncertainty, depending on forecast lead time, spatial scale, terrain,flow and forcing The relation between spatial scale and prediction skill has became evident in a moltitude of theoretical and experimental studies The output of a LAM must not be regarded as purely deterministic, but rather as an outcome of a random experiment that behaves according to a given probability density function (pdf) (Theis et al,2005)

25. Study of QPF pdf and its “moments” within the box Aim: To investigate the spatial characteristics of precipitations (observed and predicted) within the box To investigate the spatial characteristics of precipitations (observed and predicted) within the box To use pdf agreement between observed and prediced precipitation to judge the quality and the realism of HRLam (COSMO-LMI7) To use pdf agreement between observed and prediced precipitation to judge the quality and the realism of HRLam (COSMO-LMI7) Test Period: 2005/09/01-2005/11/30 It will be repeated in other period s(work in progress) 00-24 cumulated precipitation RUN 00 UTC COSMO-LAMI (+24h) [~7 km hor. res] RUN 00 UTC COSMO-LAMI (+24h) [~7 km hor. res] RUN 12 UTC ECMWF-IFS (+12h /+36h) [~50 km hor.res] RUN 12 UTC ECMWF-IFS (+12h /+36h) [~50 km hor.res]

26. DATASET E ~ 800 stations in total obtained by the regional network of raingauges centralised at the National Department of Civil Protection (DPCN) [20 – 100] stations for each box of (dimension 1.0°x1.0°)

27. The choice of the box 1.The maximum station points as possible in the area 2.Geographical and orographic homogeneity 3.Possibility to divide non homogeneous boxes in homogeneous sub-boxes

28. …about models gridpoints ~ 50 km horizontal resolution 4 points in a box (max 9!) ~ 7 km horizontal resolution 170-220 points in a box COSMO –LAMIECMWF

29. Selected Box

30. COSMO LAMIECMWFSTATIONS BOXTOTALPLAINHILLMOUNTAINTOTALPLAINHILLMOUNTAINTOTALPLAINHILLMOUNTAIN A_01_011927591264220100235126 F_01_0121210896843107413565 K_01_0120251682940406384312 D_01_0117679574062316132632 G_01_01208120344040471433 M_01_0118285405741124411258 L_01_012201207525413044112310 J_01_0121690804642204091912 C_01_0119212857763303681513 R_01_012167686544031325189 P_01_011710291424013260917 Q_01_011922116294130242193 T_01_012026513524130244200 U_01_0121249120434022246126 V_01_012066598439333234118 H_01_0120811592143102311120 S_01_011811193428422020965 W_01_011983491464031201118 X_01_012103015624403115573 Y_01_01210758451642014950 B_01_011911838066006420 N_01_0120392981341306042 I_01_012066179214040466328

31. A: Comparison of observed and predicted precipitation pdf Use of a graphical approach to compare observed and forecast distribution: Boxplot Boxplot Boxplot Quantile-Quantile plot Quantile-Quantile plot Quantile-Quantile plot Quantile-Quantile plot

32. Box A North Tyrrhenian area COSMO-LM I7 pdf is realistic, even in the tail of the distribution ECMWF pdf doesn’t reproduce high values. Over-enstimation at low values

33. Box C Central Po Valley Area COSMO-LM I7 underestimates pdf. Good reproduction of the maximum percentile

34. Box H Central Tyrrhenian area Realistic COSMO LM I7 pdf even for the tail ECMWF does not reproduce the tail

35. Box V Central Adriatic area COSMO-LAMI under- estimates precipitation as ECMWF

36. What happen for smaller box ?

37. Box A OBSVCOSMO-I7 ECMW F A1141924

38. Box A OBSV COSMO- I7 A1748 A24948 A32148 A42654 1 2 3 4

39. Box A OB SV COSMO -I7 A1020 A21220 A32724 A4420 A51520 A62224 A7620 A8821 A91323 1 2 3 47

40. Box A 1 2 3 4 5913OBSVCOSMO-I7A1012 A2312 A3512 A41812 A5012 A6812 A71512 A82012

41. Box A 1 2 3 4 5913OBSVCOSMO-I7A9312 10812 A11212 A121212 A13412 A14812 A15112 A16712

42. Box A Quantile-quantile plot

43. Box A

46. Climatological behaviour” COSMO-LM-I7 pdf is very often realistic High values (right tail of the pdf) are statistically reproduced. That is not the case for ECMWF model. Results depends on the geographical positioning of area Strong dependence of the results of pdf intercomparison on the size of the box

47. Boxplot time series Boxplot time series Quantile time series Quantile time series Scatterplot of the forecast error versus observation Scatterplot of the forecast error versus observation Mean Error and Mean Absolute Error Mean Error and Mean Absolute Error Day by day behaviour of statistical moments deduced by observed and predicted precipitation Aim: Investigate the day-by-day reproduction of the observed pdf and of some indices (IQR, 90th percentile, etc..)

48. Box A North Tyrrhenian area COSMO-LM-I7 reproduces realistically the day-by- day spread. Sometime big errors are evident. ECMWF always underestimates the spread

49. Box C Central Po Valley Area

50. Box H Central Tyrrhenian area

51. Box V Central Adriatic area

52. Box A North Tyrrhenian area

53. Box C Central Po Valley Area

54. Box H Central Tyrrhenian area

55. Box V Central Adriatic area

56. Day-by-day behauvior Results are different in the different areas Common elements: observed spread” is well reproduced by COSMO-LM-I7 even if false and miss cases are evident ECMWF is not able to simulate the observed variability, i.e. is less realistic than COSMO-LM-I7

57. Threat Score. Variable used: mean value of the pdf

58. Threat Score. Variable used: 90th percentile of the pdf Strong dependence on geographical positioning. When 90th percentile is over large precipitation values COSMO-LM-I7 is much better than ECMWF

59. Threat Score. Variable used: 90th percentile of the pdf Strong dependence on geographical positioning. When 90th percentile is over large precipitation values COSMO-LM-I7 is much better than ECMWF

60. Scores: conclusive considerations As regard mean precipitation inside the box, COSMO- LM-I7 has worse performance than ECMWF for low QPF thresholds; for larger thresholds COSMO-LM-I7 is better or similar to ECMWF Considering the tail of pdf (90th percentile) COSMO-LMI7 reproduces well strong precipitation events (more than 20 mm/24h) interesting a limited number of points (10% ) in the domain It is evident a strong dependence of the results on the geographical positioning of the areas, in strong correlation with the orography and the direction of the incident flow (for example the different behaviour between area A and C positioned upwind and downwind the Apennine chain when the mean flow is south westerly

61. Example of dependence on the geographical positioning – 90th percentile in the adriatic areas (areas V and W - black) COSMO-LM-I7 underestimates the precipitation In the tyrrhenian areas (areas H, Y and F - red) COSMO-LM-I7 overestimates precipitation

62. The end, thank you for the attention

63. Diapo in più

64. Boxplot The median for each dataset is indicated by the black center line the first and third quartiles are the edges of the yellow area, which is known as the inter-quartile range (IQR). The extreme values (within 1.5 times the IQR from the upper or lower quartile) are the ends of the lines extending from the IQR. Points at a greater distance from the median than 1.5 times the IQR are plotted individually as circle. These points represent the outliers. Points at a greater distance from the median than 1.5 times the IQR are plotted individually as circle. These points represent the outliers.

65. Quantile-Quantile Plot The quantile-quantile (q-q) plot is a graphical technique for determining if two data sets come from populations with a common distribution. A q-q plot is a plot of the quantiles of the first data set against the quantiles of the second data set. A q-q plot is a plot of the quantiles of the first data set against the quantiles of the second data set. By a quantile, we mean the fraction (or percent) of points below the given value. That is, the 0.3 (or 30%) quantile is the point at which 30% percent of the data fall below and 70% fall above that value. A 45-degree reference line is also plotted. If the two sets come from a population with the same distribution, the points should fall approximately along this reference line.