Performance assessment of a Bayesian Forecasting System (BFS) for realtime flood forecasting Biondi D. , De Luca D.L. Laboratory of Cartography and Hydrogeological Modeling, University of Calabria (Italy)

GOALS Properties of BFS under different hypothesis Adeguate verification tools for probabilistic forecasts → robust analysis of forecasting capability Interaction among different sources of error

Properties of BFS under different hypothesis GOALS Properties of BFS under different hypothesis Adeguate verification tools for probabilistic forecasts → robust analysis of forecasting capability Interaction among different sources of error Few Studies Stochastic model for rainfall prediction Effect of more intense rainfall events Journal of Hydrology , Volume 479, 4 February 2013, Pages 51-63

BFS architecture Deterministic Hydrological Model Other input Precipitation Uncertainty Processor (PUP) Hydrologic Uncertainty Processor (HUP) Probabilistic Quantitative Precipitation Forecast (PQPF) Prior Integrator INT Some refs: Krzysztofowicz,1999 Krzysztofowicz & Kelly, 2000 Kelly & Krzysztofowicz, 2000 Krzysztofowicz & Herr, 2001 Krzysztofowicz, 2001 Krzysztofowicz, 2002 Krzysztofowicz & Maranzano, 2004 Maranzano & Krzysztofowicz, 2004

Deterministic Hydrological Model Precipitation Uncertainty Processor (PUP) PQPF –PRAISE stochastic model (Prediction of Rainfall Amount Inside a Storm Event, Sirangelo et al., 2007) W total precipitation amount on T M simulations for W Sn evaluation of Yn at tn s1 s2 s3 t0 t1 t2 t3 T=1 hour Perfect PQPF PRAISE Deterministic Hydrological Model (RISE) PUP Integrator INT Other input a 1-a sn0 Sn

Deterministic Hydrological Model Hydrologic Uncertainty Processor (HUP) Deterministic Hydrological Model HUP Integrator INT other input Prior Off-line estimation Observed rainfall data (perfect input) T t1 t2 t3 Likelihood Prior y0 y1 ? y2 ? Posterior y3 ? s1 s2 s3 t1 t2 t3 Precipitation dependent processor (Krzysztofowicz & Herr, 2001) t0 W=0  V=0 n=1 n=2 n=3 W>0  V=1

Deterministic Hydrological Model INTEGRaTORE (int) Deterministic Hydrological Model (RISE) Altri input PQPF (PRAISE) PUP HUP Prior Integrator INT

Case study and data description Area 29.2 km2 DTM resolution 30 m Max Altitude 1015 m Min Altitude 75 m Mean Altitude 291 m Selected flood events ID Date Qmax (m3/s) 1 28/12/2000 17.75 2 24/12/2000 14.32 3 01/01/2003 19.57 4 07/01/2003 16.00 5 28/01/2004 28.78 6 28/02/2004 18.35 7 14/02/2005 21.25 8 26/02/2005 20.40 9 06/03/2005 18.55 November – March Precipitation forecast with Dt = 20 min Precipitation forecasting period T = 1 hour V = 0 ↔ 0 < W < 2 mm and V = 1 ↔ W ≥ 2 mm probabilistic forecasts of river discharge in 1-h steps Database Rainfall heights 1991 - 2005 River discharges 2000 – 2005 Dt = 20 min

Hypotheses about BFS a) Perfect hydrological model P n (.) is provided by PUP, and estimated on the basisi on observed discharge y0, and probability a e u, evaluated on-line. (UD) (4) b) Non-informative PQPF for discharge forecasting prior distribution Gn(·|y0) (PD), with climati prior probability u’ c) Perfect PQPF: The BFS provides a predictive distribution equal to the posterior distribution (HD) , provided by HUP. d) Total predictive distributon (TD), which takes into account both sources of uncertainty;

Evaluation tools for forecast verification (Murphy, 1993) Calibration: probabilistic correctness of the forecasts or the statistical consistency between the observed time series and its predictive distribution (Reliability) Sharpness: the spread of the predictive distribution of the forecasts (attribute only for forecasts) Accuracy: Agreement between an observed value and a representative index, assumed as the best deterministic prediction (expected value, median, etc.) CRPS: Continuous Ranked Probability Score (Brown, 1974;Hersbach, 2000) y H

Probabilistic calibration Negative Bias Positive Bias Empirical cumulative distribution Under dispersion Over dispersion zi=Y(yobs,i) zi=Y(yobs,i) should be i.i.d. U[0,1] Laio & Tamea, 2007; Renard et al., 2010

Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore Results – Calibration 5 % Kolmogorov bands n = 1 hour n = 2 hours Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore n   UD PD HD TD 1 hour 8.69 1.60 1.33 1.22 0.50 0.72 0.74 0.84 0.46 1.00 1.00  2 hours 4.31 -1.24 -0.21 0.25 0.61 0.62 0.83 0.47 0.97 - UD: significant positive bias (many observed values with a predictive probability very small or null) a very narrow distribution - PD: a marked underestimation associated to forecasted values - HD: an overestimation for forecasts (due to the adopted hydrological model) - TD: a slight overestimation associated to predicted values

Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore Results - Sharpness Box-plot for InterQuartile Range (IQR) 75 % 25 % Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore n = 1 hour n = 2 hours TD provides the worst results, with respect to the other BFS hypotheses, and median value of IQR is the highest. HD is characterized by an higher sharpness (only Hydrological Uncertainty is considered) compared with TD UD provides the narrowest bands. For n = 1 hour, PD has a median IQR similar to UD but the bands are wider. For n = 2 hours UD has a median IQR less than n=1 hour, unlike other BFS hypotheses. This is due to the presence of sn0, which implies truncated probability distributions with generally small values of CV.

Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore Results - Sharpness . Event #2 uncertainty bands for n=1 hour UD PD Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore HD TD (Biondi & De Luca, 2011)

Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore Results - Accuracy n = 1 hour UD PD 18% 40% RMSE=7.7 RMSE=6.2 Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore HD TD 44% 51% RMSE=5.2 RMSE=5.4

Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore Results - Accuracy n = 2 hours UD PD 11% 36% RMSE=7.4 RMSE=7.1 Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore HD TD Sostituire figure con n=2 36% 38% RMSE=6.1 RMSE=7.9

Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore Results - CRPS n UD PD HD TD 1 ora 5.27 3.43 2.63 2.82 2 ore 5.00 4.48 3.54 4.29 F(y) Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore n=2 n=1 Sn0 Sn0 yobs y

Conclusioni CONCLUSIONS Calibration TD: best results PD – HD: negative and positive bias, respectively UD: marked overprediction (due to sn0) Sharpness UD: the best results (due to sn0) TD: the worst results Accuracy and CRPS TD and HD: the best results (TD is the best for a real time prediction) Crucial role of HUP Similar results beyond T Importance of performing a comprehensive analysis by using different validation tools (in order to avoid misleading conclusions)

Thank you for your attention Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore