1. Performance assessment of a Bayesian Forecasting System (BFS) for realtime flood forecasting
Biondi D. , De Luca D.L.
Laboratory of Cartography and Hydrogeological Modeling, University of Calabria (Italy)

2. GOALS
Properties of BFS under different hypothesis
Adeguate verification tools for probabilistic forecasts → robust analysis of forecasting capability
Interaction among different sources of error

3. Properties of BFS under different hypothesis
GOALS
Properties of BFS under different hypothesis
Adeguate verification tools for probabilistic forecasts → robust analysis of forecasting capability
Interaction among different sources of error
Few Studies
Stochastic model for rainfall prediction
Effect of more intense rainfall events
Journal of Hydrology , Volume 479, 4 February 2013, Pages 51-63

4. BFS architecture Deterministic Hydrological Model Other input
Precipitation Uncertainty Processor (PUP)
Hydrologic Uncertainty Processor (HUP)
Probabilistic Quantitative Precipitation Forecast (PQPF)
Prior
Integrator
INT
Some refs:
Krzysztofowicz,1999
Krzysztofowicz & Kelly, 2000
Kelly & Krzysztofowicz, 2000
Krzysztofowicz & Herr, 2001
Krzysztofowicz, 2001
Krzysztofowicz, 2002
Krzysztofowicz & Maranzano, 2004
Maranzano & Krzysztofowicz, 2004

5. Deterministic Hydrological Model
Precipitation Uncertainty Processor (PUP)
PQPF –PRAISE stochastic model (Prediction of Rainfall Amount Inside a Storm Event, Sirangelo et al., 2007)
W total precipitation amount on T
M simulations for W
Sn evaluation of Yn at tn s1 s2 s3 t0 t1 t2 t3
T=1 hour
Perfect
PQPF
PRAISE
Deterministic Hydrological Model
(RISE)
PUP
Integrator
INT
Other
input a
1-a
sn0 Sn

6. Deterministic Hydrological Model
Hydrologic Uncertainty Processor (HUP)
Deterministic Hydrological Model
HUP
Integrator
INT
other
input
Prior
Off-line estimation
Observed rainfall data (perfect input) T t1 t2 t3
Likelihood
Prior y0
y1 ?
y2 ?
Posterior
y3 ? s1 s2 s3 t1 t2 t3
Precipitation dependent processor (Krzysztofowicz & Herr, 2001) t0
W=0  V=0
n=1
n=2
n=3
W>0  V=1

7. Deterministic Hydrological Model
INTEGRaTORE (int)
Deterministic Hydrological Model
(RISE)
Altri
input
PQPF
(PRAISE)
PUP
HUP
Prior
Integrator
INT

8. Case study and data description
Area
29.2 km2
DTM resolution
30 m
Max Altitude
1015 m
Min Altitude
75 m
Mean Altitude
291 m
Selected flood events ID
Date
Qmax (m3/s) 1
28/12/2000
17.75 2
24/12/2000
14.32 3
01/01/2003
19.57 4
07/01/2003
16.00 5
28/01/2004
28.78 6
28/02/2004
18.35 7
14/02/2005
21.25 8
26/02/2005
20.40 9
06/03/2005
18.55
November – March
Precipitation forecast with Dt = 20 min
Precipitation forecasting period T = 1 hour
V = 0 ↔ 0 < W < 2 mm and V = 1 ↔ W ≥ 2 mm
probabilistic forecasts of river discharge in 1-h steps
Database
Rainfall heights
River discharges – 2005
Dt = 20 min

9. Hypotheses about BFS
a) Perfect hydrological model P n (.) is provided by PUP, and estimated on the basisi on observed discharge y0, and probability a e u, evaluated on-line. (UD)
(4)
b) Non-informative PQPF for discharge forecasting prior distribution Gn(·|y0) (PD), with climati prior probability u’
c) Perfect PQPF: The BFS provides a predictive distribution equal to the posterior distribution (HD) , provided by HUP.
d) Total predictive distributon (TD), which takes into account both sources of uncertainty;

10. Evaluation tools for forecast verification (Murphy, 1993)
Calibration: probabilistic correctness of the forecasts or the statistical consistency between the observed time series and its predictive distribution (Reliability)
Sharpness: the spread of the predictive distribution of the forecasts (attribute only for forecasts)
Accuracy: Agreement between an observed value and a representative index, assumed as the best deterministic prediction (expected value, median, etc.)
CRPS: Continuous Ranked Probability Score (Brown, 1974;Hersbach, 2000) y H

11. Probabilistic calibration
Negative Bias
Positive Bias
Empirical cumulative distribution
Under dispersion
Over dispersion
zi=Y(yobs,i)
zi=Y(yobs,i) should be i.i.d. U[0,1]
Laio & Tamea, 2007; Renard et al., 2010

12. Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
Results – Calibration
5 % Kolmogorov bands
n = 1 hour
n = 2 hours
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore n UD PD HD TD
1 hour
8.69
1.60
1.33
1.22
0.50
0.72
0.74
0.84
0.46
1.00
1.00 
2 hours
4.31
-1.24
-0.21
0.25
0.61
0.62
0.83
0.47
0.97
- UD: significant positive bias (many observed values with a predictive probability very small or null) a very narrow distribution
- PD: a marked underestimation associated to forecasted values
- HD: an overestimation for forecasts (due to the adopted hydrological model)
- TD: a slight overestimation associated to predicted values

13. Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
Results - Sharpness
Box-plot for InterQuartile Range (IQR)
75 %
25 %
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
n = 1 hour
n = 2 hours
TD provides the worst results, with respect to the other BFS hypotheses, and median value of IQR is the highest.
HD is characterized by an higher sharpness (only Hydrological Uncertainty is considered) compared with TD
UD provides the narrowest bands.
For n = 1 hour, PD has a median IQR similar to UD but the bands are wider.
For n = 2 hours UD has a median IQR less than n=1 hour, unlike other BFS hypotheses. This is due to the presence of sn0, which implies truncated probability distributions with generally small values of CV.

14. Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
Results - Sharpness .
Event #2 uncertainty bands for n=1 hour UD PD
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore HD TD
(Biondi & De Luca, 2011)

15. Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
Results - Accuracy
n = 1 hour UD PD
18%
40%
RMSE=7.7
RMSE=6.2
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore HD TD
44%
51%
RMSE=5.2
RMSE=5.4

16. Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
Results - Accuracy
n = 2 hours UD PD
11%
36%
RMSE=7.4
RMSE=7.1
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore HD TD
Sostituire figure con n=2
36%
38%
RMSE=6.1
RMSE=7.9

17. Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
Results - CRPS n UD PD HD TD
1 ora
5.27
3.43
2.63
2.82
2 ore
5.00
4.48
3.54
4.29
F(y)
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
n=2
n=1
Sn0
Sn0
yobs y

18. Conclusioni CONCLUSIONS Calibration TD: best results
PD – HD: negative and positive bias, respectively
UD: marked overprediction (due to sn0)
Sharpness
UD: the best results (due to sn0)
TD: the worst results
Accuracy and CRPS
TD and HD: the best results (TD is the best for a real time prediction)
Crucial role of HUP
Similar results beyond T
Importance of performing a comprehensive analysis by using different validation tools (in order to avoid misleading conclusions)

19. Thank you for your attention
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore