1. Addressing the environmental impact of salt use on roads
Addressing the environmental impact of salt use on roads 1 1

2.
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #2
Introduction
Winter
General considerations (RSTs, frost/ice conditions)
Reuter's model
Verification through contingency tables (some theory first... then results!) 2 2

3.
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #3
RST Summary – Winter
Number of days with:
RST > 0°C RST ≤ 0°C NA
Coldest > Rocchetta / Casa Cantoniera
Warmest > Cadino / Acquaviva
Coldest month 3 3

4. Ice/Frost Formation – Winter 2014-2015 Tdew point as provided + 1°C **
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #4
Ice/Frost Formation – Winter
Data: From 15th November 2014 to 15th April 2015
Applied conditions: FROST ↔ TRST ≤ Tdew point < 0
ICE ↔ TRST ≤ Tdew point & TRST ≤ 0 & Tdew point ≥ 0
* RST sensor properly installed on A22 flyover on December 23rd
** In order to compensate for measurements errors that might be introduced for high RH values
Tdew point as provided + 1°C ** 4 4

5. Forecast verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #5
Forecast verification
Verification is the process of comparing forecasts to relevant observations.
Verification is one aspect of measuring forecast goodness.
(quality of the forecast + the user and his/her application of the forecast information)
Contingency table: display format used to analyse and record the relationship between two or more categorical variables, in this case the forecast and the observation of a weather forecast element.
→ It is the categorical equivalent of the scatterplot used to analyse the relationship between two continuous variables... 5 5

6. What if data are not binary? (occurred – not occurred event?)
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #6
Contigency tables
2x2 contingency table (tornados)
Categorical scores
Hit Rate = a/(a+c)
False Alarm Ratio = b/(a+b)
… many more!
BUT
What if data are not binary?
(occurred – not occurred event?) 6 6

7. Contingency Table for Freezing Temperatures (i.e. RST ≤ 0°C)
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #7
Contigency tables
When your data are not binary you have to:
(1) define a threshold
(2) pick a threshold that is meaningful to your end-users
Contingency Table for Freezing Temperatures (i.e. RST ≤ 0°C) 7 7

8. PROBABILITY FORECASTING
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #8
Contigency tables
That's especially true for “Observed” categories
“Ice” when RST ≤ 0°C “No ice” when RST > 0°C
For “Forecast” categories we'd better be not so deterministic!
Overnight changes for RST decay trend
Reuter's model works best in certain conditions only
Our forecast is uncertain and this uncertainty must be taken into account!
PROBABILITY FORECASTING 8 8

9. Contigency tables How to construct: Bin your data
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #9
Contigency tables
How to construct:
Bin your data
Calculate PODY and POFD by moving through bins
Plot using scatter plot 9 9

10. Contigency tables How to construct: Bin your data
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #10
Contigency tables
How to construct:
Bin your data
Calculate scores by moving through bins (different threshold!) 10 10

11. Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #11
Forecast Verification
How to construct:
Bin your data
Calculate scores by moving through bins (different threshold!)
PLOT, PLOT, PLOT!
Using a set of increasing probability thresholds (for example, 0.05, 0.15, 0.25, etc.) to make yes/no decisions
Relative Operating
Characteristic (ROC) graph
What is the ability of the forecast to discriminate between events and non-events?
(conditioned on the observations) 11 11

12. Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #12
Forecast Verification
Attributes Diagram
Points below the line = overforecasting (probabilities too high)
Points above the line = underforecasting (probabilities too low)
Brier Score What is the magnitude of the probability forecast errors?
Brier Skill Score What is the relative skill of the probabilistic forecast over that of climatology, in terms of predicting whether or not an event occurred?
How well do the predicted probabilities of an event correspond to their observed frequencies?
(conditioned on the forecasts) 12 12

13. Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #13
Forecast Verification
Relative Value Diagram
Relative value (AKA value score) = a skill score of expected expense, with climatology as the reference forecast. Because the cost/loss ratio is different for different users of forecasts, the value is generally plotted as a function of C/L.
For a cost/loss ratio C/L for taking action based on a forecast, what is the relative improvement in economic value between climatological and perfect information? 13 13

14. Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #14
Forecast Verification
Relative Value Diagram
Like ROC, it gives information that can be used in decision making.
Compute relative value curves for the entire range of probabilities
Select the optimal values (the upper envelope of the relative value curves) to represent the value of the probabilistic forecast system 14 14

15. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #15
Reuter's Forecast Verification
Aim Verifying Reuter's “frost” forecast on nights when the minimum RST is 5°C o below Procedure The minimum RST for each winter night is noted at each forecast site and compared with the minimum RSTs forecast overnight by Reuter's model. Results are entered into a contingency table where forecast categories are probabilities!
Results ... 15 15

16. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #16
Reuter's Forecast Verification
Number of days with:
0 < RST ≤ 5°C RST ≤ 0°C 16 16

17. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #17
Reuter's Forecast Verification
Date
Measured RSTmin (at a given time t)
Forecast RST at time t
Results table for overnight Reuter's forecasts
2nd time point in Reuter's interpolation 17 17

18. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #18
Reuter's Forecast Verification
Date
Measured RSTmin (at a given time t)
Forecast RST at time t
Contingency table for every overnight Reuter's forecast
2nd time point in Reuter's interpolation
Overall contingency table
“Ice” as RST ≤ 0°C
This is the starting point from which you can build the verification plots described before 18 18

19. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #19
Reuter's Forecast Verification
CADINO
no skill
climatology
no skill
no resolution 1%
20%
80%
The forecast:
distinguishes well between events and non-events
has a good reliability (slightly underforecasting)
has small probability-associated errors
has a good predictability skill with respect to climatology
gives a 66% economic benefit if you use 20% for your threshold and C/L = 1/8 = (Thornes 2001)
Station climatological frequency =
0.31 19 19

20. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #20
Reuter's Forecast Verification
SAN MICHELE
climatology
no skill
no skill
no resolution 1%
20%
80%
The forecast:
distinguishes well between events and non-events
has a good reliability (slightly overforecasting)
has small probability-associated errors
has a good predictability skill with respect to climatology
gives a 31% economic benefit if you use 20% for your threshold and C/L = 1/8 = (Thornes 2001)
Station climatological frequency =
0.41 20 20

21. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #21
Reuter's Forecast Verification
ACQUAVIVA
no skill
climatology
no skill
no resolution 1%
20%
80%
The forecast:
distinguishes well between events and non-events
has a good reliability (slightly overforecasting)
has small probability-associated errors
has a good predictability skill with respect to climatology
gives a 32% economic benefit if you use 20% for your threshold and C/L = 1/8 = (Thornes 2001)
Station climatological frequency =
0.31 21 21

22. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #22
Reuter's Forecast Verification
CASA CANTONIERA
climatology
no skill
no skill
no resolution 1%
20%
80%
The forecast:
distinguishes well between events and non-events
has a good reliability (slightly underforecasting)
has small probability-associated errors
has a good predictability skill with respect to climatology
gives a 30% economic benefit if you use 1% for your threshold and C/L = 1/8 = (Thornes 2001)
Station climatological frequency =
0.48 22 22

23. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #23
Reuter's Forecast Verification
ROCCHETTA
climatology
no skill
no skill
no resolution
80% 1%
20%
The forecast:
distinguishes well between events and non-events
has a good reliability (slightly underforecasting)
has decreasing probability-associated errors
has an good predictability skill with respect to climatology
gives a 40% economic benefit if you use 20% for your threshold and C/L = 1/8 = (Thornes 2001)
Station climatological frequency =
0.50 23 23

24. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #24
Reuter's Forecast Verification
VIADOTTO
climatology
no skill
no skill
no resolution
80% 1%
20%
The forecast:
distinguishes well between events and non-events
has a good reliability (slightly underforecasting)
has decreasing probability-associated errors
has an increasing predictability skill with respect to climatology and with respect to interpolation times
gives a 10% economic benefit if you use 20% for your threshold and C/L = 1/8 = (Thornes 2001)
Station climatological frequency =
0.48 24 24

25. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #25
Reuter's Forecast Verification
Conclusions
Cadino – San Michele – Acquaviva
Same results (similar RST and probability thresholds)
Casa Cantoniera
Similar results to the C-SM-A group but higher economic benefits for lower probability thresholds (thus higher RST threshold)
Rocchetta – Viadotto
Different results from the C-SM-A group (different RST thresholds) ← Different climatology (e.g. RST threshold at 20% equal to 2°C and 3°C respectively → RST decreases overnight more rapidly) 25 25

26. Reuter's Forecast Verification
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #26
Reuter's Forecast Verification
Observations
Rocchetta*
Cadino
*Similarly for Viadotto!
Föhn on Jan 12th, 2015 26 26

27. Deliverables As far as the Weather Service is concerned:
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #27
Deliverables
As far as the Weather Service is concerned:
D.B1.1 v2.4 Standing from FAMAS (a few integrations suggested by Roberto C to be inserted)
D.B2.4 v2.3 Done
D.B2.7 v2.2 Insert results from winter (Reuter + bulletins) 27 27

28.
8th TECHNICAL SUPERVISORY BOARD
TRENTO, 27/04/2015
SLIDE #28
What's next?
Conclude analysis on winter data (probabilistic bulletins)
Deliverables writing up!
Resources on forecast verification from:
Fowler T.L. et al., Introduction to Forecast Verification, University Corporation for Atmospheric Research, 2012
Kryjov V., Probabilistic downscaling: verification methods and metrics. Methods for probabilistic forecasts, NEACC training portal neacc.meteoinfo.ru/training/103-lecture-4-on-forecast-downscaling
Wilson L et al., Forecast Verification, Eumetcal learning module 28 28

29. Thank you for your attention
Provincia Autonoma di Trento Servizio Prevenzione Rischi Ufficio Previsioni e Pianificazione – Meteotrentino
Claudia Di Napoli
Andrea Piazza
Thank you for your attention 29 29