1. Dipartimento della Protezione Civile Italiana
Methodology and preliminary results of the H-saf precipitation products products validation
Silvia Puca
In collaboration with RMI (Belgium), BFG (Germany), OMSZ (Hungary),
UniFe and DPC (Italy), IMWG (Poland), SHMI (Slovakia), ITU TMS (Turkey)
Dipartimento della Protezione Civile Italiana

2. outline H-saf Precipitation Products;
H-saf Precipitation Products program and members of Calibration and Validation Group;
H-saf Precipitation Products;
Precipitation ground measurements used for the validation;
Validation Methodology: common and specific validation;
Up-scaling techniques;
Preliminary feedback on AMSU and AMSU+SEVIRI products;
Conclusions;
Future steps.
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

3. Precipitation Product Validation Program:
characterize the product error structure whose knowledge is needed for correct utilisation;
supporting algorithms and models tuning (i.e., calibration) during their development process;
collecting routine reporting from end-users and special reporting from experimental activities;
continuing calibration/validation activities during the pre-operational phase.
Workshop on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007, Geneva,

4. Calibration/validation
H-saf Precipitation Products Cal/Val Group
WP-2300
Calibration/validation
Italy (DPC)
WP-2310
Cal/val
Belgium
WP-2320
Germany
WP-2330
Hungary
WP-2340
Italy
WP-2350
Poland
WP-2360
Slovakia
WP-2370
Turkey
Belgium: Royal Meteorological Institute (IRM)
Germany: Federal Institute of Hydrology (BfG)
Hungary: Országos Meteorológiai Szolgálat-Távérzékelési Osztály- (HMS)
Italy: Università di Ferrara (UniFe),
Dipartimento della Protezione Civile (DPC),
Poland: Institute of Meteorology and Water Management (IMWM)
Slovakia: Slovak Hydrometeorological Institute (SHMÚ)
Turkey: Istanbul Technical University, Meteorology Department (ITU),
Middle East Technical University (METU)
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

5. H-saf Precipitation Products;
Table 1 - List of H-SAF products and indication of the Units responsible of algorithm development.
Code
Acronym
Product name
Responsible of algorithm
H-01
PR-OBS-1
Precipitation rate at ground by MW conical scanners (with indication of phase) SSMI - CRDB
Italy, CNR-ISAC
H-02
PR-OBS-2
Precipitation rate at ground by MW cross-track scanners (with indication of phase) AMSU - NN
H-03
PR-OBS-3
Precipitation rate at ground by GEO/IR supported by LEO/MW
SSMI+AMSU+ SEVIRI - blending
H-04
PR-OBS-4
Precipitation rate at ground by LEO/MW supported by GEO/IR (with flag for phase) SSMI+AMSU+ SEVIRI - morphing
H-05
PR-OBS-5
Accumulated precipitation at ground by MW+IR and MW only
Italy, CNMCA
H-06
PR-ASS-1
Instantaneous and accumulated precipitation at ground computed by a NWP model
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

6. Precipitation ground measurements used for the validation
In situ data data:
automatic rain gauges with different time resolution:
5 min, 10 min, 15 min., 30 min.
synoptic stations with different time resolution :
6 hour, 24 hour cumulated values;
Meteorological Radars with different time resolution:
Someone gauge adjusted
Data for cloud type classification, containing information about water content in vertical column and for the discrimination of the synoptic situation: NWC-saf, NWP models, MSG composite image.
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

7. PPV Raingauge network is composed by 4100 stations:
Data Sources
raingauges
Instrument characteristics
Telemetric and mechanic
time domain (near real time/ case studies)
Near real time, case studies
time resolution (15 min, 30 min)
10 – 30 min (telemetric),
3 – 24 h (mechanic)
spatial distribution (whole national territory/ limited area)
Whole national territory
number of station (please attach a map)
~390 mechanic (RMI) + 12 telemetric (RMI) telemetric (SETHY)
operational/ for research only
Operational (RMI) + research (other networks)
data quality check
Telemetric: automatically checked / mechanic: autom. + manually checked
PPV Rainauge network is composed by 4100 telemetric stations:

8. PPV Radar network is composed by 40 C-band:
Data Sources
radars
Instrument characteristics
Beam width ~1°,
max range ~150 Km, 250m,
C-band, single polarization, Doppler
polarimetric
time domain
near real time/ case studies
time resolution
5 min, 15 min, 30 min, 1h, 24h
spatial distribution
Whole national territory
number of station
33 C band +1 Ka band
operational/ for research only
Operational
data quality check
Permanent ground clutter removed; monitoring of electronic calibration
PPV Radar network is composed
by 33 C-band and 1 Ka-band:
PPV Radar network is composed
by 33 C-band and 1 Ka-band:

9. Validation Methodology: common and specific validation
Common methodology:
rain gauges and radar data;
multi categorical and continuous statistics evaluated on long time series;
Contingency tables and errors are evaluated in the same way by all the institutes involved using the same up-scaling techniques.
Statistical error evaluated by all the institute in the same way on long time series
Each Institute in addition to the common validation methodology has developed a specific validation methodology based on its own knowledge and experience.
lightning data, numerical weather prediction and nowcasting product
case studies: convective/stratiform precipitation, day/night, land/ocean
deep analysis met. events, important Calibration task
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

10. Common validation
Continuous verification statistics: calculating Mean absolute error, root mean square error, correlation coefficient, standard deviation etc;
Multi-Categorical statistics: calculating the contingency table (which allows for evaluation of false alarm rate, probability of detection, equitable threat score, Heidke skill score, etc ).
All the tools used for this validation are automatic tool.
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

11. Multi-Categorical statistics
For the rain rate (RR) derived from MW imagery (spatial resolution of Km) the precipitation classes of the multi categorical statistics were defined following the hydrologist, developers requirements and project plan:
Project Plan
Developers request
Class 1
RR < 1 mm/h
Class 2
1 mm/h ≤RR < 10 mm/h
Class 3
10 mm/h ≤ RR
Class 1
RR ≤ 0.25 mm/h
Class 2
0.25 mm/h < RR ≤ 0.5 mm/h
Class 3
0.5 mm/h < RR ≤ 1 mm/h
Class 4
1 mm/h < RR ≤ 2 mm/h
Class 5
2 mm/h < RR ≤ 4 mm/h
Class 6
4 mm/h < RR ≤ 8 mm/h
Class 7
8 mm/h < RR ≤ 16 mm/h
Class 8
16 mm/h < RR ≤ 32 mm/h
Class 9
32 mm/h < RR ≤ 64 mm/h
Class 10
64 mm /h < RR
Radar classes
Class 1
RR ≤ 0.5 mm/h
Class 2
0.5 mm/h < RR ≤ 2 mm/h
Class 3
2 mm/h < RR ≤ 10 mm/h
Class 4
10 mm/h < RR ≤ 16 mm/h
Class 5
8 mm/h < RR ≤ 16 mm/h
Class 6
16 mm/h < RR ≤ 32 mm/h
Class 7
64 mm /h < RR
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

12. H-saf Precipitation Products;
Common validation:
PR-OBS-2v1.1: August August 2007;
(AMSU: neural network algorithm trained on radar data (NEXRAD))
PR-OBS-2v2.1: August August 2007;
(AMSU: neural network algorithm trained on numerical model (MM5))
PR-OBS-3v1.0: June August 2007
(AMSU + IR: Blending techniques)
Specific validation: 12 case studies selected by the involved institues.
Radar, rain gauge, convective detection, numerical model data have been used. Descrimination between convective/stratiform precipitation, day/night, land/ocean:
PR-OBS-1v1.0 (Rain Rate: SSMI)
PR-OBS-2v1.1 (Rain Rate: AMSU)
PR-OBS-2v2.1 (Rain Rate: AMSU)
PR-OBS-3v1.0 (Rain Rate: AMSU + SEVIRI)
PR-OBS-5v1.0 (Cumulated Precipitation: AMSU + SEVIRI)
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

13. UP-SCALING on SATELLITE NATIVE GRID: A common up-scaling method has been applied to raingauge and radar data vs AMSU -B grid: a Gaussian filter,
Take into account that precipitation data in the AMSU retrieval product (H02) follows the scanning geometry and IFOV resolution of AMSU-B scan, so that each pixel along the scan has a precipitation value representative for an elliptical region with different size.
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007
Fig 1 – Left) Gaussian filter – Right) section of gaussian filter. If the Radar resolution is 1km, 1px=1km

14. : observations satellite number of pixels : raining points (rr>0,25 mm/h): class more frequent : max class :
Precipitation Product: PR-OBS-02v11 type of observation: RADAR date:
PRECIPITATION CLASSES REQUIRED BY PROJECT PLAN
POD 0, , ,005
FAR 0, , ,999
ACC 0, , ,966
BIAS 0, , ,269
POFD 0, , ,026
CSI 0, , ,001
ETS 0, , ,005
HK 0, , ,021
HSS 0, , ,010
OR 6, , ,175
PRECIPITATION CLASSES SIMILAR TO RADAR CLASSES
POD 0, , , , , , ,000
FAR 0, , , , , , ,000
ACC 0, , , , , , ,999
BIAS 0, , , , , , ,646
POFD 0, , , , , , ,001
CSI 0, , , , , , ,000
ETS 0, , , , , , ,000
HK 0, , , , , , ,001
HSS 0, , , , , , ,000
OR 7, , , , , , ,000
PRECIPITATION CLASSES REQUIRED BY DEVELOPERS
POD 0, , , , , , , , , ,000
FAR 0, , , , , , , , , ,000
ACC 0, , , , , , , , , ,999
BIAS 0, , , , , , , , , ,646
POFD 0, , , , , , , , , ,001
CSI 0, , , , , , , , , ,000
ETS 0, , , , , , , , , ,000
HK 0, , , , , , , , , ,001
HSS 0, , , , , , , , , ,000
OR 5, , , , , , , , , ,000
CONTINGENCY TABLE
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

15. Preliminary results of the common validation on OBS02v1.1 v OBS02v2.1:
Data used: Radar and rain gauge
Period : August 2006-August 2007,
Score : Accuracy

16. Preliminary results of the common validation on OBS02v1.1 v OBS02v2.1:
Data used: Radar and rain gauge
Period : August 2006-August 2007,
Score : Probability Of Detection

17. Preliminary results of the common validation on OBS02v1.1 v OBS02v2.1:
Data used: Radar and rain gauge
Period : August 2006-August 2007,
Score : False alarm ratio

18. Table 8 - Multi category scores of PR-OBS-2 v1 evaluated over one year
of radar and rain gauge data for three precipitation classes
PR-OBS-2 v1
radar
rain gauges
class 1
class 2
class 3
Accuracy
0.97
0.99
Probability Of Detection
0.96
0.20
0.01
0.98
0.13
0.17
Critical Success Index
0.92
0.07
0.00
0.03
False-alarm rate
0.02
0.82
0.78
Frequency Bias
2.41
2,47
1.3
5,42
Table 7 - Minimum, mean and maximum statistical scores of PR-OBS-2 v1
evaluated over one year of radar and rain gauge data
PR-OBS-2 v1
radar
rain gauges
min
mean
max
Mean error (mm/h)
-0.02
0.18
0.77
-3.67
-1.21
3.81
Mean absolute error (mm/h)
0.05
0.29
0.92
1.60
3.46
8.05
MSE (mm/h)2
0.01
1.32
7.37
10.81
62.32
281.11
RMSE (mm/h)
0.08
1.10
3.83
2.51
5.67
12.32
Multiplicative bias
0.20
1.55
5.02
0.11
6.03
Correlation coefficient
0.07
0.25
0.42
-0.08
0.09
0.44
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

19. Table 10 - Multi category scores of PR-OBS-2 v2 evaluated over one year
of radar and rain gauge data for three precipitation classes
PR-OBS-2 v2
radar
rain gauges
class 1
class 2
class 3
Accuracy
0.97
0.99
0.93
Probability Of Detection
0.23
0.05
0.94
0.28
0.06
Critical Success Index
0.96
0.00
0.12
False-alarm rate
0.01
0.92
0.02
0.84
0.80
Frequency Bias
0.95
3.74
12.90
3.6
0.26
Table 9 - Minimum, mean and maximum statistical scores of PR-OBS-2 v2
evaluated over one year of radar and rain gauge data
PR-OBS-2 v2
radar
rain gauges
min
mean
max
Mean error (mm/h)
0.02
0.09
0.29
-0.56
0.13
1.01
Mean absolute error (mm/h)
0.08
0.17
0.44
0.23
0.53
1.12
MSE (mm/h)2
0.32
0.80
5.43
16.5
RMSE (mm/h)
0.19
0.37
0.79
0.89
1.92
3.34
Multiplicative bias
0.9
2.34
11.3
0.94
4.61
21.6
Correlation coefficient
0.28
0.47
-0.03
0.15
0.41
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

20. Preliminary results of the common validation on OBS03v1.01:
Data used: rain gauge only
Period : June 2007-August 2007,
Score : Accuracy, Heidke skill score

21. Preliminary results of the common validation on OBS03v1.01:
Data used: rain gauge only
Period : June 2007-August 2007,
Score : Probability of Detection, False Alarm Ratio

22. Preliminary results of the common validation on OBS02v1.1 v OBS02v2.1:
Data used: Radar and rain gauge
Period : August 2006-August 2007,
Score : Corr. Coef., Mean absolute error ,Mean Error

23. Preliminary results of the common validation on OBS02v1.1 v OBS02v2.1:
Data used: Radar and rain gauge
Period : August 2006-August 2007,
Score : Root Mean Sq. Error, Standard Deviation, Multi. Bias

24. AMSU + SEVIRI: blending technique
Table 12 - Multi-category scores of PR-OBS-3 evaluated over
three months of rain gauge data for three precipitation classes
PR-OBS-3
raingauge
class 1
class 2
class 3
Accuracy
0.81
0.86
0.95
Probability Of Detection
0.18
0.07
Critical Success Index
0.83
0.06
0.03
False-alarm rate
0.90
0.94
Frequency Bias
0.92
2.29
4.29
Table 11 - The mean statistical scores of PR-OBS-3
evaluated over three months rain gauge data
PR-OBS-3
Rain gauges
Mean error (mm/h)
0.45
Mean absolute error (mm/h)
0.68
MSE (mm/h)2
25.6
RMSE (mm/h)
4.96
Multiplicative bias
4.67
Standard deviation (mm/h)
3.12
Correlation coefficient
0.11
The scores here presented were evaluated using precipitation and no-precipitation data. The large amount of no-precipitation data influenced the statistical results!!!
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

25. Preliminary results on OBS-02 and OBS-03 from statistical error evaluation and case studies analysis
Precipitation clusters are well detected by both OBS2v11, OBS2v21 and OBS3v10;
OBS02v1 seems underestimate the RR more than OBS2v2 (ME and MAE better v2 than v1);
Similar values of accuracy, FAR and POD for both (2v1 that 2v2). V1 is very good for class1 better during the autumn and winter periods than summer one;
Frontal cases are well described in both obs02v1 and obs02v2, but the RR is more realistic in obs02v1.0;
obs02v2.0 is better during the summer period, in detecting very high convective cells and convective systems than low rain rate ;
obs02v2.0 some times detects precipitation where clouds are not present, background problems;
H03 good performances for class1 but not enough data are available for a significant statistical validation (raingauge only);
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

26. It is important to take into account that: - The quality of H02 products depends on the pixel number: the closer to the ends of AMSU scan line, the lower quality. - Majority of radars used for validation are not gauge adjusted; - time sampling of radar and raingauge used are different (10 min 15 min, 30 min); - interpolated gridpoints are not at same altitude of the nearby stations; - orographic precipitation enhancement depends on meteorological structure; - Different channels are used from the same product OBS2v2.1 for different satellite pass.
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

27. Conclusions:
Radar and raingauge archive from October 2006 for all the institutes involved in the PP CalVal is available;
Common up-scaling techniques and share of procedures;
Preliminary results on OBS02 and OBS03 indicate a a good discrimination between rain/no-rain but a general underestimate of the precipitation rate;
Remarks:
Which precipitation classes can the mw and infrared data discriminate? Work on high precipitation rate.
Standardization of radar and raingauge data is necessary.
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

28. Future steps: Take into account the orography;
Add new classes as land or sea, day or night also in the common validation;
automatization of validation procedure described;
Parallax and geolocation errors should be accounted;
Quality control of the data used for validation (radar, raingauges);
Evaluation of continuous and multi-category statistic on all products: H01, H02, H03 and H05;
WS on the Evaluation of High Resolution Precipitations Products, WMO 3-5 December 2007

29. NEFODINA 01:45 AMSU 01:52 NEFODINA 02:00 AMSU 01:52
Visiting Scientist on refinement and operational implementation of a rain rate algorithm based AMSU/MHS, and SEVIRI data within the Hydrological-SAF Antonelli P., Bennartz R. Puca S., Tassa A., Zauli F.
CO RR (01:45 UTC)=48 mm/hr
CO RR (02:00 UTC)=38 mm/hr
Radar RR=42 mm/hr
NEFODINA 01:45
AMSU 01:52
RADAR on SEVIRI GRID
NEFODINA 02:00
AMSU 01:52
Redistribution of AMSU-B averaged RR ifov on SEVIRI under the hypotesis that most of the precipitation occores in convective region

30. Thank you!!