1. Verifying Precipitation Events Using Composite Statistics
Jason Nachamkin
Naval Research Laboratory, Monterey, CA

2. Looking for Tigers A region of heavy precipitation exists…
Does the estimate resemble the observations?
Is it even close (heavy precipitation within a radius of the estimate)?
To what degree is the estimate accurate?
Is the estimate more/less accurate under certain conditions?

3. Composite Verification Method
Identify events of interest in the forecasts (estimates)
Rainfall greater than 25 mm
Event contains between 100 and 1000 grid points
Define a kernel and collect coordinated samples
Square box
31x31 grid points (155x155 km for 5 km grid)
Compare forecast PDF to observed PDF
Repeat process for observed events

4. Collecting the Samples
Estimated event
Independent Observations x
This is an example of how data are collected for a typical event type. Here, the event data are collected based on the existence and position of the forecast events. Once an event is found, the kernel grid is placed at the event center, and all model and observed data are placed on the kernel grid. The process is repeated for all events, and when all of the samples are collected two distributions will exist: 1) the distribution of all predicted events, and 2) the distribution of all observations given that an event was predicted. Once the distributions are collected, their statistical properties can be compared. A second pair of distributions can be created based on the existence of observed events. In that case, the performance of the model given that an event was observed can be tracked.
The composite method has the advantage that complex objects do not need to be individually matched. This is good in situations of poor shape correlation or intermittent observations. Composite methods convey a statistical average of what can be expected when an event is either predicted or observed.
Event center
Collection kernel

5. Australian Precipitation Study
All 24-hour precipitation estimates
2 June – 20 Aug. 2003
1 – 29 Feb. 2004
Bureau of Meteorology Research Centre (Beth Ebert)
0.25 deg rain gauge analysis (land)
1000 stns (Feb), 5000, JJA
NRL Blended Satellite Algorithm (Joe Turk)
SSM/I, TRMM, AMSU-B + GOES, Meteosat
Dynamic, statistically-based adjustment
0.25 deg grid
Data interpolated to 5 km Lambert Conformal COAMPS grid

6. Jun-Aug 2003 ETS/Bias All precipitation B=SAT/BMRC
Severe underestimation by satellite algorithm
Very few events sampled

7. Kernel Grid-Average Precipitation
Average rain (mm) given an event was observed by BMRC
Average rain (mm) given an event was observed by SAT
BMRC-shade
SAT-contour
These statistics show that if a rain event is predicted the forecast is typically pretty good, though there is a slight position error (left panel). However, the right panel indicates that the model often does not predict rain events when an event is observed.
N=13
N=5
Severe underestimation of inland events by satellite algorithm.
Satellite estimates were better near the coast.

8. Statistics From Single Events
Max SAT
Max BMRC
Sample Integrated SAT
Sample Integrated BMRC

9. Event Detection Frequencies JJA
SAT more
BMRC more
Several BMRC-observed events almost completely missed
Very few events sampled

10. 24 July 2003 Precipitation 24-hr BMRC Rain (mm) 24-hr SAT Rain (mm)
128
120
112
104 96 88 80 72 64 56 48 40 32 24 16 8
These statistics show that if a rain event is predicted the forecast is typically pretty good, though there is a slight position error (left panel). However, the right panel indicates that the model often does not predict rain events when an event is observed.
“False alarm” along east coast
“Missed event” in southern interior

11. Feb 2004 ETS/Bias All precipitation B=SAT/BMRC
Severe overestimation of intense rainfall by satellite algorithm
Better areal coverage at lower intensities

12. Kernel Grid-Average Precipitation
Average rain (mm) given an event was observed by BMRC
Average rain (mm) given an event was observed by SAT
BMRC-shade
SAT-contour
These statistics show that if a rain event is predicted the forecast is typically pretty good, though there is a slight position error (left panel). However, the right panel indicates that the model often does not predict rain events when an event is observed.
N=123
N=148
Satellite much better at correctly placing events.
Satellite precipitation too heavy near event center.
Some “false alarms” in satellite estimates.

13. Event Detection Frequencies Feb
SAT more
BMRC more
When event in BMRC, satellite underestimates grid total and overestimates grid maximum.
False alarms contribute to satellite overestimation in the satellite event composite.

14. Daily Forecast Frequencies

15. 2 Feb 2004 Precipitation 24-hr BMRC Rain (mm) 24-hr SAT Rain (mm)
128
120
112
104 96 88 80 72 64 56 48 40 32 24 16 8
These statistics show that if a rain event is predicted the forecast is typically pretty good, though there is a slight position error (left panel). However, the right panel indicates that the model often does not predict rain events when an event is observed.
General precipitation areas are similar.
Satellite contains greater detail, much higher maxima.
Possible false alarm in southeast.

16. Conclusions
During June-August, the satellite estimates often missed events except near the coast.
During February, the satellite performed better though often overestimated precipitation maxima.
“False alarms” and “missed events”, where the satellite and BMRC estimates differed significantly, were common.
Separate statistics from 24-hour model forecasts over the US were better than the satellite estimates.