Verifying Precipitation Events Using Composite Statistics Jason Nachamkin Naval Research Laboratory, Monterey, CA

Looking for Tigers A region of heavy precipitation exists… Does the estimate resemble the observations? Is it even close (heavy precipitation within a radius of the estimate)? To what degree is the estimate accurate? Is the estimate more/less accurate under certain conditions?

Composite Verification Method Identify events of interest in the forecasts (estimates) Rainfall greater than 25 mm Event contains between 100 and 1000 grid points Define a kernel and collect coordinated samples Square box 31x31 grid points (155x155 km for 5 km grid) Compare forecast PDF to observed PDF Repeat process for observed events

Collecting the Samples Estimated event Independent Observations x This is an example of how data are collected for a typical event type. Here, the event data are collected based on the existence and position of the forecast events. Once an event is found, the kernel grid is placed at the event center, and all model and observed data are placed on the kernel grid. The process is repeated for all events, and when all of the samples are collected two distributions will exist: 1) the distribution of all predicted events, and 2) the distribution of all observations given that an event was predicted. Once the distributions are collected, their statistical properties can be compared. A second pair of distributions can be created based on the existence of observed events. In that case, the performance of the model given that an event was observed can be tracked. The composite method has the advantage that complex objects do not need to be individually matched. This is good in situations of poor shape correlation or intermittent observations. Composite methods convey a statistical average of what can be expected when an event is either predicted or observed. Event center Collection kernel

Australian Precipitation Study All 24-hour precipitation estimates 2 June – 20 Aug. 2003 1 – 29 Feb. 2004 Bureau of Meteorology Research Centre (Beth Ebert) 0.25 deg rain gauge analysis (land) 1000 stns (Feb), 5000, JJA NRL Blended Satellite Algorithm (Joe Turk) SSM/I, TRMM, AMSU-B + GOES, Meteosat Dynamic, statistically-based adjustment 0.25 deg grid Data interpolated to 5 km Lambert Conformal COAMPS grid

Jun-Aug 2003 ETS/Bias All precipitation B=SAT/BMRC Severe underestimation by satellite algorithm Very few events sampled

Kernel Grid-Average Precipitation Average rain (mm) given an event was observed by BMRC Average rain (mm) given an event was observed by SAT BMRC-shade SAT-contour These statistics show that if a rain event is predicted the forecast is typically pretty good, though there is a slight position error (left panel). However, the right panel indicates that the model often does not predict rain events when an event is observed. N=13 N=5 Severe underestimation of inland events by satellite algorithm. Satellite estimates were better near the coast.

Statistics From Single Events Max SAT Max BMRC Sample Integrated SAT Sample Integrated BMRC

Event Detection Frequencies JJA SAT more BMRC more Several BMRC-observed events almost completely missed Very few events sampled

24 July 2003 Precipitation 24-hr BMRC Rain (mm) 24-hr SAT Rain (mm) 128 120 112 104 96 88 80 72 64 56 48 40 32 24 16 8 These statistics show that if a rain event is predicted the forecast is typically pretty good, though there is a slight position error (left panel). However, the right panel indicates that the model often does not predict rain events when an event is observed. “False alarm” along east coast “Missed event” in southern interior

Feb 2004 ETS/Bias All precipitation B=SAT/BMRC Severe overestimation of intense rainfall by satellite algorithm Better areal coverage at lower intensities

Kernel Grid-Average Precipitation Average rain (mm) given an event was observed by BMRC Average rain (mm) given an event was observed by SAT BMRC-shade SAT-contour These statistics show that if a rain event is predicted the forecast is typically pretty good, though there is a slight position error (left panel). However, the right panel indicates that the model often does not predict rain events when an event is observed. N=123 N=148 Satellite much better at correctly placing events. Satellite precipitation too heavy near event center. Some “false alarms” in satellite estimates.

Event Detection Frequencies Feb SAT more BMRC more When event in BMRC, satellite underestimates grid total and overestimates grid maximum. False alarms contribute to satellite overestimation in the satellite event composite.

Daily Forecast Frequencies

2 Feb 2004 Precipitation 24-hr BMRC Rain (mm) 24-hr SAT Rain (mm) 128 120 112 104 96 88 80 72 64 56 48 40 32 24 16 8 These statistics show that if a rain event is predicted the forecast is typically pretty good, though there is a slight position error (left panel). However, the right panel indicates that the model often does not predict rain events when an event is observed. General precipitation areas are similar. Satellite contains greater detail, much higher maxima. Possible false alarm in southeast.

Conclusions During June-August, the satellite estimates often missed events except near the coast. During February, the satellite performed better though often overestimated precipitation maxima. “False alarms” and “missed events”, where the satellite and BMRC estimates differed significantly, were common. Separate statistics from 24-hour model forecasts over the US were better than the satellite estimates.