Spatial Verification Intercomparison Meeting, 20 February 2007, NCAR Fuzzy Verification Ebert, E.E., 2007: Fuzzy verification of high resolution gridded forecasts: A review and proposed framework. Meteorol. Appls., submitted Available online at http://www.bom.gov.au/bmrc/wefor/staff/eee/beth_ebert.htm Spatial Verification Intercomparison Meeting, 20 February 2007, NCAR

Why is it called "fuzzy"? Squint your eyes! observation forecast

Data used Any spatial forecasts and observations Best suited for high resolution Most convenient if forecast is on a grid Observations can be gridded or point data

How does it work? Look in a space / time neighborhood around the point of interest Evaluate using categorical, continuous, probabilistic scores / methods Will only consider spatial neighborhood for the moment t t + 1 t - 1 Forecast value Frequency

How does it work? (cont'd) Fuzzy methods use one of two approaches to compare forecasts and observations: observation forecast single observation – neighborhood forecast neighborhood observation – neighborhood forecast observation forecast

Methods and decision models Many fuzzy methods have been developed in recent years. The main thing that distinguishes them is whether they are NO-NF or SO-NF, and what the decision model is for what constitutes a useful forecast. Barbara Casati pointed out in Reading that her intensity-scale method really differs from the rest of these methods in that it isolates the errors at each scales, whereas the fuzzy methods essentially smooth out the behavior by scale. *NO-NF = neighborhood observation-neighborhood forecast, SO-NF = single observation-neighborhood forecast

Information provided Forecast performance depends on the scale and intensity of the event

Strengths Knowing which scales have skill suggests the scales at which the forecast should be presented and trusted Can give good results for forecasts that verify poorly using exact-match approach Suitable for discontinuous fields like precipitation Can be used to compare forecasts at different resolutions Multiple decision models and metrics Direct approach  verification of intensities Categorical approach  verification of binary events Probabilistic approach  verification of event frequency Can be extended to time domain Other diagnostics available from some methods (e.g. PP)

Weaknesses and limitations Less intuitive than object-based methods Imperfect scores for perfect forecasts for methods that match neighborhood forecasts to single observations Information overload if all methods invoked at once Let appropriate decision model(s) guide the choice of method(s) Even for a single method … there are lots of numbers to look at evaluation of scales and intensities with best performance depends on metric used (CSI, ETS, HK, etc.)

Example: 13 May 2005 The circles indicate the intensity-scale combination at which the best score was achieved. Depending on the method (i.e. on the decision model for a "good" forecast) different intensities and scales are selected. For this forecast the best performance tended to be at the larger scales for almost all methods, although there was less agreement on the intensities with best skill.

Best performer on 13 May 2005 Scale (km) Scale (km) Fractions skill score – FSS (neighborhood obs – neighborhood fcst) 260 wrf4ncar wrf2caps 132 wrf4ncep 68 - 36 20 12 4 1 2 5 10 50 100 200 Scale (km) Threshold (0.01") Multi-event contingency table - HK (single obs – neighborhood fcst) 260 wrf4ncar wrf2caps 132 68 36 20 wrf4ncep - 12 4 1 2 5 10 50 100 200 FSS is a neighborhood observation – neighborhood forecast method, which is model-oriented. Multi-event cont. table is single observation – neighborhood forecast, which is user-oriented. Even so, for most scales and intensities the two approaches agreed on which model performed best. Scale (km) Threshold (0.01")