1. Spatial Verification Intercomparison Meeting, 20 February 2007, NCAR
Fuzzy Verification
Ebert, E.E., 2007: Fuzzy verification of high resolution gridded forecasts: A review and proposed framework. Meteorol. Appls., submitted
Available online at
Spatial Verification Intercomparison Meeting, 20 February 2007, NCAR

2. Why is it called "fuzzy"?
Squint your eyes!
observation
forecast

3. Data used Any spatial forecasts and observations
Best suited for high resolution
Most convenient if forecast is on a grid
Observations can be gridded or point data

4. How does it work?
Look in a space / time neighborhood around the point of interest
Evaluate using categorical, continuous, probabilistic scores / methods
Will only consider spatial neighborhood for the moment t
t + 1
t - 1
Forecast value
Frequency

5. How does it work? (cont'd)
Fuzzy methods use one of two approaches to compare forecasts and observations:
observation
forecast
single observation – neighborhood forecast
neighborhood observation – neighborhood forecast
observation
forecast

6. Methods and decision models
Many fuzzy methods have been developed in recent years. The main thing that distinguishes them is whether they are NO-NF or SO-NF, and what the decision model is for what constitutes a useful forecast.
Barbara Casati pointed out in Reading that her intensity-scale method really differs from the rest of these methods in that it isolates the errors at each scales, whereas the fuzzy methods essentially smooth out the behavior by scale.
*NO-NF = neighborhood observation-neighborhood forecast,
SO-NF = single observation-neighborhood forecast

7. Information provided
Forecast performance depends on the scale and intensity of the event

8. Strengths
Knowing which scales have skill suggests the scales at which the forecast should be presented and trusted
Can give good results for forecasts that verify poorly using exact-match approach
Suitable for discontinuous fields like precipitation
Can be used to compare forecasts at different resolutions
Multiple decision models and metrics
Direct approach  verification of intensities
Categorical approach  verification of binary events
Probabilistic approach  verification of event frequency
Can be extended to time domain
Other diagnostics available from some methods (e.g. PP)

9. Weaknesses and limitations
Less intuitive than object-based methods
Imperfect scores for perfect forecasts for methods that match neighborhood forecasts to single observations
Information overload if all methods invoked at once
Let appropriate decision model(s) guide the choice of method(s)
Even for a single method …
there are lots of numbers to look at
evaluation of scales and intensities with best performance depends on metric used (CSI, ETS, HK, etc.)

10. Example: 13 May 2005
The circles indicate the intensity-scale combination at which the best score was achieved. Depending on the method (i.e. on the decision model for a "good" forecast) different intensities and scales are selected. For this forecast the best performance tended to be at the larger scales for almost all methods, although there was less agreement on the intensities with best skill.

11. Best performer on 13 May 2005 Scale (km) Scale (km)
Fractions skill score – FSS (neighborhood obs – neighborhood fcst)
260
wrf4ncar
wrf2caps
132
wrf4ncep 68 - 36 20 12 4 1 2 5 10 50
100
200
Scale (km)
Threshold (0.01")
Multi-event contingency table - HK (single obs – neighborhood fcst)
260
wrf4ncar
wrf2caps
132 68 36 20
wrf4ncep - 12 4 1 2 5 10 50
100
200
FSS is a neighborhood observation – neighborhood forecast method, which is model-oriented.
Multi-event cont. table is single observation – neighborhood forecast, which is user-oriented.
Even so, for most scales and intensities the two approaches agreed on which model performed best.
Scale (km)
Threshold (0.01")