General framework for features-based verification Mike Baldwin Purdue University

General framework Any verification method should be built upon the general framework for verification outlined by Murphy and Winkler (1987) Object-oriented or features-based methods can be considered an extension or generalization of the original framework Joint distribution of forecasts and observations: p(f,o)

general joint distribution p(f,o) : where f and o are vectors containing all variables, 3-D, same time o could come from data assimilation o=first guess + weight*forward_model(obs) p(fm,om) : fm and om are values of a single variable across the entire domain traditional joint distribution

general joint distribution p(G(f),G(o)) : where G is some mapping/transformation/operator that is applied to the variable values morphing filter convolution deformation fuzzy

general joint distribution p(Gm(f),Gm(o)) : where Gm is a specific aspect/attribute/characteristic that results from the mapping operator distributions-oriented measures-oriented compute some error measure or score that is a function of Gm(f),Gm(o) MSE

Terminology that we might standardize “feature” – a distinct or important physical object that can be identified within meteorological data “attribute” – a characteristic or quality of a feature, an aspect that can be measured “similarity” – the degree of resemblance between features “distance” – the degree of difference between features (others?)

framework follow Murphy (1993) and Murphy and Winkler (1987) terminology joint distribution of forecast and observed features goodness: consistency, quality, value

aspects of quality accuracy: correspondence between forecast and observed feature attributes single and/or multiple? bias: correspondence between mean forecast and mean observed attributes resolution reliability discrimination stratification

Features-based process FCST OBS Identify features

Features-based process FCST OBS Characterize features

Features-based process How to measure differences between objects? How to determine false alarms/missed events? FCST OBS Compare features

Features-based process FCST OBS Classify features

feature identification procedures for locating a feature within the meteorological data will depend on the problem/phenomena/user of interest a set of instructions that can (easily) be followed/programmed in order for features to be objectively identified in an automated fashion

feature characterization a set of attributes that describe important aspects of each feature numerical values will be the most useful

feature comparison similarity or distance measures systematic method of matching or pairing observed and forecast features determination of false alarms? determination of missed events?

classification a procedure to place similar features into groups or classes reduces the dimensionality of the verification problem similar to going from a scatter plot to a contingency table not necessary/may not always be used

SSEC MODIS archive 10 Apr 2003

feature matching

attributes Lake Fcst #1 Fcst #2 Obs #1 Obs #2 Obs #3 Lat 47.7 44.0 44.8 42.2 43.7 Lon 87.5 87 82.4 81.2 77.9 Area (km2) 82400 58000 59600 25700 19500 Volume (km3) 12000 4900 3540 480 1640 Max depth (m) 406 281 230 64 246

How to match observed and forecast objects? = missed event O1 dij = ‘distance’ between F i and O j O3 Objects might “match” more than once… If d*j > dT : missed event O2 F1 …for each forecast object, choose closest observed object …for each observed object, choose closest forecast object If di* > dT then false alarm F2 = false alarm

Example of object verf ARW 2km (CAPS) Radar mosaic Fcst_2 Obs_2 Fcst_1 Object identification procedure identifies 4 forecast objects and 5 observed objects

Distances between objects ARW 2km (CAPS) Radar mosaic Distances between objects Use dT = 4 as threshold Match objects, find false alarms, missed events O_34 O_37 O_50 O_77 O_79 F_25 5.84 4.16 8.94 9.03 11.53 F_27 6.35 2.54 7.18 6.32 9.25 F_52 7.43 9.11 4.15 9.19 5.45 F_81 9.39 6.36 2.77 5.24

ARW2 ARW4 Df = .07 Dl = .08 Df = .04 Dl = -.07 median position errors matching obs object given a forecast object NMM4 Df = .04 Dl = .22