1. Venugopal, Basu, and Foufoula-Georgiou, 2005: New metric for comparing precipitation patterns… Verification methods reading group April 4, 2008 D. Ahijevych

2. Forecast Quality Index useful for ensembles uses “surrogate fields” accounts for “close” forecasts One number

3. Outline Paper overview universal image quality index (UIQI) and modified UIQI components of forecast quality index (FQI) Geometric examples (from Sukanta and Efi) Perturbed “fake” examples (also from S and E) Cases from SPC Spring 2005 surrogates traditional skill scores expert rankings

4. Paper overview – forecast ensembles filter out similar members, and keep just enough to characterize the probability structure of forecast find “best” member and propagate it forward single measure (like RMSE and EqTh) but has important additional information

5. Paper overview - UIQI R1 and R2 are fields being compared 3 terms: covariance means standard deviations 3 properties: correlation brightness (bias) distortion (variability)

6. Paper overview – UIQI, Hausdorff UIQI entirely amplitude-based measure not efficient at telling difference between displaced patterns and amplitude error Distance-based measures Hausdorff distance

7. Paper Overview - Hausdorff A B h(A,B) forward distance

8. Paper Overview - Hausdorff A B h(B,A) backward distance

9. Paper Overview - Hausdorff A B h(B,A) backward distance h(A,B) forward distance

10. Paper Overview - Hausdorff A B H(A,B)

11. Paper Overview – partial Hausdorff A B h(A,B) ?

12. Paper overview - Hausdorff AB a1 a2 a3 b1 b2 h(A,B) forward distance

13. Paper overview - FQI

15. Paper Overview - surrogates

16. Paper overview – illustrative example RMSEEqThFQI 0 vs 168.41-0.020.39 0 vs 268.41-0.021.15 0 1 2

18. Geometric examples CSI = 0 for first 4; CSI > 0 for the 5th

19. PHD 75 mod. UIQI mod. UIQI, including zero pixels

24. when I did 10 surrogates = 271 +/-27

25. Perturbed fake cases 1. 3 pts right, -5 pts up 2. 6 pts right, -10 pts up 3. 12 pts right, -20 pts up 4. 24 pts right, -40 pts up 5. 48 pts right, -80 pts up 6. 12 pts right, -20 pts up, times 1.5 7. 12 pts right, -20 pts up, minus 0.05”

34. Spring 2005 SPC cases surrogates pictures example of distribution of forward and backward Hausdorff distances comparison to traditional methods comparison to expert scores

36. 100 surrogates – distribution of Hausdorff distance, solid/forward, dash/backward 75 th percentile Hausdorff distance (in grid spacing units) count

37. standard error surrogate mean PHD 75 mod. UIQI FQI: 0.47- 0.49 PHD 75

38. 0.26-0.28 0.34-0.370.25-0.27

39. 0.21-0.23 0.30-0.310.22-0.23

40. 0.21 0.240.25

41. 0.30 0.190.31

42. 0.51 0.420.69

43. 0.27 0.370.30

44. 0.37 0.330.40

45. 0.34 0.490.33

46. 0.42 0.540.48

47. first case really bad; experts start out too generous? r = w/o 1 st case

48. expert scores vs grid stats grid stats agree: first case was bad

49. Pearson correlation coefficient and Spearman rank correlation coefficient

50. FQI Discussion application to ensembles adding to MET...