Download presentation
Presentation is loading. Please wait.
1
Comparison of Scores in Model Comparison (MSSS,COR,RMSE and ROC)
Tomoaki OSE (MRI/JMA) With help of Yukiko NARUSE (CPD/JMA) and use of JMA/MRI CGCM and AGCM Hindcasts
2
MSSS, COR and RMSE (from SVS_LRF of WMO manual)
3
FOR RAIN ANOMALY OF NH SUMMER PREDICTION FROM NH WINTER
Dependence of MODEL comparisons on SCORES FOR RAIN ANOMALY OF NH SUMMER PREDICTION FROM NH WINTER
4
Originally defined MSSS is not useful at current Seasonal Forecasts
AGCM MSSS CGCM MSSS CGCM BIAS AGCM BIAS
5
Better COR does not necessarily correspond to better MSSS.
AGCM MSSS CGCM MSSS CGCM COR AGCM COR
6
MSSS is largely influenced by model SDs.
AGCM MSSS CGCM MSSS CGCM SDmdl/ SDanal AGCM SDmdl/ SDanal
7
The change of RMSE is more similar to the change of MSSS than COR.
AGCM RMSE CGCM RMSE CGCM COR AGCM COR
8
Comparison of SCOREs: ROC map seems to be more similar to COR map.
CGCMROC CGCMMSSS CGCM COR CGCMRMSE
9
Summary It is confirmed that better COR does not necessarily indicate better MSSS in model comparison. Because MSSS is definitely dependent on model SD (and/or BIAS) in addition to COR. The change of RMSE shows similar characteristics to the change of MSSS. Small RMSE regions do not correspond to the regions for good seasonal forecasts. COR map seems to show more similar distribution to ROC map than MSSS map.
10
Why COR for SIF modelers?
Temporal correlation is the most fundamental skill measure for SI forecasts predictability. The square of correlation is the ratio of forecast signal variance to the total variance. BIAS, SD, MSSS and RMSE can be corrected by linear transform using OBS while correlation is invariant. Correlation is widely used and most users are familiar with it and easy to understand. In order to evaluate the value of probability forecast for particular user’s application, more sophisticated skill measure may be needed anyway for each.
11
FOR T2m ANOMALY OF NH WINTER PREDICTION FROM NH SUMMER
Dependence of MODEL comparisons on SCORES FOR T2m ANOMALY OF NH WINTER PREDICTION FROM NH SUMMER
12
Comparison of SCOREs: ROC map seems to be more similar to COR map.
CGCMROC CGCMMSSS CGCMRMSE CGCM COR
13
FOR RAIN ANOMALY OF NH WINTER PREDICTION FROM NH SUMMER
Dependence of MODEL comparisons on SCORES FOR RAIN ANOMALY OF NH WINTER PREDICTION FROM NH SUMMER
14
Comparison of SCOREs: ROC map seems to be more similar to COR map.
CGCMROC CGCMMSSS CGCMCOR CGCMRMSE
15
FOR T2m ANOMALY OF NH SUMMER PREDICTION FROM NH WINTER
Dependence of MODEL comparisons on SCORES FOR T2m ANOMALY OF NH SUMMER PREDICTION FROM NH WINTER
16
Comparison of SCOREs: ROC map seems to be more similar to COR map.
CGCMROC CGCMMSSS CGCM COR CGCMRMSE
Similar presentations
