1. Verification Activity in Sub-seasonal Forecasting at JMA 1 Climate Prediction Division / JMA Yuhei Takaya WWRP/THORPEX/WCRP Sub-seasonal to Seasonal Implementation Planning Meeting 2-3 December 2011, Geneva Switzerland Thanks to N. Adachi

2. 1. Verification of real-time operational forecast Monitoring of the performance of real-time operational forecasts on a routine basis. 2. Verification of hindcast Assessing the performance of forecast systems using hindcasts (re-forecasts) for every implementation of systems. Verification of Sub-seasonal Forecast Climate Prediction Division, JMA 2

3. Verification of Hindcast 3

4. 5 members from 3 initial dates in all months during last ~ 30 years (currently 1979-2009) Setups of Hindcast Climate Prediction Division, JMA 4 5 members 10 th 20 th The last day of the month

5. Verification of JMA monthly EPS is based on the SVS-LRF. Scores include deterministic scores (i.e., RMSE, ACC) as well as probabilistic scores (i.e., ROC, BSS). Reference data are JRA-25 re-analysis, GPCP precipitation. Verification Scores Climate Prediction Division, JMA 5 ROC scores for 2-29 days mean precipitation (I.C.: Nov. 30 1979-2009) 0.8 0.5 0.6

6. Our sub-seasonal/seasonal forecasts are provided together with skill assessment based on hindcasts. Tokyo Climate Center: http://ds.data.jma.go.jp/tcc/tcc/products/model/ http://ds.data.jma.go.jp/tcc/tcc/products/model/ Provision of Skill Assessment Climate Prediction Division, JMA 6 http://ds.data.jma.go.jp/tcc/tcc/products/model/map/1mE/map1/pztmap.php

7. The number of ensemble members (5 members) is not enough to get precise assessments of the performance. Supplementary experiments with a larger number of ensemble members for limited initial dates. This would be done in collaborative work of development of integrated EPS. Current verification focus on forecast skill rather than representation of processes. More process-oriented verification would be needed. Representation from a viewpoint of climate modeling. (e.g., energy budget at the surface and TOA, MJO, etc… ) Actions Needed for Hindcast Verification Climate Prediction Division, JMA 7

8. Verification of Real-time Forecasts 8

9. Score Dependence on Periods of Climatology 9 1981-2010 climatology Anomaly correlation of GPH500 for 1979-2004, 1981-2010 climatology (operational forecasts during 2009/03/26-2011/04/21) 1979-1994 climatology 28 days mean Week-1 Week-3&4Week-2 Climate Prediction Division, JMA

10. Score Dependence on Periods of Climatology 10 1981-2010 climatology Anomaly correlation of GPH500 for 1979-2004, 1981-2010 climatology (operational forecasts dufing 2009/03/26-2011/04/21) 1979-1994 climatology 28 days mean Week-1 Week-3&4Week-2 Lesson : ACC scores are affected by change of climatology periods. Lesson : ACC scores are affected by change of climatology periods.

11. Interannual Variability of Scores Climate Prediction Division, JMA 11 T106L40 (Top 0.4hPa) BGM(NH)/26-member BGM(NH+TRO) land analysis improved BGM(TRO) improved Cu conv. Scheme improved radiation Scheme SSM/I improved Cu conv. scheme Sc scheme TL159L40 / 50-member improved radiation scheme COBE-SST (for B.C.) TL159L60 (Top 0.1hPa) improved Cu conv. scheme improved gr. wave scheme

12. Interannual Variability of Scores Climate Prediction Division, JMA 12 T106L40 (Top 0.4hPa) BGM(NH)/26-member BGM(NH+TRO) land analysis improved BGM(TRO) improved Cu conv. Scheme improved radiation Scheme SSM/I improved Cu conv. scheme Sc scheme TL159L40 / 50-member improved radiation scheme COBE-SST (for B.C.) TL159L60 (Top 0.1hPa) improved Cu conv. scheme improved gr. wave scheme Lessons from this exercise: Forecast skill would be affected by the inherent predictability of the climate variability. Sufficient length (samples) of forecast data is needed to verify the performance of sub-seasonal forecast systems. It is better to use both hindcasts and operational forecasts to assess the performance of forecast systems. Lessons from this exercise: Forecast skill would be affected by the inherent predictability of the climate variability. Sufficient length (samples) of forecast data is needed to verify the performance of sub-seasonal forecast systems. It is better to use both hindcasts and operational forecasts to assess the performance of forecast systems.

13. Ongoing work Integrated verification environment for an experiment system of integrated EPS (weekly-monthly). Re-assessment of operational forecast scores with the consistent JRA-25 analysis. (Previously forecast skill has been routinely verified against the deterministic analysis at the time.) Future work… More appropriate setups in order to verify EPS performance. (5 members hindcasts are obviously NOT enough!) Verification of precipitation with in-situ observations. Planned and Ongoing Activity in Verification Climate Prediction Division, JMA 13