1. Huiling Yuan 1, Xiang Su 1, Yuejian Zhu 2, Yan Luo 2, 3, Yuan Wang 1 1. Key Laboratory of Mesoscale Severe Weather/Ministry of Education and School of Atmospheric Sciences, Nanjing University, China 2. Environmental Modeling Center/NCEP/NWS/NOAA, College Park, Maryland, USA 3. I.M. Systems Group, Inc., College Park, Maryland, USA WWOSC 2014, Montreal, Canada, 21 August 2014 Evaluation of TIGGE ensemble predictions of Northern Hemisphere summer precipitation during 2008-2012

2. Outline Objectives The TIGGE Ensemble Prediction Systems (EPSs) data Evaluation results of TIGGE quantitative precipitation forecasts (QPFs) and probabilistic QPF (PQPFs) Summary the THORPEX Interactive Grand Global Ensemble (TIGGE)

3. Objectives Evaluate the QPF and PQPF performance of TIGGE EPSs Assess the performance change before and after major EPS upgrade Su et al. 2014, JGR

4. Evaluation of TIGGE QPFs and PQPFs Study period: 2008-2012 summer (June-August) Spatial coverage: Northern Hemisphere (NH) tropics (0-20°N) and midlatitude (20-49°N) Accumulation period: 24-h precipitation (12 UTC-12 UTC) Forecast lead time: 1-9 days Horizontal resolution: 1° ×1° (interpolation) 4 Observation data: TRMM 3B42 V7 (gauge adjustment) ECMWF portal http://tigge-portal.ecmwf.int/http://tigge-portal.ecmwf.int/

5. CenterBase time (UTC) members Horizontal resolution archived Fcst length (day) Initial perturb method Model uncertainty Major EPS upgrade time CMA (China) 00/1214+10.56º×0.56º0-10BVs-- CMC (Canada) 00/1220+11.0º×1.0º0-16EnKFPTP + SKEB multi-physics 17 Aug 2011 ECMWF00/1250+1N320 (~0.28º) N160 (~0.56º) 0-10 10-15 EDA- SVINI SPPT + SPBS9 Nov 2010 JMA1250+11.25º×1.25º0-9SVsSPPT17 Dec 2010 NCEP00/06/12/ 18 20+11.0º×1.0º0-16BV-ETRSTTP23 Feb 2010 UKMO00/1223+10.83º×0.56º0-15ETKFRP + SKEB9 Mar 2010 1. The CMC EPS was upgraded to version 2.0.2 on 17 August 2011. 2. The ECMWF EPS used a horizontal resolution of N200 (~0.45º) for 0-10 day forecasts and N128 (~0.7º) for 10-15 day forecasts before 26 January 2010. EVO-SVINI was used as the initial perturbation method before 24 Jun 2010. The SPBS method has been added on 9 November 2010. 3. The JMA EPS began to use the SPPT method on 17 December 2010. 4. The NCEP EPS was upgraded to version 8.0 and began to use the STTP method on 23 February 2010. In 14 February 2012, the NCEP EPS was upgraded to version 9.0. 5. The UKMO EPS used a horizontal resolution of 1.25º×0.83º before 9 March 2010. Referenced to the CMA EPS (frozen), the impacts of major model upgrades on the forecast performance are examined for other five EPSs. Configurations of six TIGGE EPSs

6. Verification methods of QPFs and PQPFs Area-weighted scores consider the latitude discrepancies Continuous scores: RMSE, Spatial correlation (SC), CRPSS, and spread-skill relationship Dichotomous scores: Bias, ETS, POD, FAR, BSS, attributes diagram (reliability curve and three decomposed terms of BS), ROC area and potential economic value (PEV) 6 where：w i =cos(lat) x i, y i are forecast and observation samples, N is the number of samples Su et al. 2014, JGR

7. Precipitation climatology (day +3) 7 JMA day +1 ensemble mean QPFs have large moist biases in the NH tropics Cause: JMA employs moist SVs over the entire tropics and perturbs the specific humidity with a large amplitude (Yamaguchi and Majumdar, 2010) TRMM JMA day 1 JMA NCEPCMA CMC ECMWF UKMO mm/day

8. Forecast error (RMSE) Control: dotted (…..) Ensemble mean: solid (——) Ensemble is better than control, especially in longer lead times EC ensemble best EC control is better than CMA ensemble in short lead times JMA control in NH midlatitude best

9. RMSE and frequency 9 Cause: JMA underestimates heavy rain It is not appropriate to use only RMSE to evaluate QPFs The control QPFs of JMA have the smallest RMSE in the NH midlatitude

10. Discrimination diagrams The discrimination ability decreases with the lead time EPSs have weak ability to discriminate heavy events In the NH tropics, CMA shows little discrimination ability among different rain events 10

11. Dichotomous scores of ensemble mean QPFs ECMWF best CMA very poor in the NH tropics 11

12. Spread-skill relationship The day +1 ensemble spread of JMA is the largest in the NH tropics CMC has the largest spread and it grows with the lead time: NH midlatitude: level with the ensemble mean error NH tropics: slightly overdispersive 12 Spread: dash (- - -), RMSE: solid (——)

13. PQPF error: CRPSS ECMWF best In the NH tropics, the day +1 CMA is even poorer than the day +9 ECMWF Skill of CMC rapidly drops from day +2 13

14. PQPF skill: BSS and ROC area Light rain: CMC best Heavy rain: CMC and ECMWF better CMA is very poor in the NH tropics 14

15. Attributes diagram (reliability curve, BSS, and BS terms) NH Midlatitute

16. Potential economic value Prob. thresholds of CMC are most reliable ECMWF has the highest PEV

17. Performance changes due to major EPS upgrade Spread: grey, RMSE: black Changes of spread, Spread/RMSE ratio are significant for CMC and ECMWF Change of RMSE is only significant for CMC (increase)

18. Performance changes due to major EPS upgrade Spread: grey, RMSE: black Changes of spread, Spread/RMSE ratio are significant for UKMO, NCEP, JMA Change of RMSE is only significant for UKMO (decrease)

19. ScoreCenterBeforeAfterChange SpreadCMC-CMA1.14.43.3 ECMWF-CMA-0.1-0.7-0.6 UKMO-CMA-0.4-0.20.2 NCEP-CMA-1.7-0.80.9 JMA-CMA-1.6-1.30.3 RMSECMC-CMA-0.4-0.10.3 ECMWF-CMA-0.7-0.8-0.1 UKMO-CMA-0.1-0.3 NCEP-CMA0-0.2 JMA-CMA-0.5-0.30.2 SpreadCMC-CMA0.200.590.39 /RMSEECMWF-CMA0.05-0.02-0.07 ratioUKMO-CMA-0.0500.05 NCEP-CMA-0.23-0.100.13 JMA-CMA-0.19-0.160.03 Changes due to major EPS upgrade Use the frozen CMA as the reference to eliminate the interannual variability CMC greatly increases spread and spread/RMSE ratio Similar performance changes for other lead times Changes due to major EPS upgrade of day +3 spread, RMSE, and Spread/RMSE ratio in the NH midlatitude (significant change with 95% confidence interval)

20. 20 Skill changes After EPS major upgrade: CMC: decreased skill ECMWF, UKMO, NCEP: improved JMA: no significant change

21. Wednesday February 13, 2013 Major upgrade to the Global Ensemble Prediction System (GEPS) version 3.0.0 at the Canadian Meteorological Centre “Changes installed uniquely into the forecast component include: adjustments to how physics tendencies are perturbed for convective precipitation; the physics tendencies perturbations are applied at every level except the very last one; addition of diffusion into the advection procedure; perturbation of the bulk drag coefficient in the orographic blocking scheme; and fine tuning of the adjustment factor alpha of the stochastic kinetic energy backscattering scheme.” http://collaboration.cmc.ec.gc.ca/cmc/CMOI/product_guide/docs/changes_e.html #20130213_geps 21 CMC EPS upgrade

22. Evaluation of the QPFs and PQPFs from six TIGGE EPSs in the NH midlatitude and tropics during the boreal summers of 2008-2012: Ensemble mean QPFs: CMA: large systematic biases, poor performance in the NH tropics ECMWF: less errors and best skill JMA: unusually large moist biases of day +1 QPFs in the NH tropics PQPFs: CMC: relatively good for light precipitation and short lead times, increased spread and larger errors for longer lead times; better reliability and reliable probabilistic thresholds in PEV ECMWF: best skill, except for light precipitation; best discrimination ability and highest potential economic benefit NCEP and UKMO: most sharp 22 Summary

23. The model upgrade in EPS cannot always guarantee skill improvements The enlarged ensemble spread of CMC forecasts after the upgrade increases the QPF errors Uncertainties and quality of verification data How to fairly evaluate an EPS is essential for the development and upgrade of the EPSs Comprehensive evaluation with multiple verification metrics Provide general guidance for the postprocessing of the EPSs Reliability and discrimination ability 23 Summary

24. E-mail: yuanhl@nju.edu.cn Thank you