Huiling Yuan 1, Xiang Su 1, Yuejian Zhu 2, Yan Luo 2, 3, Yuan Wang 1 1. Key Laboratory of Mesoscale Severe Weather/Ministry of Education and School of Atmospheric Sciences, Nanjing University, China 2. Environmental Modeling Center/NCEP/NWS/NOAA, College Park, Maryland, USA 3. I.M. Systems Group, Inc., College Park, Maryland, USA WWOSC 2014, Montreal, Canada, 21 August 2014 Evaluation of TIGGE ensemble predictions of Northern Hemisphere summer precipitation during

Outline Objectives The TIGGE Ensemble Prediction Systems (EPSs) data Evaluation results of TIGGE quantitative precipitation forecasts (QPFs) and probabilistic QPF (PQPFs) Summary the THORPEX Interactive Grand Global Ensemble (TIGGE)

Objectives Evaluate the QPF and PQPF performance of TIGGE EPSs Assess the performance change before and after major EPS upgrade Su et al. 2014, JGR

Evaluation of TIGGE QPFs and PQPFs Study period: summer (June-August) Spatial coverage: Northern Hemisphere (NH) tropics (0-20°N) and midlatitude (20-49°N) Accumulation period: 24-h precipitation (12 UTC-12 UTC) Forecast lead time: 1-9 days Horizontal resolution: 1° ×1° (interpolation) 4 Observation data: TRMM 3B42 V7 (gauge adjustment) ECMWF portal

CenterBase time (UTC) members Horizontal resolution archived Fcst length (day) Initial perturb method Model uncertainty Major EPS upgrade time CMA (China) 00/ º×0.56º0-10BVs-- CMC (Canada) 00/ º×1.0º0-16EnKFPTP + SKEB multi-physics 17 Aug 2011 ECMWF00/1250+1N320 (~0.28º) N160 (~0.56º) EDA- SVINI SPPT + SPBS9 Nov 2010 JMA º×1.25º0-9SVsSPPT17 Dec 2010 NCEP00/06/12/ º×1.0º0-16BV-ETRSTTP23 Feb 2010 UKMO00/ º×0.56º0-15ETKFRP + SKEB9 Mar The CMC EPS was upgraded to version on 17 August The ECMWF EPS used a horizontal resolution of N200 (~0.45º) for 0-10 day forecasts and N128 (~0.7º) for day forecasts before 26 January EVO-SVINI was used as the initial perturbation method before 24 Jun The SPBS method has been added on 9 November The JMA EPS began to use the SPPT method on 17 December The NCEP EPS was upgraded to version 8.0 and began to use the STTP method on 23 February In 14 February 2012, the NCEP EPS was upgraded to version The UKMO EPS used a horizontal resolution of 1.25º×0.83º before 9 March Referenced to the CMA EPS (frozen), the impacts of major model upgrades on the forecast performance are examined for other five EPSs. Configurations of six TIGGE EPSs

Verification methods of QPFs and PQPFs Area-weighted scores consider the latitude discrepancies Continuous scores: RMSE, Spatial correlation (SC), CRPSS, and spread-skill relationship Dichotomous scores: Bias, ETS, POD, FAR, BSS, attributes diagram (reliability curve and three decomposed terms of BS), ROC area and potential economic value (PEV) 6 where：w i =cos(lat) x i, y i are forecast and observation samples, N is the number of samples Su et al. 2014, JGR

Precipitation climatology (day +3) 7 JMA day +1 ensemble mean QPFs have large moist biases in the NH tropics Cause: JMA employs moist SVs over the entire tropics and perturbs the specific humidity with a large amplitude (Yamaguchi and Majumdar, 2010) TRMM JMA day 1 JMA NCEPCMA CMC ECMWF UKMO mm/day

Forecast error (RMSE) Control: dotted (…..) Ensemble mean: solid (——) Ensemble is better than control, especially in longer lead times EC ensemble best EC control is better than CMA ensemble in short lead times JMA control in NH midlatitude best

RMSE and frequency 9 Cause: JMA underestimates heavy rain It is not appropriate to use only RMSE to evaluate QPFs The control QPFs of JMA have the smallest RMSE in the NH midlatitude

Discrimination diagrams The discrimination ability decreases with the lead time EPSs have weak ability to discriminate heavy events In the NH tropics, CMA shows little discrimination ability among different rain events 10

Dichotomous scores of ensemble mean QPFs ECMWF best CMA very poor in the NH tropics 11

Spread-skill relationship The day +1 ensemble spread of JMA is the largest in the NH tropics CMC has the largest spread and it grows with the lead time: NH midlatitude: level with the ensemble mean error NH tropics: slightly overdispersive 12 Spread: dash (- - -), RMSE: solid (——)

PQPF error: CRPSS ECMWF best In the NH tropics, the day +1 CMA is even poorer than the day +9 ECMWF Skill of CMC rapidly drops from day +2 13

PQPF skill: BSS and ROC area Light rain: CMC best Heavy rain: CMC and ECMWF better CMA is very poor in the NH tropics 14

Attributes diagram (reliability curve, BSS, and BS terms) NH Midlatitute

Potential economic value Prob. thresholds of CMC are most reliable ECMWF has the highest PEV

Performance changes due to major EPS upgrade Spread: grey, RMSE: black Changes of spread, Spread/RMSE ratio are significant for CMC and ECMWF Change of RMSE is only significant for CMC (increase)

Performance changes due to major EPS upgrade Spread: grey, RMSE: black Changes of spread, Spread/RMSE ratio are significant for UKMO, NCEP, JMA Change of RMSE is only significant for UKMO (decrease)

ScoreCenterBeforeAfterChange SpreadCMC-CMA ECMWF-CMA UKMO-CMA NCEP-CMA JMA-CMA RMSECMC-CMA ECMWF-CMA UKMO-CMA NCEP-CMA0-0.2 JMA-CMA SpreadCMC-CMA /RMSEECMWF-CMA ratioUKMO-CMA NCEP-CMA JMA-CMA Changes due to major EPS upgrade Use the frozen CMA as the reference to eliminate the interannual variability CMC greatly increases spread and spread/RMSE ratio Similar performance changes for other lead times Changes due to major EPS upgrade of day +3 spread, RMSE, and Spread/RMSE ratio in the NH midlatitude (significant change with 95% confidence interval)

20 Skill changes After EPS major upgrade: CMC: decreased skill ECMWF, UKMO, NCEP: improved JMA: no significant change

Wednesday February 13, 2013 Major upgrade to the Global Ensemble Prediction System (GEPS) version at the Canadian Meteorological Centre “Changes installed uniquely into the forecast component include: adjustments to how physics tendencies are perturbed for convective precipitation; the physics tendencies perturbations are applied at every level except the very last one; addition of diffusion into the advection procedure; perturbation of the bulk drag coefficient in the orographic blocking scheme; and fine tuning of the adjustment factor alpha of the stochastic kinetic energy backscattering scheme.” # _geps 21 CMC EPS upgrade

Evaluation of the QPFs and PQPFs from six TIGGE EPSs in the NH midlatitude and tropics during the boreal summers of : Ensemble mean QPFs: CMA: large systematic biases, poor performance in the NH tropics ECMWF: less errors and best skill JMA: unusually large moist biases of day +1 QPFs in the NH tropics PQPFs: CMC: relatively good for light precipitation and short lead times, increased spread and larger errors for longer lead times; better reliability and reliable probabilistic thresholds in PEV ECMWF: best skill, except for light precipitation; best discrimination ability and highest potential economic benefit NCEP and UKMO: most sharp 22 Summary

The model upgrade in EPS cannot always guarantee skill improvements The enlarged ensemble spread of CMC forecasts after the upgrade increases the QPF errors Uncertainties and quality of verification data How to fairly evaluate an EPS is essential for the development and upgrade of the EPSs Comprehensive evaluation with multiple verification metrics Provide general guidance for the postprocessing of the EPSs Reliability and discrimination ability 23 Summary

Thank you