1. Statistical Post Processing
Hong Guan
Bo Cui and Yuejian Zhu
EMC/NCEP/NWS/NOAA
Presents for NWP Forecast Training Class
March 31, 2015, Fuzhou, Fujian, China

2. 1. Background

3. North American Ensemble Forecast System (NAEFS)
International project to produce operational multi-center ensemble products
Bias correction and combines global ensemble forecasts from Canada & USA
Generates products for:
Weather forecasters
Specialized users
End users
Operational outlet for THORPEX research using TIGGE archive

4. Statement
The North American Ensemble Forecast System (NAEFS) combines state of the art weather forecast tools, called ensemble forecasts, developed at the US National Weather Service (NWS) and the Meteorological Service of Canada (MSC). When combined, these tools (a) provide weather forecast guidance for the 1-14 day period that is of higher quality than the currently available operational guidance based on either of the two sets of tools separately; and (b) make a set of forecasts that are seamless across the national boundaries over North America, between Mexico and the US, and between the US and Canada. As a first step in the development of the NAEFS system, the two ensemble generating centers, the National Centers for Environmental Prediction (NCEP) of NWS and the Canadian Meteorological Center (CMC) of MSC started exchanging their ensemble forecast data on the operational basis in September First NAEFS probabilistic products have been implemented at NCEP in February The enhanced weather forecast products are generated based on the joint ensemble which has been undergone a statistical post-processing to reduce their systematic errors.

5. Summary of 6th NAEFS workshop 1-3 May, 2012 Monterey, CA
6th NAEFS workshop was held in Monterey, CA during 1-3 May There were about 50 scientists to attend this workshop whose are from Meteorological Service of Canada, Mexico Meteorological Service, UKMet, NAVY, AFWA and NOAA.
Following topics have been presented and discussed during workshop:
Review the current status of the contribution of each NWP center to NAEFS
For each NWP center, present plans for future model and product updates, for both the base models and ensemble system (including regional ensembles)
Decide on coordination of plans for the overall future NAEFS ensemble and products (added variables, data transfer for increased resolution grids, FNMOC ensemble added to NAEFS, especially for mesoscale ensemble-NAEFS-LAM)
Learn about current operational uses of ensemble forecast guidance, including military and civilian applications.

6. 7th NAEFS Workshop in Montreal, Canada
Time: June 2014
Locations:
17-18 June – Biosphere, Montreal, Canada
19 June – CMC, Dorval, Canada
Co-chairs: Andre Methot and Yuejian Zhu
Topics (or sessions)
Status and plan of Global ensemble forecast systems;
Operational data management and distribution;
Ensemble verification and validation metrics;
Reforecast, bias correction and post process;
Regional ensemble and data exchange;
Wave ensembles;
Integration of ensemble in forecasts: user feedback and recommendation;
Products – hazard weather, high impact weather and diagnostic variables;
Open discussion of the NAEFS research, development, implementation and operation plan

7. NAEFS Current Status NCEP CMC NAEFS Updated: November 18th 2014 Model
GFS
GEM
NCEP+CMC
Initial uncertainty
ETR
EnKF
ETR + EnKF
Model uncertainty/Stochastic
Yes (Stochastic Pert)
Yes (multi-physics
and stochastic)
Yes
Tropical storm
Relocation
None
Daily frequency
00,06,12 and 18UTC
00 and 12UTC
Resolution
T254L42 (d0-d8)~55km
T190L42 (d8-16)~70km
About 50km
L72
1*1 degree
Control
Yes (2)
Ensemble members
20 for each cycle
40 for each cycle
Forecast length
16 days (384 hours)
16 days
Post-process
Bias correction
(same bias for all members)
for each member
Last implementation
February 14th 2012
November 18th 2014

8. Milestones Implementations Applications: Publications (or references):
First NAEFS implementation – bias correction Version May
NAEFS follow up implementation – CONUS downscaling Version December
Alaska implementation – Alaska downscaling Version December
Implementation for CONUS/Alaska expansion Version April
Implementation 2.5km/3km NDGD products for CONUS/Alaska Version 5.00 – August 2015
Applications:
NCEP/GEFS and NAEFS – at NWS
CMC/GEFS and NAEFS – at MSC
FNMOC/GEFS – at NAVY
NCEP/SREF – at NWS
Publications (or references):
Cui, B., Z. Toth, Y. Zhu, and D. Hou, D. Unger, and S. Beauregard, 2004: “ The Trade-off in Bias Correction between Using the Latest Analysis/Modeling System with a Short, versus an Older System with a Long Archive” The First THORPEX International Science Symposium. December 6-10, 2004, Montréal, Canada, World Meteorological Organization, P
Zhu, Y., and B. Cui, 2006: “GFS bias correction” [Document is available online]
Zhu, Y., B. Cui, and Z. Toth, 2007: “December 2007 upgrade of the NCEP Global Ensemble Forecast System (NAEFS)” [Document is available online]
Cui, B., Z. Toth, Y. Zhu and D. Hou, 2012: "Bias Correction For Global Ensemble Forecast" Weather and Forecasting, Vol
Cui, B., Y. Zhu , Z. Toth and D. Hou, 2013: "Development of Statistical Post-processor for NAEFS" Weather and Forecasting (In process)
Zhu, Y., and B. Cui, 2007: “December 2007 upgrade of the NCEP Global Ensemble Forecast System (NAEFS)” [Document is available online]
Zhu, Y, and Y. Luo, 2014: “Precipitation Calibration Based on Frequency Matching Method (FMM)”. Weather and Forecasting (doi:
Glahn, B., 2013: “A Comparison of Two Methods of Bias Correcting MOS Temperature and Dewpoint Forecasts” MDL office note, 13-1
Guan, H, B. Cui and Y. Zhu, 2014: “Improvement of Statistical Post-processing Using GEFS Reforecast Information”, Weather and Forecasting (in process)

9. NAEFS Statistical Post-Processing System
Bias correction:
Bias corrected NCEP/CMC GEFS and NCEP/GFS forecast (up to 180 hrs)
Combine bias corrected NCEP/GFS and NCEP/GEFS ensemble forecasts
Dual resolution ensemble approach for short lead time
NCEP/GFS has higher weights at short lead time
NAEFS products (global) and downstream applications
Combine NCEP/GEFS (20m) and CMC/GEFS (20m)
Produce Ensemble mean, spread, mode, 10% 50%(median) and 90% probability forecast at 1*1 degree resolution
Climate anomaly (percentile) forecasts
Wave ensemble forecast system
Hydrological ensemble forecast system
Statistical downscaling
Use RTMA as reference - NDGD resolution (5km/6km), CONUS and Alaska
Generate mean, mode, 10%, 50%(median) and 90% probability forecasts

10. NAEFS bias corrected variables
Last upgrade: April 8th (bias correction)
Variables
pgrba_bc file
Total 51
GHT
10, 50, 100, 200, 250, 500, 700, 850, 925, 1000hPa 10
TMP
2m, 2mMax, 2mMin, 10, 50, 100, 200, 250, 500, 700, 850, 925, 1000hPa 13
UGRD
10m, 10, 50, 100, 200, 250, 500, 700, 850, 925, 1000hPa 11
VGRD
VVEL
850hPa 1
PRES
Surface, PRMSL 2
FLUX (top)
ULWRF (toa - OLR)
Td and RH 2m
Notes
CMC and FNMOC do not apply last upgrade yet 10

11. NAEFS downscaling parameters and products
Last Upgrade: April (NDGD resolution)
Variables
Domains
Resolutions
Total 10/10
Surface Pressure
CONUS/Alaska
5km/6km
1/1
2-m temperature
10-m U component
10-m V component
2-m maximum T
2-m minimum T
10-m wind speed
10-m wind direction
2-m dew-point T
2-m relative humidity 11
All downscaled products are generated from 1*1 degree bias corrected fcst. globally
Products include ensemble mean, spread, 10%, 50%, 90% and mode

12. 2. NAEFS bias correction

13. NAEFS bias correction For any given forecast f , it could express as :
Where a is truth (or replaced as best analysis), e is systematic error and ℇ is random error
Therefore, systematic error (or accumulated bias) could be a time average difference of forecast and truth:
In fact, an accumulated bias is changed from time to time which means e is not exactly systematic error, and ℇ is not exactly random error based on the time window for average.

14. NAEFS Bias Correction
1). Bias Estimation: We assume a bias (b) for each lead-time (t) (6-hour interval up to 384 hours), each grid point (i, j) is defined as the different of best analysis (a) and forecast (f) at the same valid time (t0) which is up on latest available analysis.
2). Decaying Average (or Kalman Filter method): Average bias will be updated by considering prior period bias and current bias by using decaying average (or Kalman Filter method ) with weight coefficient (w).

15. NAEFS Bias Correction
3). Decaying Weight: Through many experiments for different weights (w = 0.01, 0.02, 0.05, 0.1 and etc…), and different parameters, and different lead times, overall, w equals to 0.02 has been used for GEFS bias correction which is mainly using past days information (see figure).

16. Simple Accumulated Bias
NAEFS Bias Correction
4). Bias corrected forecast: The new (or bias corrected) forecast (F) will be generated by applying decaying average bias (B) to current raw forecast (f) for each lead time, at each grid point, and each parameter.
Simple Accumulated Bias
Assumption: Forecast and analysis (or observation) is fully correlated

17. NAEFS Bias Correction
5). Performance: The performance is estimated by applying NAEFS bias correction method. The bias is calculated at each grid point for raw forecast (f) and bias corrected forecast (F), then using decaying average method (w=0.02) to get current average bias, taking absolute bias for each grid point, each lead time to generate domain average absolute error (bias) which smaller value is better (see figure: example for Northern Hemisphere 2 meter temperature, decaying average (w=0.2) about 2 months period ended by April 27, 2007).

18. Forecast becomes worse after bias correction
500hPa height: 120 hours forecast (ini: )
Shaded: left – raw bias right – bias after correction
Forecast becomes worse after bias correction

19. 2 meter temperature: 120 hours forecast (ini: 2006043000)
Shaded: left – raw bias right – bias after correction
Positive bias
Negative bias

20. Comparison of raw and bias corrected T2m for Summer and Fall 2011
RMS and Spread (NH – Fall)
RMS and Spread (NH – Summer)
Get worse from bias correction
ME and ABSE (NH – Summer)
ME and ABSE (NH – Fall)
Raw forecast is bias free
No carry on bias from summer

21. NAEFS Bias Correction Several questions left behind
The correlation of prior joint sample
Assume samples are fully correlated
The weight to calculate decaying average
Optimum weights are functions of geographic and forecast lead times (should be)
Currently, w (weight) is fixed (w=0.02)
Systematic error for seasonal
Current method is lagged for seasonal information
2nd moment adjustment for current method
Slightly adjust for CMC’s ensembles
N/A for single model ensemble

22. 3. Using ensemble reforecast

23. GEFS Reforecast Configurations (Hamill et al, 2013)
Model version
GFS v9.01 – last implement – May 2011
GEFS v9.0 – last implement – Feb. 2012
Resolutions
Horizontal – T254 (0-192hrs– 55km)
T190 ( hrs – 70km)
Vertical – L42 hybrid levels
Initial conditions
CFS reanalysis
ETR for initial perturbations
Memberships
00UTC - 10 perturbations and 1 control
Output frequency and resolutions
Every 6-hrs, out to 16 days
Most variables with 1*1 degree
Data is available
current
Using Reforecast Data
Bias over 24 years (24X1=24)
25 years (25x1=25)
Bias over 25 years and 31day window
(25x31)
Bias over recent 2, 5, 10, and 25 years
within a window of 31day
(2x31, 5x31, 10x31, 25x31)
Bias over 25 years with a sample interval
of 7days within a window of 31days
and 61days (~25x4 and ~25x8) .
1985
1986
2010
2009
day
day-15
day+15

24. Using 25-year reforecast bias (1985-2009) to calibrate 2010 forecast
Summer 2010
Solid line – RMSE
Dash line - Spread
Winter 2010
Spring 2010
Fall 2010

25. Using 24-year reforecast bias (1985-2008) to calibrate 2009 forecast
Spring 2009
Winter 2009
Decaying averages are not good except for day 1-2
2% decaying is best for all lead time
Another year
Summer 2009
Fall 2009
Decaying average is equal good as reforecast, except for week-2 forecast
Decaying averages are not good except for day 1-3

26. Reforecast bias (2-m temperature)
warm bias
cold bias
cold bias
Perfect bias

27. T2m calibration for different reforecast sample sizes
sensitivity on the number of training years (2, 5, 10, and 25 years)
sensitivity on the interval of
training sample (1 day and 7 days)
Long training period (10 or 25 years) is necessary to help avoid a large impact to bias correction from a extreme year case and keep a broader diversity of weather scenario!!
Skill for 25y31d’s running mean is the best. 25y31d’s thinning mean (every 7 days) is very similar to 25y31d’s running mean y31d’s thinning mean can be a candidate to reduce computational expense and keep a broader diversity of weather scenario!!!

28. Using reforecast to improve current bias corrected product
r could be estimated by linear regression from joint samples, the joint sample mean could be generated from decaying average (Kalman Filter average)
for easy forward.
Bias corrected forecast: The new (or bias corrected) forecast (F) will be generated by applying decaying average bias (B) and reforecast bias (b) to current raw forecast (f) for each lead time, at each grid point, and each parameter.
Additional term (bias from reforecast) is added if r (correlation coefficient) is not equal one. This adjustment is expected to benefit for longer lead time forecast

29. Using reforecast to improve current bias corrected product (24-hr forecast, 2010 )
Spring
Summer
Winter
Fall
Perfect bias

30. Using reforecast to improve current bias corrected product (240-hr forecast, 2010 )
Spring
Summer
Winter
Fall
Perfect bias

31. Using 25-year reforecast bias (1985-2009) to calibrate 2010 forecast
500hPa height
Winter 2010
Very difficult to improve the skills
Spring 2010

32. 4. 2nd moment adjustment

33. Make Up This Deficient ? Improving surface perturbations
Or Post processing
Bias correction does not change the spreads

34. 2nd moment adjustment 1st moment adjusted forecast 2nd moment adj.
R=1 if E=0
Ensemble skill
Estimated by decaying averaging
Ensemble spread

35. Almost Perfect!!! After 2nd moment adjustment Winter Spring Summer
Fall

36. Surface Temperature (T2m) for Winter (Dec. 2009 – Feb
Surface Temperature (T2m) for Winter (Dec – Feb. 2010), 120-hr Forecast
25% spread increased
SPREAD
7% RMSE reduced
RMSE
SPREAD/RMSE

37. Improving Under-dispersion for North America (Dec. 2009 – Nov. 2010)
Winter
Spring
Summer
Autumn
SPP2
RAW
RAW
Winter
Spring
Summer
Autumn
improving under-dispersion for all seasons
with a maximum benefit in summer.

38. 5. Multi-model ensemble

39. Bayesian Model Average
Weights and standard deviations for each model (k - ensemble member) at step j
Sum of (s,t) represents the numbers of obs.
Finally, the BMA predictive variance is
Between-forecast variance
Within-forecast variance
It is good for perfect bias corrected forecast,
Or bias-free ensemble forecast, but we do not

40. Courtesy of Dr. Veenhuis
Model 2
Model 1
Model 3
Courtesy of Dr. Veenhuis

41. Flow Chart of Recursive Bayesian Model Process (RBMP)
Bias free
Ens forecasts
Observation or
Best analysis
Err and sprd
Variance
Weights
Prior Err and sprd
Prior
weights
Prior
variance
BMA, Decaying process, and adjustment
New weights
New variance
New Err and sprd
(We thanks to Dr. Veenhuis for allowing us to adopt his BMA codes).
2nd moment adjustment
Adjusted PDF

42. T2m for summer and Fall 2014

43. The results demonstrate:
NUOPCIBC – simple combine three bias corrected ensembles
DCBMA – decaying based Bayesian Model Average
RBMP – Recursive Bayesian Model Process (built in decaying average and internal 2nd-moment adjustment)
Solid line – RMS error
Dash line - Spread
Over-dispersion
The results demonstrate:
BMA could improve 3 ensemble’s mean, but spread could be over if original spread is larger
RBMP could keep similar BMA average future, but 2nd moment will be adjusted internally
All important time average quantities are decaying average (or recursive – save storage)

44. ROC are better for all leads
Summer 2013

45. The results demonstrate:
NUOPCIBC – simple combine three bias corrected ensembles
DCBMA – decaying based Bayesian Model Average
RBMP – Recursive Bayesian Model Process (built in decaying average and internal 2nd-moment adjustment)
Solid line – RMS error
Dash line - Spread
Over-dispersion
The results demonstrate:
BMA could improve 3 ensemble’s mean, but spread could be over if original spread is larger
RBMP could keep similar BMA average future, but 2nd moment will be adjusted internally
All important time average quantities are decaying average (or recursive – save storage)

46. ROC are better for all leads
Slightly degradation
Fall 2013
Don’t understand this over-dispersion

47. U10m for summer and Fall 2014
Assume V10m has the similar statistics as U10m
The difference of error/skills is very smaller, it is very hard to identify the improvement.
Need to consider to have additional plots for difference of NUOPC and DCBMA and NUOPC and RBMA

48. RBMP is perfect for 1st moment and 2nd moment adjustment, especially for 2nd moment.

49. NH CRPS
NH ROC
Summer 2013
Tropical
Southern hemisphere

50. The same conclusion as summer Perfect for 2nd moment adjustment

51. NH CRPS
NH ROC
Tropical
Southern hemisphere

52. 6. Statistical downscaling

53. Statistical downscaling for NAEFS forecast
Proxy for truth
RTMA at 5km resolution
Variables (surface pressure, 2-m temperature, and 10-meter wind)
Downscaling vector
Interpolate GDAS analysis to 5km resolution
Compare difference between interpolated GDAS and RTMA
Apply decaying weight to accumulate this difference – downscaling vector
Downscaled forecast
Interpolate bias corrected 1*1 degree NAEFS to 5km resolution
Add the downscaling vector to interpolated NAEFS forecast
Application
Ensemble mean, mode, 10%, 50%(median) and 90% forecasts

55. 12hr 2m temperature forecast Mean Absolute Error (MAE)
GEFS bias-corr. & down scaling fcst.
GEFS raw forecast
12hr 2m temperature forecast
Mean Absolute Error (MAE)
w.r.t RTMA for CONUS
average for September 2007
NAEFS forecast

56. NCEP/GEFS raw forecast
4+ days gain from NAEFS
NAEFS final products
From Bias correction (NCEP, CMC)
Dual-resolution (NCEP only)
Combination of NCEP and CMC
Down-scaling (NCEP, CMC)

57. NCEP/GEFS raw forecast
8+ days gain
NAEFS final products
From: Bias correction (NCEP, CMC)
Dual-resolution (NCEP only)
Combination of NCEP and CMC
Down-scaling (NCEP, CMC)

58. From Valery Dagostaro (MDL)
GMOS forecast
CONUS 2m Temperature
For September 2007
Verify against RTMA
Verify against observation
From Valery Dagostaro (MDL)
NAEFS final products
From :
Bias correction (NCEP, CMC)
Dual-resolution (NCEP only)
Down-scaling (NCEP, CMC)
Combination of NCEP and CMC
BACK

59. 6. Statistical downscaling

60. Future NAEFS Statistical Post-Processing System
Probabilistic products at 1*1 (and/or) .5*.5 degree globally
NCEP
Bias correction for each ensemble member +
High resolution deterministic forecast
Reforecast
Mixed Multi- Model Ensembles
(MMME)
Others
Down-scaling (based on RTMA)
CMC
Smart
initialization
RBMP
For blender
Probabilistic products at NSGD resolution
(e.g. 2.5km – CONUS)
Auto-adjustment of 2nd moment
Varied decaying weights