1. 4D EnVar for the NCEP GFS Daryl Kleist NOAA/NWS/NCEP
Environmental Modeling Center
Global Climate and Weather Modeling Branch
Data Assimilation Team
With acknowledgements to John Derber (EMC), Dave Parrish (EMC), Jeff Whitaker (NOAA/ESRL), Kayo Ide (UMD), Ricardo Todling (NASA/GMAO), and many others
3 December 2013
NCEP Production Suite Review

2. GSI Hybrid [3D] EnVar (ignoring preconditioning for simplicity)
Incorporate ensemble perturbations directly into variational cost function through extended control variable
Lorenc (2003), Buehner (2005), Wang et. al. (2007), etc.
bf & be: weighting coefficients for fixed and ensemble covariance respectively
xt’: (total increment) sum of increment from fixed/static B (xf’) and ensemble B
ak: extended control variable; :ensemble perturbations
- analogous to the weights in the LETKF formulation
L: correlation matrix [effectively the localization of ensemble perturbations]
T: operator mapping from ensemble grid to analysis grid

3. Control Variable Example
Control variable for member 01 at z=15 (~850mb) at end of first outer loop for analysis valid at , assimilating all observations and using operational hybrid configuration.

4. Operational Implementation: 22 May 2012
Hybrid 3D EnVar/EnKF
80 member EnKF (T254L64)
75% Ensemble, 25% static for EnVar Deterministic update
Package included other changes
NPP ATMS (MW): 7 months after launch
This is by far the fastest we have ever begun assimilating data from a new satellite sensor after launch
GPS RO Bending Angle replaced Refractivity
Summary of pre-implementation retrospective testing
Improved Tropical winds
Improved mid-latitude forecasts
Fewer Dropouts
Improved fits to observations of forecasts
Some improvement in NA precip. in winter
Increased bias in NA precip. – decreased rain/no rain skill in summer (Improved by land surface bug fix)
Overall significant improvement of GFS forecasts

5. Dual-Res Coupled Hybrid Var/EnKF Cycling
Generate new ensemble perturbations given the latest set of observations and first-guess ensemble
member 1
forecast
member 1
analysis
T254L64
EnKF
member update
member 2
forecast
member 2
analysis
recenter analysis ensemble
member 3
analysis
member 3
forecast
Ensemble contribution to background error covariance
Replace the EnKF ensemble mean analysis
and inflate
GSI
Hybrid Ens/Var
high res
forecast
high res
analysis
T574L64
Previous Cycle
Current Update Cycle

6. 500mb Anomaly Correlation pre-implementation real time
HYBRID
3DVAR
HYBRID
3DVAR

7. 4D Data Assimilation
t-3
t=0
t+3
Observations are not necessarily taken at the analysis time, so what can we do?

8. Hybrid ensemble-4DVAR [H-4DVAR_AD]
Incremental 4DVAR: bin observations throughout window and solve for increment at beginning of window (x0’). Requires linear (M) and adjoint (MT) models
ADJ
TLM
Can be expanded to include hybrid just as in the 3DHYB case
ADJ
TLM
With a static and ensemble contribution to the increment at the beginning of the window

9. 4D-Ensemble-Var [4DENSV]
As in Buehner (2010), the H-4DVAR_AD cost function can be modified to solve for the ensemble control variable (without static contribution)
Where the 4D increment is prescribed exclusively through linear combinations of the 4D ensemble perturbations
Here, the control variables (ensemble weights) are assumed to be valid throughout the assimilation window (analogous to the 4D-LETKF without temporal localization). Note that the need for the computationally expensive linear and adjoint models in the minimization is conveniently avoided.

10. Hybrid 4D-Ensemble-Var [H-4DENSV]
The 4DENSV cost function can be easily expanded to include a static contribution
Where the 4D increment is prescribed exclusively through linear combinations of the 4D ensemble perturbations plus static contribution
Here, the static contribution is considered time-invariant (i.e. from 3DVAR-FGAT). Weighting parameters exist just as in the other hybrid variants. Again, no TLM or ADJ!

11. 4D EnVar: Way Forward Natural extension to operational EnVar
Uses variational approach in combination with already available 4D ensemble perturbations (covariance estimates)
No need for development of maintenance of TLM and ADJ models
Makes use of 4D ensemble to perform 4D analysis
Very attractive, modular, usable across a wide variety of applications and models
Highly scalable
And can be improved even further
Aligns with technological/computing advances
Computationally inexpensive relative to 4DVAR (with TL/AD)
Estimates of improved efficiency by 10x or more, e.g. at Env. Canada (6x faster than 4DVAR on half as many cpus)
Compromises to gain best aspects of (4D) variational and ensemble DA algorithms
Other centers pursuing similar path forward for deterministic NWP
Canada (replace 4DVAR), UMKO? (ensemble of 4D Ensemble Var)

12. Single Observation (-3h) Example for 4D Variants
TLMADJ
4DENSV
ENSONLY
H-4DVAR_AD
bf-1=0.25
H-4DENSV
bf-1=0.25
TLMADJ
ENSONLY

13. Time Evolution of Increment
TLMADJ
ENSONLY
Solution at beginning of window same to within round-off (because observation is taken at that time, and same weighting parameters used)
Evolution of increment qualitatively similar between dynamic and ensemble specification
** Current linear and adjoint models in GSI are computationally impractical for use in 4DVAR other than simple single observation testing at low resolution
t=-3h
t=0h
t=+3h
H-4DVAR_AD
H-4DENSV

14. Low Resolution GFS/GDAS Experiments with real observations
Basic configuration
T254L64 GFS, operational observations, GFS/GDAS cycles,
PR3LOEX0
3DVAR, 2x100 iterations, r25935 (EXP-ensvar-dev branch), operational “options”
PRHLOEX1
Hybrid 3D EnVar, 80 member T126L46 ensemble with fully coupled (two-way) EnKF update, slightly re-tuned localization and inflation for lower resolution, TLNMC on total increment, 75% ensemble & 25% static
PRH4DEX1
Hybrid 4D EnVar, TLNMC on all time levels, only 1x150 iterations (for now)
Hourly TC relocation, O-G, binning of observations (not 3-hourly)

15. 500 hPa Die Off Curves
Northern Hemisphere
Southern Hemisphere
4DHYB ----
3DHYB ----
3DVAR ----
4DHYB ----
3DHYB ----
3DVAR ----
4DHYB-3DHYB
3DVAR-3DHYB
Move from 3D Hybrid (current operations) to Hybrid 4D-EnVar yields improvement that is about 75% in amplitude in comparison from going to 3D Hybrid from 3DVAR.

16. 500 hPa Day 5 Time Series
Northern Hemisphere
Southern Hemisphere
--- 4DHYB
--- 3DHYB
--- 3DVAR
--- 4DHYB
--- 3DHYB
--- 3DVAR
Again, going from 3D to 4D-hybrid yields improvement at day 5 similar (not quite as large in SH) to what was seen in going from 3DVAR to 3D-hybrid

17. Extratropical Geop. Height RMSE Differences
Northern Hemisphere
Southern Hemisphere
3DVAR-3DHYB
3DVAR-3DHYB
4DHYB-3DHYB
4DHYB-3DHYB

18. Tropical Wind RMSE Differences
3DVAR-3DHYB
4DHYB-3DHYB
Although going to 4D yields consistent, uniform improvement to tropical wind forecasts, biggest gains were seen by adding the ensemble covariances (i.e. 3D Hybrid).

19. (Preliminary) Results and Comments
4D extension has positive impact on self analysis-based grid to grid verification (particularly in Southern Hemisphere)
Need to investigate (slight) degradation in NH stratosphere
Could be an analysis artifact
Need to dig more into fits to observations (including precipitation)
4D EnVar does have slower convergence (not shown)
As configured, 4D EnVar was 40% more expensive than 3D hybrid (caveats being different iteration count, low resolution and machine variability)
TLNMC over all time levels quite expensive
Lots of time spent moving files and performing I/O, could be optimized

20. Next Steps/Future Development
Configure update and outer loop
Have tested 2x100 configuration (neutral), but default “throws away” most of 4D increment
Investigate use of IAU to ‘massage’ hourly/4D increment into model
Quasi-outer loop (rerun of nonlinear model as in 4D Var?)
TLNMC over all time levels quite expensive
Since we have 4D increment, move toward weak constraint digital filter
Test already underway. Preliminary results suggests this may prove to be a very cost-effective alternative/supplement to TLNMC
OSSE-based results (not shown) suggest that a combined constraint may be the best compromise
Current configuration results in the ensemble spread having some undesirable characteristics (true for 3D Hybrid as well)
To compensate for lack of treatment of system error, inflating initial ensemble (results in spread decay, etc.).
Stochastic physics (GFS code ready, hope to include preliminary config in FY14 upgrade for 3D Hybrid)
Improve static error covariance estimate as not to “hinder” hybrid 4D EnVar
Alternative is larger ensemble or improved localization as not to need static B
FOTO?
Target for GDAS/GFS is potentially the FY15 upgrade

21. Backup Slides

22. Incremental Analysis Update
Figure Courtesy: Ricardo Todling

23. (4D) Incremental Analysis Update
In addition to TLNMC, we currently utilize full-field digital filter initialization in the GFS
Can have undesirable impacts such as filtering the semi-diurnal tide
IAU (Bloom et al.) has been used at GMAO and elsewhere as alternative to DFI
Increment is passed to model throughout the window as a forcing term
However, this has typically been done using a 3D increment (rescaled) through the window
4D EnVar lends itself to a minor modification of the IAU procedure to use the 4D increment
This also provides a mechanism for passing the 4D solution to the model
Perhaps a way to help spin up/spin down of clouds, precipitation, etc.

24. Figure Courtesy: ECMWF
Outer Loop for 4DVAR
Outer loop utilizing full (high resolution) non-linear model could be adapted to 4D EnVar. This has proven to be beneficial within the context of an EnKF (Quasi Outer Loop, Yang et al. 2012)
Figure Courtesy: ECMWF

25. Other areas of interest/Long term research goals
More intelligent choice of weighting between static/ensemble contributions to solution
Scale-dependence already being investigated
Quantitative, fully adaptive, 3D weights?
Unification of ensemble and deterministic updates
MLEF (Zupanski)
EVIL (Augline)
Configuring ensemble perturbations that are usable for both data assimilation and medium range ensemble prediction
Stochastic physics & treatment of model error will help us get there
Improved, adaptive localization within EnVar

26. Normal Mode Constraint for H-4DENSV
As in the operational hybrid 3DEnVar, a noise control mechanism such as the TLNMC (normal mode constraint) can be applied. First, consider the case where only the static contribution is filtered:
It is trivial (though computationally expensive) to expand this to the total increment over all (or any subset of) time levels
This explicitly filters out the portions of the increment that project onto gravity modes for all time levels.

27. Weak Constraint Digital Filter for H-4DENSV
Weak constraint digital filters are common practice for use with 4DVAR. A similar weak constraint can be formulated for use in this context:
Where xi denotes the filtered/initialized state at time m, where in this context is the middle of the window
The filtered state used in the penalty term is constructed using the typical digital filter coefficients (h) applied to the unfiltered states (xu)at each of the corresponding time levels (k)

28. Impact of Constraints on Single 4DENSV Analysis

29. Impact of Constraints on Single 4DENSV Analysis
Experiment
Total Tendency
Gravity Mode Tendency
Ratio
4DENSV
2.994e-3
2.884e-3
0.96
4DENSV+TLNMC
3.313e-4
2.029e-4
0.61
4DENSV+JCDFI
1.305e-3
1.279e-3
0.98
4DENSV+BOTH
8.874e-5
6.608e-5
0.74
H-4DENSV
1.791e-3
1.597e-3
0.89