1. Developing 4D-En-Var: thoughts and progress
Ramble:
What is the purpose of an ensemble forecast? The dominant philosophy at the moment is that ensemble forecasts provide uncertainty information that supplements a deterministic forecast. In that sense an ensemble forecast is always the bridesmaid, but never the bride.
However, there are some murmurings in the ensemble and data assimilation communities that maybe one day we will no longer have deterministic forecasting systems. This may happen when global models reach the gap around 10km where modellers fear to tread. It may also happen at very high resolution where predictability is lost extremely quickly.
What I’m going to talk about today is a different criterion for making that transition – when an ensemble forecasting system provides a better deterministic forecast than can be achieved using a single model alone. Combining forecasts from models with different resolutions can already meet this criterion, so in some sense that day has already come, even though we still base our output on a single model run. On the way we also take in the more familiar poor-man’s ensembles. Only at the end of the talk do we return to genuine ensemble prediction systems, and using some of these ideas in that more traditional context.
Neill Bowler
Andrew Lorenc, Adam Clayton, Dale Barker … (and more…)
© Crown copyright Met Office

2. 4D-En-Var / 5D-En-Var (Lorenc, 2003, Liu et al., 2008)
What are we building?
4D-En-Var / 5D-En-Var (Lorenc, 2003, Liu et al., 2008)
© Crown copyright Met Office

3. Hybrid data assimilation – 4D-Var
4D-Var cost-function:
Now the background term is split into two (Wang et al., 2007):
Where
© Crown copyright Met Office

4. Hybrid data assimilation – 4D-En-Var
4D-Var hybrid
4D-En-Var hybrid
Tangent-linear model
Potentially 4-dimensional
3D hybrid
Benefit: Reduced cost (no tangent-linear)
Disadvantage: Tangent-linear implied from limited size ensemble
© Crown copyright Met Office

5. Using the DEnKF method
DEnKF (Sakov & Oke, 2008): First update the mean
Then update the ensemble perturbations
EnVar: First analyse for each member
Then post-process to get the analysis states
Run an assimilation for each member
Find the ensemble mean
Revert the ensemble perturbation back towards the forecast
© Crown copyright Met Office

6. Why build 4D-En-Var / 5D-En-Var?
© Crown copyright Met Office

7. Key reasons for choosing 4D-En-Var
DA and ensemble unified
Maintenance
Consistency of perturbations with DA (Berre et al., 2006)
Simultaneous processing of obs
All observations processed at same time, so do not have to choose order of observations
Model-space localisation
Localisation in model space – avoids issues with satellite radiances and localisation between observation points
Localisation in transformed variables – improved balance
Hybrid covariances
Can run DA and ensemble using a hybrid background-error covariance
Not too expensive
Not as cheap as some options, but we can still run many members
© Crown copyright Met Office

8. Using hybrid covariances
Berre et al (2006) indicated that ensemble setup should mimic analysis system
Kalman gain used in updating perturbations should be the same as used in updating analysis
Test using Lorenz ’95 40-variable model (identical twin)
Hybrid covariance matrix used in updating the control forecast
Use hybrid covariances in updating perturbations?
© Crown copyright Met Office

9. Using hybrid covariances
Obs error 1.0, observations every grid-point
5 ensemble members
Hybrid covariance used in control analysis
© Crown copyright Met Office

10. Where is our building at?
© Crown copyright Met Office

11. Initial coding 4D-En-Var has been coded in the Met Office VAR system
It includes the ability to do an ensemble of analyses
Adaptive inflation (used by MOGREPS) has been integrated with VAR code
A few flies:
There are still some bugs in some aspects of the code
Need to work on optimising performance
© Crown copyright Met Office

12. Can we afford it? Successful runs on 4 nodes of IBM Power6
Takes ~1h for 23 analyses (1 deterministic + 22 ensemble)
Approximately 20x cheaper than equivalent 4D-Var
Requires most of the memory of those 4 nodes (50 GB per node, requires around 155GB)
Increasing ensemble size and resolution increases these both
Can do analysis at lower resolution
© Crown copyright Met Office

13. Example fields
Run 4D-En-Var with 22m for a single analysis cycle (6Z on 1st Dec 2011)
Run with all observations (conventional + satellite)
80 iterations in solver
Compare with increments from the operational analysis (hybrid 4D-Var, 50% ensemble, 80% climatological)
Gives an idea of some of the differences between the two systems
© Crown copyright Met Office

14. Example increment: hybrid 4D-Var theta L15 (~850hPa)
© Crown copyright Met Office

15. Example increment: 4D-En-Var theta L15 (~850hPa)
© Crown copyright Met Office

16. Example increment: 4D-En-Var Hybrid theta L15 (~850hPa) (70% ensemble, 30% static)
© Crown copyright Met Office

17. Forecast ensemble spread MOGREPS theta L15 (~850hPa)
© Crown copyright Met Office

18. Analysis ensemble spread 4D-En-Var theta L15 (~850hPa)
© Crown copyright Met Office

19. Ratio of ensemble spread 4D-En-Var theta L15 (~850hPa)
© Crown copyright Met Office

20. Ratio of ensemble spread 4D-En-Var theta L15 (~850hPa) (70/30 hybrid)
© Crown copyright Met Office

21. Conclusion
Developed 4D-En-Var for use in data assimilation and ensemble
In testing and development phase
Optimise performance: improve convergence, parallelise minimisation
Compare with 4D-Var, using the same input data from the ensemble (will require a larger ensemble)
Test 4D-En-Var in ensemble mode, comparing with current ETKF
Develop an additive inflation scheme to represent model error
Test different methods of multiplicative inflation
© Crown copyright Met Office

22. Thanks for listening
© Crown copyright Met Office

23. References
Berre L, Stefanescu S, and Pereira M, The representation of the analysis effect in three error simulation techniques. Tellus, 58A: 196–209.
Greybush SJ, Kalnay E, Miyoshi T, Ide K, Hunt BR Balance and ensemble Kalman filter localization techniques, Mon. Wea. Rev. 139: 511–522
Liu C, Xiao Q, Wang B An ensemble-based four-dimensional variational data assimilation scheme. Part I: Technical formulation and preliminary test. Mon. Wea. Rev.136: 3363–3373
Lorenc AC, The potential of the ensemble Kalman filter for NWP - a comparison with 4D-Var. Q. J. Royal. Met. Soc., 129:
Sakov P, Oke PR A deterministic formulation of the ensemble Kalman filter: an alternative to ensemble square root filters. Tellus 60A: 361–371
Wang X, Snyder C, Hamill TM, On the theoretical equivalence of differently proposed ensemble-3DVar hybrid analysis schemes, Mon. Wea. Rev., 135:
© Crown copyright Met Office

24. Example increment: 4D-En-Var u-wind
© Crown copyright Met Office

25. Example increment: hybrid 4D-Var u-wind
© Crown copyright Met Office

26. Ratio of ensemble spread (smoothed, non-hybrid)
© Crown copyright Met Office

27. Ratio of ensemble spread (smoothed, 70/30 hybrid)
© Crown copyright Met Office

28. 4D-Var Hybrid MOGREPS-G 4D-Var Linear model Linear model 21Z 0Z 3Z 6Z
With a 4D-Var hybrid the ensemble forecast runs to the start of the next DA window, then provides information to the assimilation. 4D-Var uses a linear model to propagate this information through the time window. The resulting analysis is used to inform the starting conditions of the next ensemble forecast, which in turn provides uncertainty information to the assimilation as before. Note that it is the use of a linear model which greatly increases the cost of 4D-Var above other data-assimilation methods.
Linear model
Linear model
21Z 0Z 3Z 6Z 9Z
12Z
15Z
© Crown copyright Met Office

29. 4D-En-Var MOGREPS-G 4D-Ens-Var 21Z 0Z 3Z 6Z 9Z 12Z 15Z
4D-Ens-Var uses the fact that the ensemble forecast normally runs through the entire time of the next data assimilation window. Rather than providing uncertainty information only at the start of the DA window, information is provided throughout the entire window. This means that a linear model is no longer required to propagate the ensemble information through the window, making the assimilation much cheaper without great loss of accuracy. As before, once the analysis is produced this is used by the ensemble in producing the next set of forecasts, which are once again fed to the data assimilation.
This is an approach which has been tested with the Canadian ensemble, with promising results, and is being developed for use in America.
21Z 0Z 3Z 6Z 9Z
12Z
15Z
© Crown copyright Met Office

30. Ensemble 4D-En-Var / 5D-En-Var
4D-Ens-Var
EnKF
MOGREPS-G
4D-Ens-Var
At the Met Office we are planning to pursue the 4D-Ens-Var idea a step further. Even when 4D-Ens-Var is being used to perform the data assimilation it is normal to use an EnKF to create the ensemble perturbations. Although minor, this persists an inconsistency in the update of the ensemble perturbations and the data assimilation. We aim to develop a system where 4D-Ens-Var is used to update the ensemble perturbations, which one might call and ensemble of 4D-Ens-Vars. When using the same system to update the ensemble and data assimilation one should get the maximum benefit from the ensemble information.
21Z 0Z 3Z 6Z 9Z
12Z
15Z
© Crown copyright Met Office

31. Issue with observation localisation
Consider an idealised case (following Greybush et al, 2011)
With covariance localisation:
© Crown copyright Met Office

32. Issue with observation localisation
Covariance localisation
Observation localisation
© Crown copyright Met Office

33. Covariance localisation:
Observation localisation:
(a)
(b)
(a)+(b)
© Crown copyright Met Office

34. Using hybrid covariances
Obs error 1.0, observations every grid-point
5 ensemble members
Hybrid covariance used in control analysis
Hybrid does not help in EnSRF
© Crown copyright Met Office

35. What’s wrong with a square-root hybrid?
The update equation for the analysis-error covariance is
If the Kalman gain is the optimal choice
Then this simplifies to
Square-root filters assume the simplified form of the analysis error covariance
If we use a hybrid, then we are not using the optimal Kalman gain, and a square-root filter may not be appropriate
© Crown copyright Met Office