A unifying framework for hybrid data-assimilation schemes Peter Jan van Leeuwen Data Assimilation Research Center (DARC) National Centre for Earth Observation (NCEO) University of Reading

The solution is a pdf! Bayes theorem: Data assimilation: general formulation

Incorporating approximate pdfs: proposal densities We know the actual pdfs will be non-Gaussian. However, we can use any other density in Bayes theorem but will have to compensate: in which q(x) is the proposed density, so the Gaussian, and p(x) the real prior. 4DVar: the prior is Gaussian EnKF: prior assumed to be Gaussian Both: Observation errors are assumed to be Gaussian

A variational method looks for the most probable state, which is the maximum of the posterior pdf also called the mode. Instead of looking for the maximum one solves for the minimum of a so-called costfunction: The pdf can be rewritten as in which Find min J from variational derivative: J is costfunction or penalty function Variational methods

Issues with 4DVar 1) Prior is assumed Gaussian 2) Previous observations inform starting point, not B matrix 3) No posterior error estimate 4) No model errors 5) Pdf cannot be multimodal! 6) Mode of the wrong pdf!!! Possible pdf at time t=0

4DVar gives mode of wrong pdf Nonlinear model Mode at t=0 Evolved mode at end of window 4DVar not a natural method for forecasting!

Combine the two: Hybrid Methods 4EDnVKaFr: x x ? EnKF 4DVar x x

x x ETKF-4DVar x x x x x x An enormous effort to find a good B for the convective scale…

Which problem have we attacked? 1) Prior is assumed Gaussian 2) Previous observations inform mode, not B matrix 3) No posterior error estimate 4) No model errors 5) Posterior pdf cannot be multimodal! 6) 4Dvar gives mode of wrong pdf! 7) Extra linearity by replacing ensemble mean by 4Dvar solution 4DEnVar is computationally more efficient but solves non of the red issues above.

Ensemble of 4DVars x x x x x x x x x x x x Perturb observations

Which problem have we attacked? 1) Prior is assumed Gaussian 2) Previous observations inform mode, not B matrix 3) No posterior error estimate 4) No model errors 5) Posterior pdf cannot be multimodal! 6) 4Dvar gives mode of wrong pdf! 7) Extra linearity assumption by perturbing observations… Long-window 4DVar does not solve any of the red above.

Unifying framework: nonlinear filtering Use ensemble with the weights.

What are these weights? The weight is the normalised value of the pdf of the observations given model state. For Gaussian distributed variables is is given by: One can just calculate this value That is all !!! Or is it? More is needed for high-dimensional problems…

Standard Particle filter

Particle filter with proposal transition density

How to pull particles to observations? Ensure statistics is ok, we don’t want to change the problem… Use e.g. simple relaxation Use EnKF ? Use 4Dvar, 4DEnVar, EnKS ? (Plus Equivalent-weight-like step)

Use proposal densities: We use a different prior: Use this in Bayes: Generate ensemble from q(x) (e.g. EnKF, or ensemble of 4DVars):

Use proposal densities: Use this in Bayes: or with

Example: ‘4Dvar’ as proposal x x x x x x

Example: ‘4DVar’ as proposal x x x x x x Model errors are essential !

What problem did we attack? 1) Prior is assumed Gaussian 2) Previous observations inform mode, not B matrix 3) No posterior error estimate 4) No model errors 5) Posterior pdf cannot be multimodal! 6) 4Dvar gives mode of wrong pdf! 7) Extra linearity assumption by perturbing observations… B matrix not needed !!! We now need Q matrix !!!

Unifying framework: proposal densities for Bayes Theorem 4DVarNo posterior variance EnKFLast step: EnKSLast step as EnKF, all other steps ETKF-4DVarEnKF as above, followed by 4DVar: no 4DVar at t=0 4DEnVarSame as ETKF-4DVar Ens 4DVar no 4DVar at t=0 4DVar-PF4DVar with B=0, include Q, for each particle

Conclusions EnKF, EnKS, ensembles of 4DVars, 4DEnsVar, etc. can be viewed as proposal density samples in a particle filter: nonlinear hybrids! Weights will diverge if these are used as is. However, efficient schemes that control the weights are available Forces the community to work on model errors !!! Finally… Can reduce B matrix effort Opens the road to systematic model improvement