Filtering of variances and correlations by local spatial averaging Loïk Berre Météo-France

Outline 1. Contrast between two extreme approaches in Var/EnKF ? 2.The spatial structure of sampling noise and signal 3. Spatial averaging of ensemble-based variances 4. Spatial averaging of innovation-based variances 5. Spatial averaging of correlations

Two usual extreme approaches in Var/EnKF Covariance modelling :  Var: often globally averaged (spatially).  robust with a very small ensemble, but lacks heterogeneity completely.  EnKF: often « purely local ».  a lot of geographical variations potentially, but requires a rather large ensemble, and it ignores spatial coherences. An attractive compromise is to calculate local spatial averages of covariances.

Is the usual ensemble covariance optimal ?  Usual estimation (« raw »): B(N) = 1/(N-1)  i e b (i) e b (i) T = best estimate, for a given ensemble size N ?  No. Better to account for spatial structures of sampling noise and signal. Similar to accounting for spatial structures of errors in data assimilation, through B and R.

The spatial structure of sampling noise & signal in ensemble variance fields

Spatial structure of sampling noise (Fisher and Courtier 1995 Fig 6, Raynaud et al 2008a)  While the signal of interest is large scale, the sampling noise is rather small scale. True variance field V* ~ large scale  Sampling noise V e =V(N)-V* ~ large scale ? NO. N = 50 L(  b ) = 200 km

Spatial structure of sampling noise (Raynaud et al 2008b) Spatial covariance of sampling noise V e =V(N)-V* : V e (V e ) T = 2/(N-1) B* ° B* where B* ° B* is the Hadamard auto-product of B* =  b (  b ) T.  The spatial structure of sampling noise V e is closely connected to the spatial structure of background errors  b.

Spatial structure of sampling noise (Raynaud et al 2008b) Length-scale L( V e ) of sampling noise: L( V e ) = L(  b ) / 2  The sampling noise V e is smaller scale than the background error field  b … … which is smaller scale than the variance field V* of background error ? Correlation function of sampling noise: cor(V e [i], V e [j]) = cor(  b [i],  b [j])²

Spatial structure of signal (Houtekamer and Mitchell 2003, Isaksen et al 2007) Large scale features (data density contrasts, synoptic situation, …) tend to predominate in ensemble-based sigmab maps, which indicates that the signal of interest is large scale.

Spatial structure of sampling noise & signal (Isaksen et al 2007)  Experimental result : when increasing the ensemble size, small scale details tend to vanish, whereas the large scale part remains.  This indicates/confirms that the sampling noise is small scale, and that the signal of interest is large scale.  General expectation : increasing the ensemble size reduces sampling noise, whereas the signal remains. N = 10N = 50

« RAW »  b ENS #1 Differences correspond to sampling noise, which appears to be small scale. Common features correspond to the signal, which appears to be large scale. COMPARISON BETWEEN TWO “RAW”  b MAPS (Vor, 500 hPa) FROM TWO INDEPENDENT 3-MEMBER ENSEMBLES « RAW »  b ENS #2

Optimized spatial averaging of ensemble-based variances

“OPTIMAL” FILTERING OF THE BACKGROUND ERROR VARIANCE FIELD V b * ~  V b with  = signal / (signal+noise) Accounting for spatial structures of signal and noise leads to the application of a low-pass filter  (as K in data assim°):  (Raynaud et al 2008b)

(Raynaud et al 2008a) “OPTIMAL” FILTERING OF THE BACKGROUND ERROR VARIANCE FIELD SIMULATED « TRUTH »FILTERED SIGMAB’s (N = 6) RAW SIGMAB’s (N = 6)

RESULTS OF THE FILTERING (REAL ENSEMBLE ASSIMILATION)  b ENS 2 « RAW »  b ENS 2 « FILTERED »  b ENS 1 « FILTERED »  b ENS 1 « RAW »

LINK BETWEEN LOCAL SPATIAL AVERAGING AND INCREASE OF SAMPLE SIZE  The ensemble size N is MULTIPLIED(!) by a number Ng of gridpoint samples. If N=6 and Ng=9, then the total sample size is N x Ng = 54.  The 6-member filtered estimate is as accurate as a 54-member raw estimate, under a local homogeneity asumption. Ng=9 latitude longitude Multiplication by a low-pass spectral filter  Local spatial averaging (convolution)

DOES SPATIAL AVERAGING OF VARIANCES HAVE AN IMPACT IN THE (VERY) END ? with FILTERED  b with RAW  b obs-guessobs-analysis Ensemble Var at Météo-France (N=6; operational since July 2008) (Raynaud 2008)

Validation with spatially averaged innovation-based variances

Spatial filtering of innovation-based sigmab’s (Lindskog et al 2006) « Filtered » innovation-based sigmab’s « Raw » innovation-based sigmab’s  Some relevant geographical variations (e.g. data density effects), especially after spatial averaging.

Innovation-based sigmab estimate (Desroziers et al 2005) cov( H dx, dy ) ~ H B H T  This can be calculated for a specific date, to examine flow-dependent features, but then the local sigmab is calculated from a single error realization ( N = 1 ) !  Conversely, if we calculate local spatial averages of these sigmab’s, the sample size will be increased, and comparison with ensemble can be considered.

before spatial averaging after spatial averaging (with a radius of 500 km) Innovation-based sigmab’s « of the day » HIRS 7 (28/08/ h)  After spatial averaging, some geographical patterns can be identified. Can this be compared with ensemble estimates ? (Desroziers 2006)

Ensemble sigmab’s « Innovation-based » sigmab’s  Spatial averaging makes the two estimates easier to compare and to validate. Validation of ensemble sigmab’s « of the day » HIRS 7 (28/08/ h)

Spatial averaging of correlations

Spatial structure of raw correlation length-scale field Sampling noise : artificial small scale variations. (Pannekoucke et al 2007) RAW « TRUTH » N = 10 L(  b ) = 1/ (-2 d²cor/ds²) s=0

Raw ensemble correlation length-scale field « of the day » Geographical patterns are difficult to identify, due to sampling noise (N = 6).

Spatial structure of raw correlation length-scale field Reduction of small scale sampling noise, when the ensemble size increases. Sampling noise ~ relatively small scale.  Use spatial filtering. ex : wavelets. N = 5 N = 15 N = 30 N = 60 (Pannekoucke 2008)

Wavelet diagonal modelling of B (Fisher 2003, Pannekoucke et al 2007) It amounts tolocal spatial averages of correlation functions cor(x,s): cor W (x,s) ~  x’ cor(x’,s)  (x’,s) with scale-dependent weighting functions  : small-scale contributions to correlation functions are averaged over smaller regions than large-scale contributions.

Wavelet filtering of correlation functions Wavelet approach : sampling noise is reduced, leading to a lesser need of Schur localization. RAWWAVELET (Pannekoucke et al 2007) N = 10

Impact of wavelet filtering on analysis quality Wavelet approach : sampling noise is reduced, and there is a lesser need of Schur localization. (Pannekoucke et al 2007) N = 10 wavelet raw homogeneous Schur Length

Wavelet filtering of flow-dependent correlations Synoptic situation (geopotential near 500 hPa) (Lindskog et al 2007, Deckmyn et al 2005) Anisotropic wavelet based correlation functions ( N = 12 )

Wavelet filtering of correlations « of the day » Raw length-scales (Fisher 2003, Pannekoucke et al 2007) Wavelet length-scales N = 6

Conclusions  Spatial structures of signal and sampling noise tend to be different.  This leads to « optimal » spatial averaging/filtering techniques (as in data assimilation).  The increase of sample size reduces the estimation error (thus it helps to use smaller (high resolution) ensembles).  Useful for ensemble-based covariance estimation, but also for validation with flow-dependent innovation-based estimates.

Perspectives  Make a bridge with similar/other filtering techniques. ex : spatial BMA technique in probab. forecasting (Berrocal et al 2007). ex : ergodic space/time averages in turbulence. ex : local time averages of ensemble covariances (Xu et al 2008).  The 4D nature of atmosphere & covariance estimation may suggest 4D filtering techniques (instead of 2D currently).  There may be a natural path towards « a data assimilation/filtering » of ensemble- & innovation-based covariances, in order to achieve an optimal covariance estimation ?

Thank you for your attention !