1. Bill Campbell and Liz Satterfield JCSDA Summer Colloquium on Satellite Data Assimilation 27 Jul - 7 Aug 2015 Accounting for Correlated Satellite Observation Error in NAVGEM 1

2. 2 Why is Correlated Error Important? Dow Jones Industrial Average and the Subprime Mortgage Crisis Lehman Brothers file for bankruptcy 9/15/08 Peak of the Housing Bubble 10/12/07 Market hits bottom at less than 50% of its peak value 3/06/09

3. R Everything the data assimilation (DA) system knows about the error of every observation everywhere is contained in R Observation error is defined as the difference between a measured quantity and the truth, which is the true state of the continuous atmosphere discretized, mapped into model space, then mapped into observation space (via interpolation, radiative transfer, etc.) Observation error variance is the time mean squared observation error, which is not observed directly, but computed statistically Observation error covariance is the time mean of the product of the errors of different observations (e.g. a temperature measurement error in Fort Collins and a wind speed measurement error in Denver) All covariance matrices are symmetric and positive definite Covariance has physical units; correlation is normalized covariance, with values between -1 and 1 Observation error covariance matrix R Observation error covariance matrix R should contain the best estimates of variances on the diagonal, covariances everywhere else 3 Observation Error Defined

4. 4 Sources of Observation Error IMPERFECT OBSERVATIONS True Temperature in Model Space T=28°T=38°T=58° T=30°T=44°T=61° T=32°T=53°T=63° T=44° 1)Instrument error (usually, but not always, uncorrelated) H 2)Mapping operator (H) error (interpolation, radiative transfer) 3)Pre-processing, quality control, and bias correction errors 4)Error of representation (sampling or scaling error), which can lead to correlated error:

5. 5 Correlated Error in Operational DA R Until recently, most operation DA systems assumed no correlations between observations at different levels or locations (i.e., a diagonal R) To compensate for observation errors that are actually correlated, one or more of the following is typically done: – Discard (“thin”) observations until the remaining ones are uncorrelated (Bergman and Bonner (1976), Liu and Rabier (2003)) – Local averaging (“superobbing”) (Berger and Forsythe (2004)) – Inflate the observation error variances (Stewart et al. (2008, 2013) R Theoretical studies (e.g. Stewart et al., 2009 & 2013) indicate that including even approximate correlation structures outperforms diagonal R with variance inflation * In January, 2013, the Met Office went operational with a vertical observation error covariance submatrix for the IASI instrument, which showed forecast benefit in seasonal testing in both hemispheres (Weston et al. (2014))

6. 6 Error Covariance Estimation Methods There are four statistical methods we are aware of for observation error covariance estimation: Hollingsworth & Lönnberg (1986) (H-L), Fisher(2003), Desroziers (2005), and Daescu (2013) R H-L, Fisher (aka background error method), and Daescu can diagnose observation error variances (diag(R)), and H-L can diagnose vertical correlation, but none of these can diagnose horizontally-correlated observation error B The background error method assumes that the forecast error covariance matrix B is essentially correct, which is difficult to validate R RB Desroziers accounts for all types of correlated error, but the diagnosed R depends on the initial specification of the observation and forecast error covariance matrices (R and B, respectively) R The diagnosed R must be made symmetric and positive definite, and it may be necessary to adjust its eigenvalue spectrum to address convergence issues None of these methods is perfect

7. 7 Hollingsworth-Lönnberg Method (Hollingsworth and Lönnberg, 1986) Use innovation statistics from a dense observing network Assume horizontally uncorrelated observation errors Calculate a histogram of background innovation covariances binned by horizontal separation Fit an isotropic correlation model, extrapolate to zero separation to estimate the correlated (forecast) and uncorrelated (observation) error partition Correlation between observation error and forecast error: possibly induced by QC methods or bias correction methods Observation error variance Bias terms Extrapolate red curve to zero separation, and compare with innovation variance (purple dot) Mean of ob minus forecast (O-F) covariances, binned by separation distance u2u2 c2c2 Assumes no spatially-correlated observation error

8. 8 R From O-F, O-A, and A-F statistics from any model (e.g. NAVGEM), the observation error covariance matrix R, the representer HBH T, and their sum can be diagnosed R This method is sensitive to the R and HBH T that is prescribed in the DA system An iterative approach may be necessary R 1 R Diagnose R 1, which will be different from the original R R 1 Symmetrize R 1, possibly adjusting its eigenvalue spectrum R 1 Implement R 1 and run NAVGEM R 2 R true Diagnose R 2, which we hope will be closer to R true Desroziers Method (Desroziers et al. 2005)

9. 4DVar Primal Formulation Scale by B -1/2 4D-var iteration is on this problem -- We need to invert R!

10. 4DVar Dual Formulation Scale by R -1/2 C 4D-Var iteration is on this problem – No need to invert C! Iteration is done on partial step and then mapped back with BH T An advantage of the dual formulation is that correlated observation error can be implemented directly No matrix inverse is required, which lifts some restrictions on the feasible size of a non-diagonal R In particular, implementing horizontally correlated observation error is significantly less challenging

11. 11 Correlated Observation Error for the ATMS Microwave Sounder ATMS Microwave Sounder 13 temperature channels 9 moisture channels

12. Current observation error correlation matrix used for ATMS, and for ALL observations 12 Error Covariance Estimation for the ATMS Statistical Estimate Channel Number 4 5 6 7 8 9 10 11 12 13 14 15 18 19 20 21 22 Channel Number 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 4 5 6 7 8 9 10 11 12 13 14 15 18 19 20 21 22 Desroziers’ method estimate of interchannel portion of observation error correlation matrix for ATMS 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 4 5 6 7 8 9 10 11 12 13 14 15 18 19 20 21 22 Channel Number Current Treatment Channel Number Temperature Moisture

13. Current observation error correlation matrix used for ATMS, and for ALL observations 13 Iterating Desroziers Old Statistical Estimate Channel Number 4 5 6 7 8 9 10 11 12 13 14 15 18 19 20 21 22 Channel Number 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 4 5 6 7 8 9 10 11 12 13 14 15 18 19 20 21 22 Desroziers’ method estimate of interchannel portion of observation error correlation matrix for ATMS 4 5 6 7 8 9 10 11 12 13 14 15 18 19 20 21 22 Channel Number New Statistical Estimate Channel Number The change is not large, which is (weak) evidence that the procedure may converge

14. 14 Practical Implementation: What about Convergence? The condition number of a matrix X is defined by σ max (X)/σ min (X), which is the ratio of the maximum singular value of X to the minimum one (same as the eigenvalue ratio if X is symmetric and positive definite). It is a commonly-used barometer for the numerical stability of matrices. How to improve conditioning: 1.Increase the diagonal values (additively) of the matrix until the desired condition number is reached (e.g. Weston et al. (2014)). 2.Preconditioning 3.Find a positive definite approximation to the matrix such that the condition number is constrained.

15. Condition Number Constrained Correlation Matrix Approximation We want to find a positive definite approximation to the matrix is minimized. In the trace norm, simply set all of the smallest singular values equal to the value that gives the desired condition number, and then reconstruct the matrix with the singular vectors to obtain the approximate matrix. Another method, used by Weston et al. 2014, is to increase the diagonal values of the matrix until the desired condition number is reached. We believe there is better theoretical justification for the first method using the Ky Fan p-k norm (Tanaka, M. and K. Nakata, 2014) The Ky Fan p-k norm of where denotes the i th largest singular value of When p=2 and k=n, it is called the Frobenius norm; when p=1 and k=n, it is called the trace norm.

16. 16 Condition Number Constrained Correlation Matrix Approximation 0 0.05 0.1 0.15 0.2 Cond. Number = 18 Channel Number 5 7 9 11 13 15 19 21 Channel Number 0 0.05 0.1 0.15 0.2 5 7 9 11 13 15 19 21 Channel Number 5 7 9 11 13 15 19 21 Difference between original and conditioned correlation matrix Trace Norm Increase Diagonal

17. Convergence and the Cauchy Interlacing Theorem Cauchy interlacing theorem Let A be a symmetric n × n matrix. The m × m matrix B, where m ≤ n, is called a compression of A if there exists an orthogonal projection P onto a subspace of dimension m such that P*AP = B. The Cauchy interlacing theorem states:compression Theorem. If the eigenvalues of A are α 1 ≤... ≤ α n, and those of B are β 1 ≤... ≤ β j ≤... ≤ β m, then for all j < m + 1, Notice that, when n − m = 1, we have α j ≤ β j ≤ α j+1, hence the name interlacing theorem. Cauchy interlacing theorem Let A be a symmetric n × n matrix. The m × m matrix B, where m ≤ n, is called a compression of A if there exists an orthogonal projection P onto a subspace of dimension m such that P*AP = B. The Cauchy interlacing theorem states:compression Theorem. If the eigenvalues of A are α 1 ≤... ≤ α n, and those of B are β 1 ≤... ≤ β j ≤... ≤ β m, then for all j < m + 1, Notice that, when n − m = 1, we have α j ≤ β j ≤ α j+1, hence the name interlacing theorem. What happens when radiance profiles are incomplete (i.e., at a given location, some channels are missing, usually due to failing QC checks)?

18. 18 Conjugate Gradient Convergence Goal C

19. 19 Using the Desroziers Diagnostic for IASI Channel Selection Water vapor channels 2889, 2994, 2948, 2951, and 2958 have very high error correlation (>0.98) The eigenvectors corresponding to the 4 smallest eigenvalues project only on to these 5 channels It makes sense to use the Desroziers diagnostic to do a posteriori channel selection, which has the bonus of improving the condition number of the correlation matrix, and thus solver convergence

20. 20 Conjugate Gradient Convergence Goal

21. 21 The Desroziers error covariance estimation methods can quantify correlated observation error Minimal changes can be made to the estimated error correlations to fit operational time constraints After accounting for correlations, variances could potentially be reduced (assuming correct background) Correctly accounting for vertically correlated observation error in data assimilation has already yielded superior forecast results at multiple operational NWP centers without a large computational cost What about horizontally correlated error? Main Conclusions

22. Correlated Error RedSat 140 180 220 260 300 °K Uncorrelated Error WhiteSat 140 180 220 260 300 °K Truth (No Error) True Atmosphere Imagine two hypothetical satellite instruments looking down on Hurricane Sandy uncorrelated error WhiteSat sees a fuzzy version of the truth; current DA systems handle uncorrelated error well RedSat sees a warped version of the truth; current DA systems handle correlated error poorly High resolution (i.e. future) instruments are more likely to have correlated error, which can be much more subtle than this example 22 Visualizing Horizontally Correlated Observation Error Observation Error 140 180 220 260 300 °K

23. 23 How can we best estimate errors in Desroziers/Hollingsworth- Lönnberg diagnostics? – Should we expect agreement between different methods? – Will the Desroziers diagnostic converge if both R and B are incorrectly specified? – Amount of data required to estimate covariances? Seasonal dependence? – Best methods to symmetrize the Desroziers matrix? How to gauge improvement? – Do we also need to adjust to see overall improvement to the system? – How do we maintain the correct ratio for DA? What about convergence? – Should we do an eigenvalue scaling to improve the condition number? Discussion

24. 24References IMPERFECT OBSERVATIONS Hollingsworth, A. and P. Lonnberg, 1986. The statistical structure of short-range forecast errors as determined from radiosonde data. Part I: The wind field. Tellus, 38A, pp 111-136. Desroziers, G., et al., 2005. Diagnosis of observation, background and analyis-error statistics in observation space. Quart. J. Roy. Meteor. Soc., 131, pp. 3385-3396. Bormann, N. and P. Bauer, 2010: Estimates of spatial and interchannel observation-error characteristics for current sounder radiances for numerical weather prediction. I: Methods and application to ATOVS data. Quart. J. Roy. Meteor. Soc., 136, pp. 1036-1050. Gorin, V. E., and M. D. Tsyrulnikov, 2011. Estimation of Multivariate Observation-Error Statistics for AMSU-A data. Mon. Wea. Rev., 139, pp. 3765-3780. Stewart, L. M. et al., 2013. Data assimilation with correlated observation errors: experiments with a 1-D shallow water model. Tellus, 65. Weston, P. P. et al., 2014. Accounting for correlated error in the assimilation of high-resolution sounder data. Quart. J. Roy. Meteor. Soc., 140, pp. 2420-2429. Tanaka, M., and K. Nakata, 2014. Positive definite matrix approximation with condition number constraint. Optim. Lett., 8, pp. 939-947.

25. 25 We ran NAVGEM 1.3 at T425L60 resolution with the full suite of operational instruments for two months, from July 1, 2013 through Aug 31, 2013 R The control experiment (atid) used a diagonal R for the ATMS instrument R The initial ATMS experiment (atms) used the R diagnosed from the Desroziers method applied to three months of innovation statistics R The second ATMS experiment (atms2) used the R diagnosed from the Desroziers method applied to the first experiment Results from these experiments were neutral, but… Experimental Design

26. 26 Current Results with Proposed Scorecard

27. 27 We chose to assimilate only Channel 2889 (iasid_oneh20). This resulted in faster convergence of the 4DVar solver, as well as +1 on the FNMOC scorecard. R The initial IASI experiment (iasicor_oneh20) used the R diagnosed from the Desroziers method applied to three months of innovation statistics, and yielded +2 on the FNMOC scorecard. Experimental Design

28. 28 Current Results with Standard Scorecard

29. 29 Correlated Observation Error and the ATMS Advanced Technology Microwave Sounder (ATMS) 13 temperature channels 9 moisture channels

30. Correlated Error RedSat 140 180 220 260 300 °K Uncorrelated Error WhiteSat 140 180 220 260 300 °K Truth (No Error) True Atmosphere Imagine two hypothetical satellite instruments looking down on Hurricane Sandy WhiteSat sees a fuzzy version of the truth; current DA systems handle uncorrelated error well RedSat sees a warped version of the truth; current DA systems handle correlated error poorly Neither satellite is better than the other; they are simply different Many future instruments will have error characteristics like RedSat; some current ones already do R Nearly all major weather centers use a diagonal R that treats every satellite like WhiteSat, which is clearly wrong 30 Observation Error Visualized 140 180 220 260 300 °K

31. 31 Conjugate Gradient Convergence

32. 32 Conjugate Gradient Convergence

33. Assess Forecast Impact with Observation Sensitivity Tools Adjoint-based system (Langland and Baker, 2004) enables rapid assessment of changes to the DA system Jul01-Aug02, 2013 Add vertical correlated error to ATMS channels Add vertical correlated error to ATMS channels NAVDAS-AR Results