1. Some thoughts on error handling for FTIR retrievals Prepared by Stephen Wood and Brian Connor, NIWA with input and ideas from others...

2. With the possibility of archiving profiles in an NDSC data base there's a need to consider how we present the errors in these profiles. We need either some consistency in how this is done or a way of flagging how the errors that are archived were arrived at. There are a number of issues this raises. This presentation is aimed at outlining some of these, and hopefully promoting a lively discussion! Overview/Introduction

3. Recap of error terms 1)Measurement error: S meas =G y S ε G y T 2)Smoothing error: S smooth = (A – I) S x (A – I) T 3)Model parameter error: S par = G y ( ∑K b S b K b T ) G y T 4)Forward model error (from things not modelled properly in the forward model): difficult to quantify For reference Make the distinction between covariance values given to the retrieval S a,S e which “ define ” the retrieval and how it maps things, (i.e. K x, K b, G y, A), and then more realistic estimates of covariances like S x, the best estimate of covariance of x. This applies to all the error calculations

4. Expand on model parameter errors There are several model parameters which can be treated independently in error calculations, rather than lumping them all together in one vector b. Various b can be single parameters (e.g. a line strength, SZA) or vectors (e.g. temperature profile). Whether or not a model parameter has temporarily been put in the vector of retrieved values, the first step in error evaluation is calculating K b for a given b. If a given b can be put in as a retrieved variable then K b can often be calculated directly by the retrieval code. If this can ’ t be done, K b can be calculated from forward model runs with K b =[F(x,b+Δb)–F(x,b)]/Δb (but mention the catch with sfit2 normalisation)

5. Combining model parameter errors Once you have a K b (vector or matrix) then estimating the error involves estimating S b (single value or square matrix) and then calculating G y K b S b K b T G y T These contributions can then be summed over all b. While the various S b might be different dimensions, all the products K b S b K b T do have the same dimensions. Since the measurement error also has the same dependence on G y, a further possibility is to add all the K b S b K b T terms directly to an S ε (measurement covariance) and transform with G y to give a combined error that covers measurement and model parameter errors

6. a)Perturbation methods. Run the retrieval with some input changed by a typical amount, evaluate the change in the retrieval (difficult to generalise to multiple dimensions) b)Tools developed to aid with error evaluation. For sfit2, one is written in IDL (sfit2ers, error simulation) and there has been an equivalent MATLAB tool produced. These have taken the approach of evaluating some error terms for a typical case, the intention being to apply the results to all retrievals. There are limitations to this ●Not all errors are handled. Usually only measurement and smoothing error ●The quality of spectra may vary and so be different from the typical case ●The simulation of what the retrieval does may not be perfect How have we been doing error analysis and characterisation?

7. Batched error calculations We have tried one approach at Lauder for producing error and kernel calculations for each retrieval in a dataset. We run batches of sfit2 with K-files (contain K x, S e -1, S a -1 ) from each retrieval saved. We then have a post-processing tool (in IDL) that picks this information up and does kernel and error calculations. As it doesn’t handle all model parameter errors, this currently has no more capability than an error code built-in to sfit2 could have, but is more easily modified and extendable to evaluate more errors on individual retrievals and to match the differing requirements of various projects. It could read in information that quantified other errors to include in the calculations.

8. Which errors should be included in an archive? It is unrealistic to expect a full error analysis to be included with each retrieval in an archive. Such a complete analysis would include errors from A priori uncertainty (smoothing error) Random spectral noise (Measurement error) Interfering gases ILS and how it is modelled. Other instrumental effects (detector non-linearities, channelling, filter or background shape) Spectroscopic data (line strengths, widths) The assumed value of solar zenith angle Assumed pressure or temperature profiles the refraction and ray tracing calculations forward model errors Which of these should be included in an archived error analysis?

9. Our suggestion for deciding what errors to include The idea of not including smoothing error in archives is a good one, as long as the information to evaluate it or to apply similar smoothing to comparative data is included. One suggestion is to include only those errors that contribute random or uncorrelated errors to the measured spectrum and hence the retrieved profiles. This includes measurement noise, temperature uncertainties, interfering gases. Errors that produce fully correlated errors in the spectrum and are likely to produce a similar error in retrieved profiles. These errors could be characterised in metadata within the archive, giving likely magnitudes and effects on the profile, and not included in the error calculation with each retrieval. One example is line parameter errors.

10. Traps There are some easy mistakes that can be made consider a model parameter b, like an interfering gas, the error is S par = G y (K b S b K b T )G y T consider a model parameter b, like an interfering gas, the error is S par = G y (K b S b K b T )G y T Should S b is the variability covariance of b, or should be the remaining uncertainty (covariance) in b ? Should S b is the variability covariance of b, or should be the remaining uncertainty (covariance) in b ? For example: think about how T error is handled. Is the covariance needed the full variability in T or the uncertainty in the T profiles that are used? For example: think about how T error is handled. Is the covariance needed the full variability in T or the uncertainty in the T profiles that are used?

11. End

12. Specific problems/issues with sfit2ers Last year a number of bugs and issues with sfit2ers were found and fixed. Since then a few more have surfaced which have not yet been addressed point spacing dependence. For use as an evaluation tool, sfit2ers needs to be independent of having a measured spectrum and so sets its own point spacing, but this means it's not matching what sfit2 does with a real spectrum. The current point spacing is too dense and will be changed. Is the solution to have a switch to choose how point spacing is set? Evaluation of off-diagonal values of S x (climatological covariance) doesn't match the way S a is evaluated internally in sfit2. Input of non-diagonal S e doesn't work for multiple windows yet.

13. From Talk S. Wood 2007

14. Several error sources to consider & evaluate Several error sources to consider & evaluate An inclusive error analysis tool? – difficult An inclusive error analysis tool? – difficult An estimate for each retrieval? An estimate for each retrieval? Error evaluation might use different S to retrieval Error evaluation might use different S to retrieval Have considered errors as independent, but … Have considered errors as independent, but … Suggestion of a “ how-to ” guide and examples Suggestion of a “ how-to ” guide and examples Do data users also need a guide, an error “ tool ” ? Do data users also need a guide, an error “ tool ” ? Small group to produce a guide? discussion points from IRWG '06

15. How many errors is enough? Noise S meas =G y S ε G y T Noise S meas =G y S ε G y T FM model parameters. S par = G y ( ∑K b S b K b T )G y T FM model parameters. S par = G y ( ∑K b S b K b T )G y T ILS, SZA, other gases, line parameters... which? ILS, SZA, other gases, line parameters... which? are they fit routinely or fixed? consider both, but treat accordingly are they fit routinely or fixed? consider both, but treat accordingly Derivatives K b may be easy (from code) or difficult (finite difference of two FM runs). Derivatives K b may be easy (from code) or difficult (finite difference of two FM runs). Temperature – no derivatives in SFIT2, I have a finite difference method set up. Temperature – no derivatives in SFIT2, I have a finite difference method set up.

16. Decisions about archiving errors How much error analysis is desirable or adequate for archiving? How much error analysis is desirable or adequate for archiving? how much should errors be broken down in an archive or is it OK to just record one covariance? Perhaps 2, total random and total systematic?, in some case the difference is clear, not in others... how much should errors be broken down in an archive or is it OK to just record one covariance? Perhaps 2, total random and total systematic?, in some case the difference is clear, not in others... Perhaps a description field for errors Perhaps a description field for errors seems to be good support for leaving out smoothing error, but provide A, and x a so user can calculate it seems to be good support for leaving out smoothing error, but provide A, and x a so user can calculate it