1. How much information is hidden in residual spectra of DOAS fits
How much information is hidden in residual spectra of DOAS fits? And how can the information within the residual spectra be quantified?
Johannes Lampel1, Peter Lübcke2, Simon Warnach2, Udo Frieß2, Ulrich Platt2, Steffen Beirle1, Thomas Wagner1
1 Max Planck Institut for Chemistry, Mainz, Germany
2 Institute of Environmental Physics, Heidelberg, Germany

2. Questions / Motivation
Residual spectra are often ignored … or not?
Is there a measure to objectively quantify the „quality“ of a set of residual spectra?
Publications typically show ONE good-looking fit, what about the other data?
What is limiting us?
Noise test
‚Weird‘ effects, sub-optical structures (Odd/Even-structures, …)
How to apply Stutz&Platt 1998?

3. Total Information content
Information content (surprise value, self-information)
Total information content (Shannon-Entropy)
Extremes:
All but one pi=0 → no information!, E=0
All pi = const → maximum information (E=ln N)

4. Relative Information
Relative information content (Kullback-Leibler divergence)
Noise: qi = constant (here: for an infinite number of spectra)
For a finite set of spectra: later …
[Adler et al 2006]

5. Overview Origin of residual spectra Principal Component Analysis (PCA)
Basics
Applications
Information content
Conclusions

6. Origin of residual spectra
Residual spectra are a composition of
Photon shot noise
Instrumental
Noise (Gaussian, random)
Instabilities
Temperature, wavelength calibration, Offset, DC
Imperfections
Slit function: approximated? Constant in λ?
Nonlinearity of the detector / Problems of the electronics
Instrumental stray light, …
Missing or insufficiently known absorbers
Ignored RT effects, changing atmospheric conditions
… (optical or non-optical?)

7. Allan plots
If residual spectra were noise only, the relative RMS of the sum of n residuals would scale with n-1/2 due to Poisson statistics

8. Residual optical depth (low-pass filtered)
In Reality:
Residual optical depth (low-pass filtered)
Open Path CE-DOAS M91

9. In Reality:
(Synthetic data)

10. In Reality:
(Synthetic data)

11. How to proceed? If you know the reason for your problems:
Laboratory measurements
Theoretical calculations
multi-linear regression with known variables …
If you don‘t know the reason for your problems:
Reduce the dimensionality of your problem!

12. Principal Component Analysis (PCA)
Grey points: synthetic data, constructed using the blue and green vector.
PCA (cyan and red): „diagonal“ of data and orthogonal
ICA (yellow and purple): Under the assumption of non-gaussianity of the individual contributions the original vectors can be restored
PCA
ICA

13. Applications of PCA General: DOAS:
General surveillance, Face recognition, …
Quality control (welding processes, …) …
DOAS:
Ferlemann 1998 (Balloon-DOAS)
OMI SO2 retrieval (Li et al 2014), HCHO (Li et al 2015)
Detection of instrumental problems

14. Residual optical depth Open Path CE-DOAS M91
In Reality:
Residual optical depth Open Path CE-DOAS M91

15. Principal Component Analysis (PCA)
We have no previous knowledge
Cooking recipe:
Calculate covariance matrix Cij = < xi - μ | xj - μ > (diagonal entries are variances, off-axis entries are correlation coefficients)
Find a new base in which C is diagonal: C = T-1DT (no correlations any more)
Transform your residuals to this new base R‘ = TR
Discard unimportant parts based on Eigenvalues (D)
D is the connection to Information content

16. Toy Example Noise (RMS: 4.10-4) 100 channels
Systematic structures (RMS: ) (1-3x)

17. Eigenvalues
Noise (RMS):
Sys. Contr. (RMS)

18. Eigenvalues
Noise (RMS):
Sys. Contr. (RMS)

19. Eigenvalues
Noise (RMS):
Sys. Contr. (RMS)

20. Eigenvalues
Noise (RMS):
Sys. Contr. (RMS)

21. Standard Deviations (Not normalized)
N=3340, n=700

22. Total Information content
Information content (surprise value)
Total information content
Extremes:
All but one pi=0 → no information!, E=0
All pi = p → maximum information

23. Indep. variables
Number of residual spectra

24. Ground-based MAX-DOAS Examples: (M91, Peruvian Upwelling 2012, Acton 300i)
BrO-fit ( nm):
#channel: 357
DoF DOAS fit: 15
Information content: 257 -> 25% less than noise (20% less than randomized residuals)
IO-fit ( nm)
E: 10% less than noise
E: 5% less than randomized
(See also Poster #2)

25. 10-3

26. … and the same structure is found in Mainz, Peru, Antarctica, …

27. Ring
HCHO
NO2
10-4
RMS O4
Exp. Time

28. What did we learn? The structure between 332-358nm is
Independent of used O4 cross-section
Almost identical for different instruments
Observed at different latitudes
Causing residual structures of up to
„Something“ tropospheric (R=0.8 for O4)
This does not help much, but we know that there is potentially a problem to retrieve tropospheric BrO dSCDs around several 1013 molec/cm2.

29. But …
There is always some correlation with the Ring signal, but never really good (R=0.5)

30. (Small Ring signal!)

33. NOVAC SO2 Evaluation
Network of Scanning DOAS instruments at different Volcanoes
Alternative Evaluation: Fraunhofer Reference spectrum calculated from a solar Atlas

35. H2O vapour (614-683nm, without plant spectra)

36. Conclusions What do you do with your residual spectra?
PCA as a diagnosis tool
It provides spectral hints to remaining problems
Time series with information on optical density
(Relative) Information content as a measure to quantify the remaining residual structures
Questions:
Improvements (other decomposition algorithms)? Other applications?
What do you do with your residual spectra?

37. Thank you for your attention! Literature
(PCA) Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning, Volume 1. Springer New York,
(KLD) A. Adler, R. Youmaran, and S. Loyka. Towards a measure of biometric information. In Electrical and Computer Engineering, CCECE '06
(ICA) Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural Networks, 13: , 2000.

40.
2.10-2

41. Total information content
Example: Data compression „Mississippi“ -> 88 bit ASCII / 176 bit Unicode can be compressed to 21bit (24%)
(wikipedia)

42. In Reality:

43. Covariance Matrix

44. Eigenvalues
Noise (RMS):
Sys. Contr. (RMS)

45. Eigenvalues
Noise (RMS):
Sys. Contr. (RMS)

46. Eigenvalues
Noise (RMS):
Sys. Contr. (RMS)

47. Eigenvalues
Noise (RMS):
Sys. Contr. (RMS)