1. 1 Information Content Tristan L’Ecuyer

2. 2 Historical Perspective Information theory has its roots in telecommunications and specifically in addressing the engineering problem of transmitting signals over noisy channels. Papers in 1924 and 1928 by Harry Nyquist and Ralph Hartley, respectively introduce the notion of information as a measurable quantity representing the ability of a receiver to distinguish different sequences of symbols. The formal theory begins with Shannon (1948), the first to establish the connection between information content and entropy. Since this seminal work, information theory has grown into a broad and deep mathematical field with applications in data communication, data compression, error-correction, and cryptographic algorithms (codes and ciphers). Claude Shannon (1948), “A Mathematical Theory of Communication”, Bell System Technical Journal 27, pp. 379- 423 and 623-656.

3. 3 Link to Remote Sensing Shannon (1948): “The fundamental problem of communication is that of reproducing at one point, either exactly or approximately, a message selected at another point.” Similarly, the fundamental goal of remote sensing is to use measurements to reproduce a set of geophysical parameters, the “message”, that are defined or “selected” in the atmosphere at the remote point of observation (eg. satellite). Information theory makes it possible examine the capacity of transmission channels (usually in bits) accounting for noise, signal gaps, and other forms of signal degradation. Likewise in remote sensing we can use information theory to examine the “capacity” of a combination of measurements to convey information about the geophysical parameters of interest accounting for “noise” due to measurement error and model error.

4. 4 Corrupting the Message: Noise and Non-uniqueness Measurement and model error as well as the character of the forward model all introduce non-uniqueness in the solution. ∆y ∆x Linear ModelQuadratic ModelCubic Model ∆x < ∆x ∆x < ∆x < ∆x Unwanted Solutions

5. 5 Forward Model Errors (∆y) Uncertainty due to unknown “influence parameters” that impact forward model calculations but are not directly retrieved often represents the largest source of retrieval error Errors in these parameters introduce non-uniqueness in the solution space by broadening the effective measurement PDF Forward ProblemInverse Problem “Influence” parameters Forward model errors Measurement error Errors in Inversion Uncertainty in “influence” parameters

6. 6 Error Propagation in Inversion Bi-variate PDF of (sim. – obs.) measurements. Width dictated by measurement error and uncertainty in forward model assumptions. R 0.64μm R 2.13μm σ TB σ ∆TB Obs. Error in product from width of posterior distribution from application of Bayes theorem. τ R eff στστ σ Reff Sol n

7. 7 Visible Ice Cloud Retreivals τ = 2 0.66 μm Reflectance 2.13 μm Reflectance τ = 10 20 30 50 8 μm 12 μm 24 μm 48 μm τ = 2 τ = 45±5; R e = 11±2 τ = 18±2; R e = 19±2 Due to assumptions: τ = 16-50; R e = 9-21 Nakajima and King (1990) technique based on a conservative scattering visible channel for optical depth and an absorbing near- IR channel for r eff Influence parameters are crystal habit, particle size distribution, and surface albedo.

8. 8 CloudSat Snowfall Retrievals Snowfall retrievals relate reflectivity, Z, to snowfall rate, S This relationship depends on snow crystal shape, density, size distribution, and fall speed Since few, if any of these factors can be retrieved from reflectivity alone, they all broaden the Z-S relationship and lead to uncertainty in the retrieved snowfall rate

9. 9 Snowfall Rate (mm h -1 ) Reflectivity (dBZe) Hex Columns 4-arm Rosettes 6-arm Rosettes 8-arm Rosettes Impacts of Crystal Shape (2-7 dBZ)

10. 10 Impacts of PSD (3-6 dBZ) ν = 0 ν = 1 ν = 2 Snowfall Rate (mm h -1 ) Reflectivity (dBZe) Sensitivity to ν Snowfall Rate (mm h -1 ) Reflectivity (dBZe) Sekhon/Srivastava a & b = -10% a & b = +10% Sensitivity to PSD Shape

11. 11 Implications for Retrieval Given a “perfect” forward model, 1 dB measurement errors lead to errors in retrieved snowfall rate of less than 10 % Ideal Case Reflectivity Snowfall Rate (mm h -1 ) “Reality” Reflectivity Snowfall Rate (mm h -1 ) PSD and snow crystal shape, however, spread the range of allowable solutions in the absence of additional constraint

12. 12 Quantitative Retrieval Metrics Four useful metrics for assessing how well formulated a retrieval problem: – S x – the error covariance matrix provides a useful diagnostic of retrieval performance measuring the uncertainty in the products – A – the averaging kernel describes, among other things, the amount of information that comes from the measurements as opposed to a priori information – Degrees of freedom – Information content All require accurate specification of uncertainties in all inputs including errors due to forward model assumptions, measurements, and any mathematical approximations required to map geophysical parameters into measurement space.

13. 13 Degrees of Freedom The cost function can be used to define two very useful measures of the quality of a retrieval: the number of degrees of freedom for signal and noise denoted d s and d n, respectively where S a is the covariance matrix describing the prior state space and K represents the Jacobian of the measurements with respect to the parameters of interest. d s specifies the number of observations that are actually used to constrain retrieval parameters while the d n is the corresponding number that are lost due to noise Clive Rogers (2000), “Inverse Methods for Atmospheric Sounding: Theory and Practice”, World Scientific, 238 pp. dsds dndn

14. 14 Degrees of Freedom Using the expression for the state vector that minimizes the cost function it is relatively straight-forward to show that where I m is the m x m identity matrix and A is the averaging kernel. NOTE: Even if the number of retrieval parameters is equal to or less than the number of measurements, a retrieval can still be under- constrained if noise and redundancy are such that the number of degrees of freedom for signal is less than the number of parameters to be retrieved.

15. 15 Entropy-based Information Content The Gibbs entropy is the logarithm of the number of discrete internal states of a thermodynamic system where p i is the probability of the system being in state i and k is the Boltzmann constant. The information theory analogue has k=1 and the p i representing the probabilities of all possible combinations of retrieval parameters. More generally, for a continuous distribution (eg. Gaussian):

16. 16 Entropy of a Gaussian Distribution For the Gaussian distributions typically used in optimal estimation we have: For an m-variable Gaussian dist.:

17. 17 Information Content of a Retrieval The information content of an observing system is defined as the difference in entropy between an a priori set of possible solutions, S(P 1 ), and the subset of these solutions that also satisfy the measurements, S(P 2 ): If Gaussian distributions are assumed for the prior and posterior state spaces as in the O. E. approach, this can be written: since, after minimizing the cost function, the covariance of the posterior state space is:

18. 18 Interpretation Qualitatively, information content describes the factor by which knowledge of a quantity is improved by making a measurement. Using Gaussian statistics we see that the information content provides a measure of how much the ‘volume of uncertainty’ represented by the a priori state space is reduced after measurements are made. Essentially this is a generalization of the scalar concept of ‘signal-to- noise’ ratio.

19. 19 Measuring Stick Analogy Information content measures the resolution of the observing system for resolving solution space. Analogous to the divisions on a measuring stick: the higher the information content, the finer the scale that can be resolved. A: Biggest scale = 2 divisions  H = 1 Full range of a priori solutions A C C: Finer still = 8 divisions  H = 3 B B: Next finer scale = 4 divisions  H = 2 D D: Finest scale = 16 divisions  H = 4

20. 20 Blue  a priori state space Green  state space that also matches MODIS visible channel (0.64 μm) Red  state space that matches both 0.64 and 2.13 μm channels Yellow  state space that matches all 17 MODIS channels Liquid Cloud Retrievals Prior State Space0.64 μm (H=1.20) LWP (gm -3 ) R e (μm) LWP (gm -3 ) R e (μm) 0.64 & 2.13 μm (H=2.51) 17 Channels (H=3.53)

21. 21 Snowfall Retrieval Revisited With a 140 GHz brightness temperature accurate to ±5 K as a constraint, the range of solutions is significantly narrowed by up to a factor of 4 implying an information content of ~2. Radar Only Reflectivity Snowfall Rate (mm h -1 ) Radar + Radiometer Snowfall Rate (mm h -1 ) Reflectivity

22. 22 Return to Polynomial Functions Order, NX1X1 X2X2 Error (%)dsds H 11.9841.988181.9331.45 21.9961.99891.9852.19 51.9992.00031.9983.16 σ y = 10% σ a = 100% σ y = 25% σ a = 100% σ y = 10% σ a = 10% Order, NX1X1 X2X2 Error (%)dsds H 11.4011.43280.5680.07 21.6821.77171.0990.21 51.9271.97631.7840.83 Order, NX1X1 X2X2 Error (%)dsds H 11.9091.929411.6590.65 21.9761.986211.9111.29 51.9961.99881.9872.25 X 1 = X 2 = 2; X 1a = X 2a = 1 1 2 5

23. 23 Application: MODIS Cloud Retrievals The concept of information content provides a useful tool for analyzing the properties of observing systems within the constraints of realistic error assumptions. As an example, consider the problem of assessing the information content of the channels on the MODIS instrument for retrieving cloud microphysical properties. Application of information theory requires: – Characterize the expected uncertainty in modeled radiances due to assumed temperature, humidity, ice crystal shape/density, particle size distribution, etc. (i.e. evaluate S y ); – Determine the sensitivity of each radiance to the microphysical properties of interest (i.e. compute K); – Establish error bounds provided by any available a priori information (eg. cloud height from CloudSat); – Evaluate diagnostics such as S x, A, d s, and H 1.L’Ecuyer et al. (2006), J. Appl. Meteor. 45, 20-41. 2.Cooper et al. (2006), J. Appl. Meteor. 45, 42-62.

24. 24 Error Analyses Fractional errors reveal a strong scene-dependence that varies from channel to channel. LW channels are typically better at lower optical depths while SW channels improve at higher values.

25. 25 Sensitivity Analyses The sensitivity matrices also illustrate a strong scene dependence that varies from channel to channel. The SW channels have the best sensitivity to number concentration in optically thick clouds and effective radius in thin clouds. LW channels exhibit the most sensitivity to cloud height for thick clouds and to number concentration for clouds with optical depths between 0.5-4. 0.646 μm 2.130 μm 11.00 μm

26. 26 Information Content Information content is related to the ratio of the sensitivity to the uncertainty – i.e. the signal-to-noise. H dsds 14 km11 km9 km 14 km11 km9 km

27. 27 The Importance of Uncertainties Rigorous specification of forward model uncertainties is critical for an accurate assessment of the information content of any set of measurements. Uniform 10% ErrorsRigorous Errors 11 km

28. 28 The Role of A Priori Information content measures the amount state space is reduced relative to prior information. As prior information improves, the information content of the measurements decreases. The presence of cloud height information from CloudSat, for example, constrains the a priori state space and reduces the information content of the MODIS observations. Without CloudSat With CloudSat 11 km