1. OEM retrievals with IASI, AMSU and MHS: Final Presentation Summary of Study Findings 5 February R.Siddans, D. Gerber (RAL),

2. Purpose of the Study
To evaluate the benefit of adding microwave (MW) channels to the measurement vector of Eumetsat’s optimal estimation method (OEM) based scheme for retrieving temperature, humidity and ozone from the infra-red (IR) sounder IASI.
Eumetsat provide the description and input data of the baseline (IR-only) OEM scheme which is to be extended in the study.
The study should also extend the scheme
To fit surface spectral emissivity (IR and MW)
To work in the presence of (some) cloud (but not precipitation)
Additionally the impact of some specific AMSU channels (reflecting Metop- A performance) is to be studied

3. Study Tasks / Work Breakdown Schedule
KO in December 2013
WP1000 : Definition of the Sy matrix in the microwaves
Input via consultancy from Bill Bell (Met Office)
WP2000: OEM(MWIR/Metop-B) over ocean, clear sky
Set up IASI OEM to match EUMETSAT L2 PPF configuration
Run retrievals (IR and MWIR) and analyse residuals
WP3000 : OEM(MWIR/Metop-B) over land, clear sky, with fixed emissivities
WP4000 : OEM(MWIR/Metop-B) over land, clear sky, with variable emissivities
WP5000 : OEM(MWIR/Metop-B) in partial or full cloudy IFOVs
WP6000 : Retrievals with one or more missing AMSU channels
WP7000 : Delivery of datasets and final reporting

4. AMSU/MHS Channels AMSU-A # Freq.GHz 1 23.8 2 31.4 3 50.3 4 52.8 5 53.6
1 23.8
2 31.4
3 50.3
4 52.8
5 53.6
6 54.4
8 55.5
15 89
MHS
# Freq/ GHz
1 89
2 150

5. IASI/AMSU / MHS View Geometry
All instruments have ~circular FOV
IASI has 4 detectors each with 12km diameter footprint (at nadir)
AMSU + MHS have 48 and 16km diameter footprints (respectively)
Positions systematically co-located wrt each other across track.
Eumetsat provided nearest neighbour co-located observations from all 3 sensors ( based on IASI spatial sampling)

6. Overview of the Eumetsat IASI scheme
OEM solves under-constrained inverse problem using a prior estimates of the parameters to be retrieved, yielding the most likely solution given the characterised errors on the measurements and the prior estimate. This is achieved by solving the cost function:
Where
x is state vector (parameters to be retrieved)
y is measurement vector (a subset of IASI PC re-constructed / filtered radiances), with errors characterised by covariance Sy
F(x) is forward model which predicts measurements given state (RTTOV v10.2)
xa is the a priori state, which is assumed to have error characterised by Sa
The cost function is minimised using iterative approach (Newtonian)
K is the weighting function matrix – derivatives of F(x) with respect to x

7. Overview of the Eumetsat Scheme: A priori
A priori state (and first guess) from separate statistical retrieval which uses selected IASI, AMSU and MHS measurements as predictors
The relationship between the predictors and the state is derived using a the piece-wise linear regression (PWLR) scheme, training 12 days of measurements against co-located ECMWF analyses
The state is expressed in terms of principle components of the covariance of the PWLR profiles against the analyses.
28 principle components are used to represent temperature, 18 for humidity and 10 for ozone.
The OEM retrieves the weights of each of these profile principle components (+surface temperature)
Temperature is retrieved in K, humidity and ozone in ln(ppmv)
The use of PWLR as prior, means the prior state is already rather good.
The PWLR is also relatively insensitive to cloud (it uses MW observations & can exploit statistical relationships between predictors/state.
-> It is a challenge for the OEM to be “better” than PWLR

8. Overview of the Eumetsat Scheme: Measurements
Observations in 139 IASI channels are used (selected via trade-off of information content vs processing speed).
These are obtained from PC compressed L1 data.
To suppress instrument artefacts, reconstruction of radiances from the PC scores is followed by a projection onto the forward model subspace as defined by a transform provided by Eumetsat
See Tech.Note “Canonical angles between the IASI Observation and forward model sub-spaces”, Tim Hultberg, Thomas August
Measurement covariance is estimated using differences between observations and FM simulations based on ECMWF analyses (for sub-set of days)
A bias correction spectrum, prescribed as a function of view zenith angle is also applied to the “double-filtered” measurements

9. Example measurement / fit residuals from IR OEM
Metop B
Mid-latitude ocean
17 April 2013

10. IR only retrieval diagnostics (mid-lat)

11. Task 1: Definition of the Sy matrix for the MW channels
Literature review of use of AMSU+MHS in OEM
Dr William Bell (Met Office) consultant to consortium to provide expertise on use of AMSU+MHS
Met Office will provide estimates of the NEDT for all of the AMSU-A / MHS channels, as well as the observation covariances used in the operational assimilation of ATOVS radiances
In addition we estimate statistics (bias and covariances) of the AMSU/MHS departures from the provided IASI OEM and piece-wise linear regression (PWLR) profiles.
Currently measurement covariances are based on differences between MW observations and those computed using the IASI OEM retrieved state

12. Comparison to Met Office and ECMWF estimated errors
Met Office estimates
Study results from
Observation – simulations from IR only retrieved state
ECMWF estimates

13. Observation coveariance derived from MW residuals from IASI retrieval

14. Task 2: OEM (MWIR/Metop-B) over ocean, clear-sky
Task 3: OEM (MWIR/Metop-B) over land clear-sky
Implement the IASI PPF OEM settings in RAL code
Apply scheme to selected days of IASI and AMSU/MHS cloud-free data over ocean/land, to generate results for IR only and MW+IR (MWIR).
Days selected: 17 April, 17 July, 17 October 2013 (Metop B)
Evaluate results using the diagnostics such as:
PWLR, OEM(IR), OEM(MWIR) cf reference profiles (ECMWF analysis)
vertical profiles of bias, standard deviation
histograms and scatter plots for selected pressure levels
maps of departures
DOFS, AKs, fit residuals of OEM(IR) cf OEM(MWIR)
RTTOV internal emissivity atlases (TELSEM/CNRM/Wisconsin) + sea model used
Eumetsat cloud, precipitation and sea-ice masking used

16. Summary of changes in DOFS adding MW channels

17. Standard ir-only; mid-lat sea example

18. E.g. orbit cross-section: Temperature

19. Statistical assessment of retrievals
Based on comparing retrieval to analysis (ANA), Eumetsat retrieval (ODV), PWLR and analysis smoothed by averaging kernel (ANA_AK):
x’ = a + A ( t - a )
Where a is the a priori profile from the PWLR, t is the supposed "true", A is the retrieval averaging kernel
Profiles smoothed/sampled to grid more closely matching expected vertical resolution (than 101 level RTTOV grid), to avoid spurious structures:
Temperature: 0, 1, 2, 3, 4, 6, 8, 10, 12, 14, 17, 20, 24, 30, 35,40,50 km.
Water vapour: 0, 1, 2, 3,4, 6, 8, 10, 12, 14, 17,20 km
Ozone: 0, 6, 12, 18, 24, 30, 40 km.
The grid is defined relative to the surface pressure / z*.
Profile results further summarised (for maps, tables) into 3 layers
BL: 0-2 km z* (above surface)
LT: 0-6 km
UT: 6-12 km
The mean value of individual profiles taken over these ranges, then stats calculated (summarises bias over these layers)

20. MWIR vs IR Temperature Std.deviations higher over land
Retrieval slightly better than PWLR especially over land
Except retrieval biased near surface
This varies with surface type/day/night, maybe analysis/sampling error
No obvious benefit of MW – agreement with Ana x AK degrades, but this is more a reflection of increased sensitivity

21. MWIR vs IR Humidity Std.deviations much higher over land, near surface
PWLR better than retrieval near ground over land, but not in UT
Bias structure largely resolved by averaging kernels (effect of steep gradient)
Again, no obvious benefit of MW + agreement with Ana x AK degrades

22. MWIR vs IR-only
Use of “climatological prior” instead of PWLR gives more obvious benefit of MW channels even in cloud-free scenes…
Temperature
Water vapour
PWLR prior
Climatological prior
However basic performance degraded, especially near-surface H2O over land

23. Task 4: Addition of surface emissivity to state vector
Basic approach
Emissivity included in state vector in terms of principle components: Eigenvectors of global covariance from RTTOV atlases (for the 3 days assessed in the study)
A priori covariance is diagonal, filled with corresponding Eigenvalues
Have co-located spectra for MW+IR so can include correlations between MW and IR in the prior constraint
Land and sea (and permanent land ice) spectra included in the covariance, so retrieval should ~work around coast.
IR Atlas based on Wisconsin principle components of natural materials spectral patterns defined (at 416 wavelengths from cm-1) but
Only leading 6 patterns used (limit from MODIS channels)
Spectral shapes of further patterns needed to explain IASI observations, but no measure of their occurrence globally is available to define the prior constraint for these.
Other patterns included by deriving residual patterns not explained by RTTOV-atlas based patterns; Shift of mean emissivity also fitted.

24. Surface emissivity Eigenvectors and values fitted
(including MW correlations).

25. IASI/AMSU/MHS Study – emissivity ret(desert)
MWIR Emis = RTTOV
MWIR Emis: Ret emissivity
Original scheme Fitted emissivity Fitted emissivity + bias correction

26. Fitted emissivity cf RTTOV day vs night

27. Surface Temperature - Analysis
Stability of retrieval tested by setting prior emissivity to 1 (no spectral dependence)
Retrieved emissivity only slightly affected; bias slightly high, with compensating change in surface T, but impact on fitted profiles is very small.
A priori emissivity=RTTOV
A priori emissivity=1
0.5K change in mean surface T
(no change in std.dev)

28. MWIR with emissivity vs standard IR OEM
No significant change to T, but LT water vapour improves when emissivity fit…
Temperature
Water vapour
PWLR prior
Climatological prior
This is true also for IR-only retrieval; benefit of MW small

29. MWIR vs IR-only (no emissivity fit)
Temperature
Water vapour
PWLR prior
Climatological prior

30. Water vapour maps: Emissivity benefit…
MWIR; with emissivity
MWIR; no emissivity

31. Task 5: OEM(MWIR/Metop-B) in partial or full cloudy IFOVs
Benefit of MW mainly expected to be in allowing OEM to function in (partially) cloudy scenes, once cloud impact on IR accounted for in OEM.
Two approaches implemented in study to account for cloud:
McNally Watts approach used to identify cloud, then channels ignored in retrieval, depending on their vertical sensitivity
RTTOV’s simple black-body cloud used in FM, cloud height and fraction retrieved.
Results assessed using Eumetat L2 cloud parameters to segment comparison with analysis/PWLR.

32. McNally – Watts Method
Approach to cloud-screen individual spectral channels so some (high altitude sensitive) channels can still be used in presence of (lower) cloud
Instead of identifying cloud signatures in the measurements directly, the method tries to identify the altitude range in each channel for which a given measurement is not affected by cloud (i.e. indistinguishable from cloud-free case)
Method described in
McNally, A. P., Watts, P. D., A cloud detection algorithm for high-spectral- resolution infrared sounders, Q. J. R. Meteorol. Soc. (2003), 129, pp. 3411– 3423
In the following work we label the McNally – Watts method as “WMC” (to avoid confusion with the MW acronym)
Here we implement exactly the scheme as provided by NWP SAF – however there are several tunable parameters (not optimised here)

33. Performance of OEM Cloud Retrieval
Alternative to WMC: Retrieve fraction and height of cloud described using RTTOVs black-body cloud model.
Retrieved cloud fraction consistent with IR picture (incl. feathered edges)
Estimated error due to cloud indicates lack of info from IR
Retrieved cloud height consistent with cloud fraction (5km background value from a priori)
Retrieved TSurface under cloud bank reasonable under cloud
Retrieved cloud parameters found to compare quite well to operational L2. WMC estimated “height” much higher.

34. MW+IR with emissivity
Ignore cloud
MW+IR with emissivity
+McNally/Watts
MW+IR with emissivity
+ retrieve cloud
MW+IR with emissivity
+ retrieve cloud
(climatological prior)
Either approach to cloud strongly mitigates T errors below cloud, giving results comparable / better than PWLR
PWLR

35. Ret-Analysis BL Temp: Ignore cloud

36. Ret-Analysis BL Temp: Fit cloud

37. Ret-Analysis BL Temp: McNally Watts

38. Summary stats for Lower Tropospheric Temperature, over land as function of L2 cloudiness flag
ESD/K
DOFS
Ret-Ana PWLR-Ana
Mean difference /K
Standard OEM
Ret-Ana PWLR-Ana
Std.dev./K
Ignore cloud IR only
Ignore cloud MWIR
WMC
IR only
WMC
MWIR
Retrieve cloud
IR only
Retrieve cloud
MWIR
Only retrievals
with cost < 500
N. Cases
/1000
Cost

39. Summary stats for Lower Tropospheric Humidity, over land as function of L2 cloudiness flag
Standard OEM
Ignore cloud IR only
Ignore cloud MWIR
WMC
IR only
WMC
MWIR
Retrieve cloud
IR only
Retrieve cloud
MWIR
Only retrievals
with cost < 500
N. Cases
/1000
Cost
ESD/%
DOFS
Ret-Ana PWLR-Ana
Mean difference /%
Ret-Ana PWLR-Ana
Std.dev./%

40. Cloud summary When cloud is ignored:
Cost similar for cloud mask 1 or 2. However there is an increased negative bias in Retrieval – Analysis temperature for mask 2 cases.
Cost+ temperature bias increases greatly for cloud mask 3 + 4
MW provides a useful test on retrievals (raises cost in cloudy scenes)
When applying McNally Watts almost all scenes converge
Cost is generally very low because the screening is conservative – leads to large loss of information in cloudy scenes
But MW clearly increases information in cloudy scenes
Desirable to avoid using WMC in clear scenes and perhaps optimise settings to retain more information when cloudy
Retrieving cloud retains more information, but appears more cloud affected
Desirable to refine quality control / error budge in cloudy scenes
Either approach leads to large increase in number of useful scenes, greatly reducing T bias
Water vapour degraded std.dev. follows increase in ESD
Benefit of MW channels apparent in both cases.

41. Task 6: Retrievals with one or more missing AMSU channels
Results analysed with a view to drawing conclusions re performance of Metop-A vs Metop-B and for the MWIR retrievals with/without
Channel 7 (missing for Metop-A)
Channels 3,7 and 8.
Eumetsat provided Metop A data for two days:
23 March 2010
17 Oct 2013
New days processed, and analysed, after checking for possible differences in bias correction / noise
No significant impact of loosing channel 7
Loss of channels 3,7,8 still has minor impact
Metop A/B reasonably consistent; same bias correction / noise gave comparable results in 2013, but 2010 case gave degraded results, partly due to poor performance of L2 cloud mask.

42. Study conclusions
This study has explored the potential to improve on the operational IASI optimal estimation method (OEM) based retrieval by including information of the microwave sounders AMSU-A and MHS.
Measurement errors for the MW channels have been determined by computing observation – simulation statistics, based on using the IR only retrievals, together with RTTOV. Results consistent with NWP community values.
The Eumetsat OEM, and the piece-wise linear retrieval (PWLR) which is used as its prior, already found to perform well in cloud free scenes.
The study extended the scheme to include the fitting of spectral emissivity
Improves fit cost, ozone over desert and esp. lower trop water vapour
Approach seems stable and does not greatly affect convergence
Clearly recommend implementing this operationally
Also recommend implementing retrieval of scale factors for existing bias- correction spectra, to avoid fit problems in cold scenes.
Accounting for cloud (WMC or retrieval) greatly improves OEM coverage
MW channels clearly beneficial in cloudy scenes
With MW channels, could use climatological prior constraint
Use PWLR only as first guess to minimise iterations

43. Suggestions for further work
Temperature, humidity and emissivity products potentially very valuable in trace gas retrievals; potential clearly noted in parallel RAL study to define OEM for methane.
Ozone is also retrieved and has been compared to analysis (+MACC) here, however the approach is not really suitable for testing the quality of the ozone (analysis quality less good than for T, humidity).
Recommend to study ozone performance by extended (over time) comparison to ozone-sondes and chemical transport models.
Further work needed to refine cloud treatment
Optimise WMC settings and/or refine quality control when cloud retrieved
Improve modelling of cloud
Adopt OCA-type FM to allow e.g. ice + dust spectral properties to be modelled (possibly also multi-layer cloud?)
Improve treatment of emissivity – still have high fit cost over desert – may be related to unmodelled spectral structure, or deficiency of RTTOV v10 reflection model (v11 has Lambertian reflection instead of specular)
Study only considered few days – may need to consider more carefully time- dependent instrumental errors for AMSU/MHS (IASI stable)