OEM retrievals with IASI, AMSU and MHS: Final Presentation Summary of Study Findings 5 February 2015 R.Siddans, D. Gerber (RAL),

Purpose of the Study To evaluate the benefit of adding microwave (MW) channels to the measurement vector of Eumetsat’s optimal estimation method (OEM) based scheme for retrieving temperature, humidity and ozone from the infra-red (IR) sounder IASI. Eumetsat provide the description and input data of the baseline (IR-only) OEM scheme which is to be extended in the study. The study should also extend the scheme To fit surface spectral emissivity (IR and MW) To work in the presence of (some) cloud (but not precipitation) Additionally the impact of some specific AMSU channels (reflecting Metop- A performance) is to be studied

Study Tasks / Work Breakdown Schedule KO in December 2013 WP1000 : Definition of the Sy matrix in the microwaves Input via consultancy from Bill Bell (Met Office) WP2000: OEM(MWIR/Metop-B) over ocean, clear sky Set up IASI OEM to match EUMETSAT L2 PPF configuration Run retrievals (IR and MWIR) and analyse residuals WP3000 : OEM(MWIR/Metop-B) over land, clear sky, with fixed emissivities WP4000 : OEM(MWIR/Metop-B) over land, clear sky, with variable emissivities WP5000 : OEM(MWIR/Metop-B) in partial or full cloudy IFOVs WP6000 : Retrievals with one or more missing AMSU channels WP7000 : Delivery of datasets and final reporting

AMSU/MHS Channels AMSU-A # Freq.GHz 1 23.8 2 31.4 3 50.3 4 52.8 5 53.6 1 23.8 2 31.4 3 50.3 4 52.8 5 53.6 6 54.4 7 54.94 8 55.5 9 57.29 10 57.29 11 57.29 12 57.29 13 57.29 14 57.29 15 89 MHS # Freq/ GHz 1 89 2 150 3 183.3 4 183.3 5 190.3

IASI/AMSU / MHS View Geometry All instruments have ~circular FOV IASI has 4 detectors each with 12km diameter footprint (at nadir) AMSU + MHS have 48 and 16km diameter footprints (respectively) Positions systematically co-located wrt each other across track. Eumetsat provided nearest neighbour co-located observations from all 3 sensors ( based on IASI spatial sampling)

Overview of the Eumetsat IASI scheme OEM solves under-constrained inverse problem using a prior estimates of the parameters to be retrieved, yielding the most likely solution given the characterised errors on the measurements and the prior estimate. This is achieved by solving the cost function: Where x is state vector (parameters to be retrieved) y is measurement vector (a subset of IASI PC re-constructed / filtered radiances), with errors characterised by covariance Sy F(x) is forward model which predicts measurements given state (RTTOV v10.2) xa is the a priori state, which is assumed to have error characterised by Sa The cost function is minimised using iterative approach (Newtonian) K is the weighting function matrix – derivatives of F(x) with respect to x

Overview of the Eumetsat Scheme: A priori A priori state (and first guess) from separate statistical retrieval which uses selected IASI, AMSU and MHS measurements as predictors The relationship between the predictors and the state is derived using a the piece-wise linear regression (PWLR) scheme, training 12 days of measurements against co-located ECMWF analyses The state is expressed in terms of principle components of the covariance of the PWLR profiles against the analyses. 28 principle components are used to represent temperature, 18 for humidity and 10 for ozone. The OEM retrieves the weights of each of these profile principle components (+surface temperature) Temperature is retrieved in K, humidity and ozone in ln(ppmv) The use of PWLR as prior, means the prior state is already rather good. The PWLR is also relatively insensitive to cloud (it uses MW observations & can exploit statistical relationships between predictors/state. -> It is a challenge for the OEM to be “better” than PWLR

Overview of the Eumetsat Scheme: Measurements Observations in 139 IASI channels are used (selected via trade-off of information content vs processing speed). These are obtained from PC compressed L1 data. To suppress instrument artefacts, reconstruction of radiances from the PC scores is followed by a projection onto the forward model subspace as defined by a transform provided by Eumetsat See Tech.Note “Canonical angles between the IASI Observation and forward model sub-spaces”, Tim Hultberg, Thomas August Measurement covariance is estimated using differences between observations and FM simulations based on ECMWF analyses (for sub-set of days) A bias correction spectrum, prescribed as a function of view zenith angle is also applied to the “double-filtered” measurements

Example measurement / fit residuals from IR OEM Metop B Mid-latitude ocean 17 April 2013

IR only retrieval diagnostics (mid-lat)

Task 1: Definition of the Sy matrix for the MW channels Literature review of use of AMSU+MHS in OEM Dr William Bell (Met Office) consultant to consortium to provide expertise on use of AMSU+MHS Met Office will provide estimates of the NEDT for all of the AMSU-A / MHS channels, as well as the observation covariances used in the operational assimilation of ATOVS radiances In addition we estimate statistics (bias and covariances) of the AMSU/MHS departures from the provided IASI OEM and piece-wise linear regression (PWLR) profiles. Currently measurement covariances are based on differences between MW observations and those computed using the IASI OEM retrieved state

Comparison to Met Office and ECMWF estimated errors Met Office estimates Study results from Observation – simulations from IR only retrieved state ECMWF estimates

Observation coveariance derived from MW residuals from IASI retrieval

Task 2: OEM (MWIR/Metop-B) over ocean, clear-sky Task 3: OEM (MWIR/Metop-B) over land clear-sky Implement the IASI PPF OEM settings in RAL code Apply scheme to selected days of IASI and AMSU/MHS cloud-free data over ocean/land, to generate results for IR only and MW+IR (MWIR). Days selected: 17 April, 17 July, 17 October 2013 (Metop B) Evaluate results using the diagnostics such as: PWLR, OEM(IR), OEM(MWIR) cf reference profiles (ECMWF analysis) vertical profiles of bias, standard deviation histograms and scatter plots for selected pressure levels maps of departures DOFS, AKs, fit residuals of OEM(IR) cf OEM(MWIR) RTTOV internal emissivity atlases (TELSEM/CNRM/Wisconsin) + sea model used Eumetsat cloud, precipitation and sea-ice masking used

Summary of changes in DOFS adding MW channels

Standard ir-only; mid-lat sea example

E.g. orbit cross-section: Temperature

Statistical assessment of retrievals Based on comparing retrieval to analysis (ANA), Eumetsat retrieval (ODV), PWLR and analysis smoothed by averaging kernel (ANA_AK): x’ = a + A ( t - a ) Where a is the a priori profile from the PWLR, t is the supposed "true", A is the retrieval averaging kernel Profiles smoothed/sampled to grid more closely matching expected vertical resolution (than 101 level RTTOV grid), to avoid spurious structures: Temperature: 0, 1, 2, 3, 4, 6, 8, 10, 12, 14, 17, 20, 24, 30, 35,40,50 km. Water vapour: 0, 1, 2, 3,4, 6, 8, 10, 12, 14, 17,20 km Ozone: 0, 6, 12, 18, 24, 30, 40 km. The grid is defined relative to the surface pressure / z*. Profile results further summarised (for maps, tables) into 3 layers BL: 0-2 km z* (above surface) LT: 0-6 km UT: 6-12 km The mean value of individual profiles taken over these ranges, then stats calculated (summarises bias over these layers)

MWIR vs IR Temperature Std.deviations higher over land Retrieval slightly better than PWLR especially over land Except retrieval biased near surface This varies with surface type/day/night, maybe analysis/sampling error No obvious benefit of MW – agreement with Ana x AK degrades, but this is more a reflection of increased sensitivity

MWIR vs IR Humidity Std.deviations much higher over land, near surface PWLR better than retrieval near ground over land, but not in UT Bias structure largely resolved by averaging kernels (effect of steep gradient) Again, no obvious benefit of MW + agreement with Ana x AK degrades

MWIR vs IR-only Use of “climatological prior” instead of PWLR gives more obvious benefit of MW channels even in cloud-free scenes… Temperature Water vapour PWLR prior Climatological prior However basic performance degraded, especially near-surface H2O over land

Task 4: Addition of surface emissivity to state vector Basic approach Emissivity included in state vector in terms of principle components: Eigenvectors of global covariance from RTTOV atlases (for the 3 days assessed in the study) A priori covariance is diagonal, filled with corresponding Eigenvalues Have co-located spectra for MW+IR so can include correlations between MW and IR in the prior constraint Land and sea (and permanent land ice) spectra included in the covariance, so retrieval should ~work around coast. IR Atlas based on Wisconsin principle components of natural materials. 416 spectral patterns defined (at 416 wavelengths from 700-2774 cm-1) but Only leading 6 patterns used (limit from MODIS channels) Spectral shapes of further patterns needed to explain IASI observations, but no measure of their occurrence globally is available to define the prior constraint for these. Other patterns included by deriving residual patterns not explained by RTTOV-atlas based patterns; Shift of mean emissivity also fitted.

Surface emissivity Eigenvectors and values fitted (including MW correlations).

IASI/AMSU/MHS Study – emissivity ret(desert) MWIR Emis = RTTOV MWIR Emis: Ret emissivity Original scheme Fitted emissivity Fitted emissivity + bias correction

Fitted emissivity cf RTTOV day vs night

Surface Temperature - Analysis Stability of retrieval tested by setting prior emissivity to 1 (no spectral dependence) Retrieved emissivity only slightly affected; bias slightly high, with compensating change in surface T, but impact on fitted profiles is very small. A priori emissivity=RTTOV A priori emissivity=1 0.5K change in mean surface T (no change in std.dev)

MWIR with emissivity vs standard IR OEM No significant change to T, but LT water vapour improves when emissivity fit… Temperature Water vapour PWLR prior Climatological prior This is true also for IR-only retrieval; benefit of MW small

MWIR vs IR-only (no emissivity fit) Temperature Water vapour PWLR prior Climatological prior

Water vapour maps: Emissivity benefit… MWIR; with emissivity MWIR; no emissivity

Task 5: OEM(MWIR/Metop-B) in partial or full cloudy IFOVs Benefit of MW mainly expected to be in allowing OEM to function in (partially) cloudy scenes, once cloud impact on IR accounted for in OEM. Two approaches implemented in study to account for cloud: McNally Watts approach used to identify cloud, then channels ignored in retrieval, depending on their vertical sensitivity RTTOV’s simple black-body cloud used in FM, cloud height and fraction retrieved. Results assessed using Eumetat L2 cloud parameters to segment comparison with analysis/PWLR.

McNally – Watts Method Approach to cloud-screen individual spectral channels so some (high altitude sensitive) channels can still be used in presence of (lower) cloud Instead of identifying cloud signatures in the measurements directly, the method tries to identify the altitude range in each channel for which a given measurement is not affected by cloud (i.e. indistinguishable from cloud-free case) Method described in McNally, A. P., Watts, P. D., A cloud detection algorithm for high-spectral- resolution infrared sounders, Q. J. R. Meteorol. Soc. (2003), 129, pp. 3411– 3423 In the following work we label the McNally – Watts method as “WMC” (to avoid confusion with the MW acronym) Here we implement exactly the scheme as provided by NWP SAF – however there are several tunable parameters (not optimised here)

Performance of OEM Cloud Retrieval Alternative to WMC: Retrieve fraction and height of cloud described using RTTOVs black-body cloud model. Retrieved cloud fraction consistent with IR picture (incl. feathered edges) Estimated error due to cloud indicates lack of info from IR Retrieved cloud height consistent with cloud fraction (5km background value from a priori) Retrieved TSurface under cloud bank reasonable under cloud Retrieved cloud parameters found to compare quite well to operational L2. WMC estimated “height” much higher.

MW+IR with emissivity Ignore cloud MW+IR with emissivity +McNally/Watts MW+IR with emissivity + retrieve cloud MW+IR with emissivity + retrieve cloud (climatological prior) Either approach to cloud strongly mitigates T errors below cloud, giving results comparable / better than PWLR PWLR

Ret-Analysis BL Temp: Ignore cloud

Ret-Analysis BL Temp: Fit cloud

Ret-Analysis BL Temp: McNally Watts

Summary stats for Lower Tropospheric Temperature, over land as function of L2 cloudiness flag ESD/K DOFS Ret-Ana PWLR-Ana Mean difference /K Standard OEM Ret-Ana PWLR-Ana Std.dev./K Ignore cloud IR only Ignore cloud MWIR WMC IR only WMC MWIR Retrieve cloud IR only Retrieve cloud MWIR Only retrievals with cost < 500 N. Cases /1000 Cost

Summary stats for Lower Tropospheric Humidity, over land as function of L2 cloudiness flag Standard OEM Ignore cloud IR only Ignore cloud MWIR WMC IR only WMC MWIR Retrieve cloud IR only Retrieve cloud MWIR Only retrievals with cost < 500 N. Cases /1000 Cost ESD/% DOFS Ret-Ana PWLR-Ana Mean difference /% Ret-Ana PWLR-Ana Std.dev./%

Cloud summary When cloud is ignored: Cost similar for cloud mask 1 or 2. However there is an increased negative bias in Retrieval – Analysis temperature for mask 2 cases. Cost+ temperature bias increases greatly for cloud mask 3 + 4 MW provides a useful test on retrievals (raises cost in cloudy scenes) When applying McNally Watts almost all scenes converge Cost is generally very low because the screening is conservative – leads to large loss of information in cloudy scenes But MW clearly increases information in cloudy scenes Desirable to avoid using WMC in clear scenes and perhaps optimise settings to retain more information when cloudy Retrieving cloud retains more information, but appears more cloud affected Desirable to refine quality control / error budge in cloudy scenes Either approach leads to large increase in number of useful scenes, greatly reducing T bias Water vapour degraded std.dev. follows increase in ESD Benefit of MW channels apparent in both cases.

Task 6: Retrievals with one or more missing AMSU channels Results analysed with a view to drawing conclusions re performance of Metop-A vs Metop-B and for the MWIR retrievals with/without Channel 7 (missing for Metop-A) Channels 3,7 and 8. Eumetsat provided Metop A data for two days: 23 March 2010 17 Oct 2013 New days processed, and analysed, after checking for possible differences in bias correction / noise No significant impact of loosing channel 7 Loss of channels 3,7,8 still has minor impact Metop A/B reasonably consistent; same bias correction / noise gave comparable results in 2013, but 2010 case gave degraded results, partly due to poor performance of L2 cloud mask.

Study conclusions This study has explored the potential to improve on the operational IASI optimal estimation method (OEM) based retrieval by including information of the microwave sounders AMSU-A and MHS. Measurement errors for the MW channels have been determined by computing observation – simulation statistics, based on using the IR only retrievals, together with RTTOV. Results consistent with NWP community values. The Eumetsat OEM, and the piece-wise linear retrieval (PWLR) which is used as its prior, already found to perform well in cloud free scenes. The study extended the scheme to include the fitting of spectral emissivity Improves fit cost, ozone over desert and esp. lower trop water vapour Approach seems stable and does not greatly affect convergence Clearly recommend implementing this operationally Also recommend implementing retrieval of scale factors for existing bias- correction spectra, to avoid fit problems in cold scenes. Accounting for cloud (WMC or retrieval) greatly improves OEM coverage MW channels clearly beneficial in cloudy scenes With MW channels, could use climatological prior constraint Use PWLR only as first guess to minimise iterations

Suggestions for further work Temperature, humidity and emissivity products potentially very valuable in trace gas retrievals; potential clearly noted in parallel RAL study to define OEM for methane. Ozone is also retrieved and has been compared to analysis (+MACC) here, however the approach is not really suitable for testing the quality of the ozone (analysis quality less good than for T, humidity). Recommend to study ozone performance by extended (over time) comparison to ozone-sondes and chemical transport models. Further work needed to refine cloud treatment Optimise WMC settings and/or refine quality control when cloud retrieved Improve modelling of cloud Adopt OCA-type FM to allow e.g. ice + dust spectral properties to be modelled (possibly also multi-layer cloud?) Improve treatment of emissivity – still have high fit cost over desert – may be related to unmodelled spectral structure, or deficiency of RTTOV v10 reflection model (v11 has Lambertian reflection instead of specular) Study only considered few days – may need to consider more carefully time- dependent instrumental errors for AMSU/MHS (IASI stable)