1 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Cloud and Aerosol Synergetic Products from EarthCARE Retrievals CASPER Final Review (AR) Cloud and Aerosol Synergetic Products from EarthCARE Retrievals January 19 th -20 th, 2009, ESTEC [19 th - room Fr413 / 20 th - Space Expo] Scientific Presentation for: ACM-Ice-Reading (variational synergetic ice retrieval) D. Donovan (KNMI), G.J. van Zadelhoff (KNMI) P. Kollias (McGill), W. Szyrmer (McGill), Aleksandra Tatarevic (McGill), R. Hogan (Univ. reading), J. Delanoe (Univ. Reading), F. Berger (DWD), K. Barfus (DWD), Juan-R. Acarreta (DMS) DEIMOS Space S.L. (2009) Robin Hogan and Julien Delanoe University of Reading

2 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Overview Introduction 1.Summary of achievements in Casper 2.Overview of synergy products, need for target classification CASPER Algorithm: ACM-Ice-Reading (including AC-Ice-Reading) 1.Why this algorithm is needed ? 2.Input Data and Product Definition 3.Theoretical description 4.Summary of the performance and error analysis 5.Verification and Validation 1.Blind-test cases using aircraft data 2.ECSIM cases 3.Application of a similar algorithm to CloudSat, CALIPSO and MODIS Conclusions 1.Generalizing to unified synergy algorithm 2.Recommendations for necessary post-Casper work

3 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Summary of achievements Identified the synergy products required by EarthCARE Reviewed the relevant literature for each of them (PARD) Prioritized future work on synergy algorithms for EarthCARE Described a retrieval algorithm for ice clouds that uses radar, HSRL lidar and infrared radiances (ATBD) Developed the code for the algorithm Integrated it into ECSIM Tested the code on simulated data Applied a similar algorithm (simple backscatter rather than HSRL) to a month of CloudSat/CALIPSO/MODIS data

4 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Synergy (Level 2b) overview Target classification –Radar-lidar target classification Two-instrument algorithms –Various combinations of radar (Z, v), lidar (backscatter, HSRL), MSI (IR, solar)… –To estimate ice, liquid, aerosol, precipitation properties –Too many combinations possible – need to be selective Three-instrument algorithms –Needs variational framework Higher level products –L2b-2D Cloud fraction, overlap, mean water content and inhomogeneity on pseudo-model grid

5 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Reading contribution to Casper Implemented in Casper (ATBD) Planned in Casper (PARD)

6 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Target Classification product Importance: MANDATORY –Classification essential to facilitate synergetic algorithms –Also useful to provide one file from which subsequent algorithms could run by regridding, storing errors etc. Maturity: NOVEL/MATURE –This work has been carried out on ground-based radar and lidar data –Application to CloudSat/CALIPSO is ongoing but less mature Wang & Sassen (2001) designed CloudSat 2B-CLDCLASS –Attempt to match traditional classification of stratocu, altostratus etc. But subsequent algorithms actually want to know –Target phase (liquid/ice) and where we cant be sure –Whether cloud or precipitation –Details: Hail/graupel, melting ice, warm/cold rain? –Other targets: aerosol, insects, molecular –Co-existence of the above target types

7 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Ground-based classification Example of target classification during the Cloudnet project –In this case the classes are: ice, liquid cloud, drizzle/rain and aerosol Ground-based cloud radar observations from Chilbolton

8 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. CloudSat/CALIPSO Cloudsat radar CALIPSO lidar Preliminary target classification Insects Aerosol Rain Supercooled liquid cloud Warm liquid cloud Ice and supercooled liquid Ice Clear No ice/rain but possibly liquid Ground This is an example of how such a product might look Priority for development after CASPER

9 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Radar-Lidar-MSI ice cloud product Why is this product required? –Ice clouds an important component of the radiation budget of the earth, and their properties still vary widely in climate models –A lot is known on how to combine radar and lidar in synergy, and indeed much of the preparatory work has been carried out –By adding MSI information the profile of cloud properties should more consistent with the broadband radiation measurements, which is a key mission requirement –This product should be the first official global radar-lidar retrieval of ice cloud properties, particularly effective radius Therefore this product is a key EarthCARE output and is Mandatory –Radar-Lidar (AC-Ice-Reading) version is Mature –Radar-Lidar-Radiometer (ACM-Ice-Reading) version is Novel

10 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Input data required Platform and orbit parameters –time(t), longitude(t), latitude(t), altitude(t), height(t,z)... Instrument characteristics –lidar_div, lidar_fov, C_lid(t) Measurements –Z(t, z), bscat_Mie(t, z), bscat_Ray(t, z), radiance(t, ) Measurement errors –Standard error in each of the input data Cloud mask –mask_radar(t, z), mask_lidar(t, z), cloud_phase(t, z) Met and surface data (ECMWF) –temperature(t, z), pressure(t, z), q(t, z), ozone(t, z) –surf_pressure(t), skin_temperature(t), surf_emissivity(t) Note that these are as in the KNMI merged file

11 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Product definition Platform and orbit parameters –Repeated from the input data Directly retrieved variables –Extinction (t, z), N0* (t, z), lidar_ratio (t, z) Variables derived from retrieved variables –IWC (t, z), re (t, z), optical depth (t) Forward modelled variables at final iteration –Z_fwd (t,z), bscat_Mie_fwd (t,z), bscat_Ray_fwd (t,z), radiance_fwd(t,) Measures of convergence –n_iterations (t), chi_squared (t, iteration) Status flags –retrieval_flag (t, z), instrument_flag(t, z), radiance_flag(t) Error standard deviations –ln_extinction_err (t, z), ln_N0*_err(t, z), ln_lidar_ratio_err(t, z) –ln_IWC_err (t, z), ln_effective_radius_err(t, z), optical_depth_err (t)

12 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Previous 2-instrument algorithms Various combinations of instruments similar to those on EarthCARE have been tried for ice clouds before –Lidar and radar definitely the most promising! –Radar ZD 6, lidarD 2 so the combination provides particle size Radar Lidar Wang & Sassen, Donovan et al., Tinel et al., Delanoe and Hogan IR Not feasible: radar would need to see to cloud top Chiriaco et al.; limited to thin clouds Solar Benedetti et al., Polonsky et al. (CloudSat); need to do liquid simultaneously, day only Possible; limited to thin clouds with no liquid beneath, day only Possible; need radar to be sure no liquid cloud beneath, day only ICE RadarLidarIRSolar

13 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Radar-lidar ice algorithm history Dont correct lidar for attenuation: Intrieri et al. (1993) –Limited to very thin clouds; lidar ratio must be assumed Invert lidar separately: Mace et al. (1998), Wang & Sassen (2002), Okamoto et al. (2003) –Extinction error increases into cloud due to assumed lidar ratio Optimal estimation but lidar inverted separately: Mitrescu et al. (2005) –Same lidar errors as Wang & Sassen Invert lidar with radar: Donovan & van Lammeren (2001), Tinel et al. (2005) –Lidar ratio is retrieved: much more accurate Full optimal-estimation (=variational) approach: Delanoe & Hogan (2008) –Same strengths as Donovan & van Lammeren –Allows extra constraints/obs to be included, e.g. infrared radiances –Can blend into regions detected only by radar or lidar –Provides retrieval errors and error covariances Exploit the HSRL channels when available (new in CASPER)

14 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Formulation of the problem Observation vector –Elements may be missing State vector –Retrieved variables HSRL Mie channel HSRL Rayleigh channel Radar reflectivity Radiances Extinction coefficient Lidar ratio Ratio N 0 */ 0.6

15 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Synergetic retrieval framework New ray of data: define state vector x Use merged file to specify variables describing ice cloud at each gate Radar model Radar reflectivity Lidar model Including HSRL channels and multiple scattering Radiance model IR channels Compare to observations Check for convergence Gauss-Newton iteration Derive a new state vector Forward model Not converged Converged Calculate errors and proceed to next ray of data Minimize cost function of the form: J = squared difference between observations and forward model + squared difference between state vector and a-priori + smoothness constraints

16 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Why N 0 */ 0.6 ? In-situ aircraft data show that N 0 */ 0.6 has temperature dependence that is independent of IWC Therefore we have a good a-priori estimate to constrain the retrieval Also assume vertical correlation to spread information in height, particularly to parts of the profile detected by only one instrument

17 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Why N 0 *??? We need to be able to forward model Z and other variables from x Large scatter between extinction and Z implies 2D lookup-table is required When normalized by N0*, there is a near-unique relationship between /N 0 * and Z/N 0 * (as well as r e, IWC/N 0 * etc.)

18 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Photon variance-covariance method –Hogan (Applied Optics 2006, JAS 2008) –As light propagates through a medium where size r » wavelength, narrow- angle forward-scattering widens beam –Write down differential equations for Total energy P Positional variance Directional variance Covariance –E.g. r s Lidar field-of-view (equivalent medium theorem allows forward scattering on the return journey to be neglected) Modelling HSRL with multiple scattering –Thus can calculate positional variance versus range z and hence the fraction of light remaining in field of view –Very efficient: time proportional to number of pixels squared Modelling HSRL channels in CASPER: a straightforward modification –Particles and molecules already treated separately since molecules dont have a forward-scattering lobe

19 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Effect of errors on retrievals Source of errorExtinctionEff. Radius r e IWC Any error in lidar calibrationNo effect Any change in absolute value of lidar ratio No effect Radar calibration a factor of 2 too high (+3 dB) No effect+5 mm+10% Uncertainties in the representation of small crystals No effect±15% Uncertainties in mass–size relationshipNo effect±30% Difference in radar and lidar footprints±8%±1 mm±8% Partly taken from Hogan et al. (JTECH 2006)

20 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. No HSRL Measurements Retrievals Aircraft-observed ice size spectra used to generate pseudo- measurements (Hogan et al 2006 blind test case) Note that the same lidar forward model (which includes multiple scattering) is used in generating the pseudo- measurements and in the retrieval Lidar ratio S assumed constant in retrieval

21 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. S free to vary with height (except for smoothness constraint) Can reproduce features of true S More accurate extinction where lidar has signal Some noise from Rayleigh channel in retrieved S: need more smoothness With HSRL Measurements Retrievals

22 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. ECSIM fractal cirrus case KNMI have run ECSIM using a cirrus cloud generated by the Hogan and Kew (2005) model ECSIM radar reflectivity ECSIM lidar Mie channel ECSIM lidar Rayleigh channel

23 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. ECSIM fractal cirrus case KNMI have run ECSIM using a cirrus cloud generated by the Hogan and Kew (2005) model At the final iteration, the variational scheme attempts to forward model the observations but without reproducing instrument noise ECSIM radar reflectivity ECSIM lidar Mie channel ECSIM lidar Rayleigh channel Retrieval forward model

24 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Comparison with truth True ECSIM extinction coefficient Retrieved extinction coefficient Retrieved effective radius Retrieved ice water content Slightly poorer agreement than blind-test profiles, presumably because ECSIM instrument simulator different from forward model in the retrieval algorithm

25 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. …Add radiances to retrieval True ECSIM extinction coefficient Retrieved extinction coefficient Retrieved effective radius Retrieved ice water content Merits of radiances are inconclusive; HSRL already accurate, and forward-model errors may be important…

26 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. CloudSat-CALIPSO-MODIS example 1000 km Lidar observations Radar observations

27 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. CloudSat-CALIPSO-MODIS example Lidar observations Lidar forward model Radar observations Radar forward model

28 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Extinction coefficient Ice water content Effective radius Forward model MODIS m observations Radar-lidar retrieval

29 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Radiances matched by increasing extinction near cloud top …add infrared radiances Forward model MODIS m observations

30 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Radar-lidar complementarity CloudSat radar CALIPSO lidar MODIS 11 micron channel Time since start of orbit (s) Height (km) Cirrus detected only by lidar Mid-level liquid clouds Deep convection penetrated only by radar Retrieved extinction (m -1 )

31 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. 1-month optical depth comparison Mean of all skies Mean of clouds CloudSat-CALIPSO MODIS Mean optical depth from CloudSat-CALIPSO is lower than MODIS simply because CALIPSO detected many more optically thin clouds not seen by MODIS Hence need to compare PDFs as well

32 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. First comparison with ECMWF log10(IWC[kg m -3 ])

33 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. A-Train Temperature (°C) Comparison with model IWC Met OfficeECMWF Global forecast model data extracted underneath A-Train A-Train ice water content averaged to model grid –Met Office model lacks observed variability –ECMWF model has artificial threshold for snow at around kg m -3 Temperature (°C)

34 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Post-Casper work Essential further work for ACM-Ice-Reading algorithm: –Verify that when one of the instruments is missing, algorithm will approximate existing two-instrument algorithms, e.g. Chiriaco et al. (lidar-MSI) –Check infrared forward model (e.g. against RTTOV) –Apply to real HSRL data and compare to in-situ truth Broader outlook for synergy algorithms for EarthCARE –Develop unified algorithm to retrieve all species (ice cloud, liquid cloud, aerosols and precipitation) simultaneously; often several present in the same profile –Considerable work required to select best state vector etc.

35 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. General synergetic framework New ray of data: define state vector Use classification to specify variables describing each species at each gate Ice: extinction coefficient and N 0 * Liquid: liquid water content and number concentration Rain: rain rate and mean drop diameter Aerosol: extinction coefficient and particle size Radar model Including surface return and multiple scattering Lidar model Including HSRL channels and multiple scattering Radiance model Solar and IR channels Compare to observations Check for convergence Gauss-Newton iteration Derive a new state vector Forward model Not converged Converged Proceed to next ray of data (Black) Ingredients delivered in Casper (Delanoe and Hogan JGR 2008) (Red) Ingredients remaining to be developed

36 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Components of liquid-cloud algorithm Source of informationCaveats Radar reflectivity of cloud: if sensitive to the cloud droplets then Z is strongly related to LWC Not applicable if cloud contains drizzle droplets, which is the case in most clouds over the ocean (Fox & Illingworth 1997) MSI optical depth provides path constraint (e.g. Austin & Stephens 2001 combined with radar) and size information Only works in daylight with no other clouds in the profile ; less accurate over more reflective surfaces Surface return provides estimate of path attenuation, proportional to LWP (Smith and Illingworth, in prep) Only over the sea, need to find clear-sky regions to each side for baseline; dependence on surface wind stress For < 1.5 (e.g. some supercooled clouds), can be derived from integrated lidar backscatter (Hogan et al. 2003) Only a small fraction of optically thin clouds, and more difficult when no cloud above In optically thick clouds, multiple scattering can result in exponential tail related to (Polonsky and Davis 2004) Narrow EarthCARE field of view probably means that the exponential tail is too weak Width in range of lidar backscatter peak in optically thick clouds is related to number concentration (OConnor) Needs validation ; need to check dependence on entrainment of dry air and effect of cloud inhomogeneity in sampling period HSRL provides extinction coefficient near cloud top Unclear how useful this information is further into the cloud due to dilution by entrainment of dry air near cloud top Rate of increase of depolarization ratio due to multiple scattering provides a measure of extinction coefficient Currently no fast forward model for the effects of multiple scattering on depolarization

37 All Rights Reserved. No part of this document may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of DEIMOS Space S.L. Recommendations Radar/lidar mis-pointing should be < 500 m, equivalent to RMS error in Z of 0.5 dB Most two-instrument algorithms should be tested as limiting cases of a more general multi-instrument algorithm, covering EarthCARE in case of instrument failure The target classification should be developed as a priority, and would include the measurements on the same grid, to facilitate synergetic algorithms Level 2b-2D products should be produced: cloud fraction, overlap, etc., under EarthCARE but averaged to typical model resolution; satisfies a key mission requirement A flexible optimal-estimation software library should be developed to facilitate implementation of synergetic algorithms, in particular the best-estimate algorithms A scattering library and associated tools should be developed, to enable the look-up tables required by all algorithms to be generated consistently across the full spectrum A concerted effort is required to validate algorithms using a wide variety of data sources, including the A-train, ECSIM and dedicated aircraft campaigns Areas requiring focussed attention: –Need a shortwave forward model to allow shortwave radiances to be utilized –Utilizing surface return over the ocean to detect small liquid attenuation –Exploiting multiply-scattered radar returns in using appropriate forward model –Treating the complex microphysics of deep convective cloud adequately –Developing appropriate constraints on vertical profile of retrieved variables, such as continuity of mass flux across the melting layer in precipitating situations