1. Robin Hogan Julien Delanoe Department of Meteorology, University of Reading, UK Towards unified radar/lidar/radiometer retrievals for cloud radiation studies

2. Motivation Clouds are important due to their role in radiative transfer –A good cloud retrieval must be consistent with broadband fluxes at surface and top-of-atmosphere (TOA) Increasingly, multi-parameter cloud radar and lidar are being deployed together with a range of passive radiometers –We want to retrieve an optimum estimate of the state of the atmosphere that is consistent with all the measurements –But most algorithms use at most only two instruments/variables and dont take proper account of instrumental errors The variational framework is standard in data assimilation and passive sounding, but has only recently been applied to radar –Mathematically rigorous and takes full account of errors –Straightforward to add extra constraints and extra instruments In this talk it will be shown how radar, lidar and infrared radiometers can be combined for ice cloud retrievals –Demonstrated on ground-based and satellite (A-train) observations –Discuss challenges of extending to other clouds and other instruments

3. Surface/satellite observing systems Ground-based sites ARM and Cloudnet NASA A-Train Aqua, CloudSat, CALIPSO, PARASOL ESA EarthCARE For launch in 2013 Radar 35 and/or 94 GHz Doppler, Polarization 94 GHz CloudSat94 GHz CPR Doppler Lidar Usually 532 or 905 nm Polarization 532 & 1064 nm CALIOP Polarization 355 nm ATLID Polarization, HSRL VIS/IR radiometers Some have infrared radiometer, sky imager, spectrometer MODIS, AIRS, CALIPSO IIR (Imaging Infrared Radiometer) Multi-Spectral Imager (MSI) Microwave radiometers Dual-wavelength radiometer (e.g. 22 & 28 GHz) AMSR-E (6, 10, 18, 23, 36, 89 GHz) Polarization None Broadband radiometers Surface BBR Europe/Africa sites have GERB overhead CERES (TOA only)BBR (TOA only) –Broadband radiometers used only to test retrievals made using the other instruments

4. Radar and lidar Advantages of combining radar, lidar and radiometers –Radar Z D 6, lidar D 2 so the combination provides particle size –Radiances ensure that the retrieved profiles can be used for radiative transfer studies Some limitations of existing radar/lidar ice retrieval schemes (Donovan et al. 2000, Tinel et al. 2005, Mitrescu et al. 2005) –They only work in regions of cloud detected by both radar and lidar –Noise in measurements results in noise in the retrieved variables –Elorantas lidar multiple-scattering model is too slow to take to greater than 3rd or 4th order scattering –Other clouds in the profile are not included, e.g. liquid water clouds –Difficult to make use of other measurements, e.g. passive radiances –Difficult to also make use of lidar molecular scattering beyond the cloud as an optical depth constraint –Some methods need the unknown lidar ratio to be specified A unified variational scheme can solve all of these problems

5. Why not invert the lidar separately? Standard method: assume a value for the extinction-to- backscatter ratio, S, and use a gate-by-gate correction –Problem: for optical depth >2 is excessively sensitive to choice of S –Exactly the same instability for radar (Hitschfeld & Bordan 1954) Better method (e.g. Donovan et al. 2000): retrieve the S that is most consistent with the radar and other constraints –For example, when combined with radar, it should produce a profile of particle size or number concentration that varies least with range Implied optical depth is infinite

6. Example from US ARM site: Need to distinguish insects from cloud First step: target classification Ice Liquid Rain Aerosol Insects Combining radar, lidar with temperature from a model allows the type of cloud (or other target) to be identified –Example from Cloudnet processing of ARM data (Illingworth et al., BAMS 2007)

7. Formulation of variational scheme For each ray of data we define: Observation vector State vector –Elements may be missing –Logarithms prevent unphysical negative values Attenuated lidar backscatter profile Radar reflectivity factor profile (on different grid) Ice visible extinction coefficient profile Ice normalized number conc. profile Extinction/backscatter ratio for ice Visible optical depth Aerosol visible extinction coefficient profile Liquid water path and number conc. for each liquid layer Infrared radiance Radiance difference

8. The cost function The essence of the method is to find the state vector x that minimizes a cost function: + Smoothness constraints Each observation y i is weighted by the inverse of its error variance The forward model H(x) predicts the observations from the state vector x Some elements of x are constrained by an a priori estimate This term penalizes curvature in the extinction profile

9. Solution method An iterative method is required to minimize the cost function New ray of data Locate cloud with radar & lidar Define elements of x First guess of x Forward model Predict measurements y from state vector x using forward model H(x) Predict the Jacobian H=y i /x j Has solution converged? 2 convergence test Gauss-Newton iteration step Predict new state vector: x k+1 = x k +A -1 {H T R -1 [y-H(x k )] -B -1 (x k -b)-Tx k } where the Hessian is A=H T R -1 H+B -1 +T Calculate error in retrieval No Yes Proceed to next ray

10. How do we solve this? The best estimate of x minimizes a cost function: At minimum of J, dJ/dx=0, which leads to: –The least-squares solution is simply a weighted average of m and b, weighting each by the inverse of its error variance Can also be written in terms of difference of m and b from initial guess x i : Generalize: suppose I have two estimates of variable x : –m with error m (from measurements) –b with error b (background or a priori knowledge of the PDF of x )

11. The Gauss-Newton method We often dont directly observe the variable we want to retrieve, but instead some related quantity y (e.g. we observe Z dr and dp but not a ) so the cost function becomes –H(x) is the forward model predicting the observations y from state x and may be complex and non-analytic: difficult to minimize J Solution: linearize forward model about a first guess x i –The x corresponding to y=H(x), is equivalent to a direct measurement m : …with error: x y xixi x i+1 x i+2 Observation y (or m )

12. Substitute into prev. equation: –If it is straightforward to calculate y/ x then iterate this formula to find the optimum x If we have many observations and many variables to retrieve then write this in matrix form: –The matrices and vectors are defined by: State vector, a priori vector and observation vector The Jacobian Error covariance matrices of observations and background Where the Hessian matrix is

13. Radar forward model and a priori Create lookup tables –Gamma size distributions –Choose mass-area-size relationships –Mie theory for 94-GHz reflectivity Define normalized number concentration parameter –The N 0 that an exponential distribution would have with same IWC and D 0 as actual distribution –Forward model predicts Z from extinction and N 0 –Effective radius from lookup table N 0 has strong T dependence –Use Field et al. power-law as a-priori –When no lidar signal, retrieval relaxes to one based on Z and T (Liu and Illingworth 2000, Hogan et al. 2006) Field et al. (2005)

14. Lidar forward model: multiple scattering 90-m footprint of Calipso means that multiple scattering is a problem Elorantas (1998) model –O (N m /m !) efficient for N points in profile and m-order scattering –Too expensive to take to more than 3rd or 4th order in retrieval (not enough) New method: treats third and higher orders together –O (N 2 ) efficient –As accurate as Eloranta when taken to ~6th order –3-4 orders of magnitude faster for N =50 (~ 0.1 ms) Hogan (Applied Optics, 2006). Code: www.met.rdg.ac.uk/clouds Ice cloud Molecules Liquid cloud Aerosol Narrow field-of-view: forward scattered photons escape Wide field-of- view: forward scattered photons may be returned

15. Poster P3.10: Multiple scattering CloudSat multiple scattering To extend to precip, need to model radar multiple scattering New model agrees well with Monte Carlo

16. Radiance forward model MODIS solar channels provide an estimate of optical depth –Only very weakly dependent on vertical location of cloud so we simply use the MODIS optical depth product as a constraint –Only available in daylight –Likely to be degraded by 3D cloud effects MODIS, CALIPSO and SEVIRI each have 3 thermal infrared channels in atmospheric window region –Radiance depends on vertical distribution of microphysical properties –Single channel: information on extinction near cloud top –Pair of channels: ice particle size information near cloud top Radiance model uses the 2-stream source function method –Efficient yet sufficiently accurate method that includes scattering –Provides important constraint for ice clouds detected only by lidar –Ice single-scatter properties from Anthony Barans aggregate model –Correlated-k-distribution for gaseous absorption (from David Donovan and Seiji Kato)

17. Ice cloud: non-variational retrieval Donovan et al. (2000) algorithm can only be applied where both lidar and radar have signal Observations State variables Derived variables Retrieval is accurate but not perfectly stable where lidar loses signal Donovan et al. (2000) Aircraft- simulated profiles with noise (from Hogan et al. 2006)

18. Variational radar/lidar retrieval Noise in lidar backscatter feeds through to retrieved extinction Observations State variables Derived variables Lidar noise matched by retrieval Noise feeds through to other variables

19. …add smoothness constraint Smoothness constraint: add a term to cost function to penalize curvature in the solution (J = i d 2 i /dz 2 ) Observations State variables Derived variables Retrieval reverts to a-priori N 0 Extinction and IWC too low in radar-only region

20. …add a-priori error correlation Use B (the a priori error covariance matrix) to smooth the N 0 information in the vertical Observations State variables Derived variables Vertical correlation of error in N 0 Extinction and IWC now more accurate

21. …add visible optical depth constraint Integrated extinction now constrained by the MODIS-derived visible optical depth Observations State variables Derived variables Slight refinement to extinction and IWC

22. …add infrared radiances Better fit to IWC and r e at cloud top Observations State variables Derived variables Poorer fit to Z at cloud top: information here now from radiances

23. Convergence The solution generally converges after two or three iterations –When formulated in terms of ln( ), ln( ) rather than the forward model is much more linear so the minimum of the cost function is reached rapidly

24. Radar-only retrieval Retrieval is poorer if the lidar is not used Observations State variables Derived variables Profile poor near cloud top: no lidar for the small crystals Use a priori as no other information on N 0

25. Radar plus optical depth Note that often radar will not see all the way to cloud top Observations State variables Derived variables Optical depth constraint distributed evenly through the cloud profile

26. Radar, optical depth and IR radiances Observations State variables Derived variables

27. Observed 94-GHz radar reflectivity Observed 905-nm lidar backscatter Forward model radar reflectivity Forward model lidar backscatter Ground-based example Lidar fails to penetrate deep ice cloud

28. Retrieved extinction coefficient Retrieved effective radius r e Retrieved normalized number conc. parameter N 0 Error in retrieved extinction Lower error in regions with both radar and lidar Radar only: retrieval tends towards a-priori

29. Ground based example Radagast Campaign (AMMA) –Based in Niamey, Niger ARM Mobile Facility –MMCR cloud radar –532-nm micropulse lidar –SEVIRI radiometer aboard MeteoSat 2nd Generation: 8.7, 10.8, 12µm channels Ice cloud case, 22 July 2006

30. Example from the AMF in Niamey 94-GHz radar reflectivity 532-nm lidar backscatter Forward model at final iteration 94-GHz radar reflectivity 532-nm lidar backscatter Observations

31. Retrievals in regions where radar or lidar detects the cloud Retrieved visible extinction coefficient Retrieved effective radius Results: radar+lidar only Large error where only one instrument detects the cloud Retrieval error in ln(extinction)

32. TOA radiances increase retrieved optical depth and decrease particle size near cloud top Cloud-top error greatly reduced Retrieval error in ln(extinction) Retrieved visible extinction coefficient Retrieved effective radius Results: radar, lidar, SEVERI radiances

33. CloudSat/CALIPSO retrieval 0352 0355 0358 Oct 13, 2006 0352-0358 Radar Reflectivity from CloudSat Attenuated lidar backscatter from CALIPSO Height [km] AVHRR

34. Forward model Observed radar reflectivity, 95 GHz Attenuated lidar backscatter, 532 nm Radar reflectivity forward model Attenuated lidar backscatter forward model

35. Preliminary results (radar+lidar) October 13th 2006 Granule 2006286023036_02443 between 3h52 and 3h58 UTC Retrieved error in ln(extinction) Height [km] Retrieved number concentration Height [km] Retrieved effective radius Height [km] Retrieved visible extinction coefficient, log 10 (m -1 ) Height [km] Supercooled water?

36. MODIS radiances Radar Reflectivity from CloudSat Attenuated lidar backscatter from CALIPSO Radiances W sr -1 m -2 Forward model MODIS 8.4–8.7 micron 10.78–11.25 micron 11.77 – 12.27 micron Height [km] Radiances not used in retrieval, just forward modeled for comparison

37. CloudSat/CALIPSO example Radar Reflectivity from CloudSat 2006 Day 286 Attenuated lidar backscatter from CALIPSO Supercooled water: strong signal from lidar, weak (or nothing) from radar Radar fails to detect thin cirrus

38. Conclusions and ongoing work New radar/lidar/radiometer cloud retrieval scheme –Applied to ground based or satellite data –Appropriate choice of state vector and smoothness constraints ensures the retrievals are accurate and efficient –Can include any relevant measurement if forward model is available –Could provide the basis for cloud/rain data assimilation Extension to other cloud types –Retrieve properties of liquid-water layers, drizzle and aerosol –Incorporate microwave radiances and wide-angle radar/lidar multiple- scattering forward models for deep precipitating clouds Other activities –Validate using aircraft underflights –Use in radiative transfer model to compare with TOA & surface fluxes –Build up global cloud climatology to evaluate models

39. Attenuated lidar backscatter from CALIPSO CloudSat/CALIPSO example 2006 Day 172 Molecular signal Radar Reflectivity from CloudSat Precipitation: Both radar and lidar are attenuated and suffer from multiple scattering

40. Enforcing smoothness 1 Cubic-spline basis functions –Let state vector x contain the amplitudes of a set of basis functions –Cubic splines ensure that the solution is continuous in itself and its first and second derivatives –Fewer elements in x more efficient! Forward model Convert state vector to high resolution: x hr =Wx Predict measurements y and high-resolution Jacobian H hr from x hr using forward model H(x hr ) Convert Jacobian to low resolution: H=H hr W Representing a 50-point function by 10 control points The weighting matrix

41. Enforcing smoothness 2 Twomey matrix, for when we have no useful a priori information –Add a term to the cost function to penalize curvature in the solution: d 2 x/dr 2 (where r is range and is a smoothing coefficient) –Implemented by adding Twomey matrix T to the matrix equations