1. What can we learn about clouds and their representation in models from the synergy of radar and lidar observations?
Robin Hogan, Julien Delanoë, Nicky Chalmers,
Thorwald Stein, Nicola Pounder, Anthony Illingworth
University of Reading
Thanks to Richard Forbes, Steve Woolnough,
Alessandro Battaglia, Doug Parker

2. Spaceborne radar, lidar and radiometers
EarthCare
The A-Train
NASA
700-km orbit
CloudSat 94-GHz radar (launch 2006)
Calipso 532/1064-nm depol. lidar
MODIS multi-wavelength radiometer
CERES broad-band radiometer
AMSR-E microwave radiometer
EarthCARE (launch 2013)
ESA+JAXA
400-km orbit: more sensitive
94-GHz Doppler radar
355-nm HSRL/depol. lidar
Multispectral imager
Broad-band radiometer
Heart-warming name

3. Overview What do spaceborne radar and lidar see?
Towards a “unified” retrieval of ice clouds, liquid clouds, precipitation and aerosol
Variational retrieval framework
Results from CloudSat-Calipso ice-cloud retrieval
Consistency with top-of-atmosphere radiative fluxes
Evaluation and improvement of models
Spatial structure of mid-latitude and tropical cirrus
CloudSat simulator to evaluate Unified Model over Africa
Challenges and opportunities from multiple scattering
Fast forward model
Multiple field-of-view lidar retrieval
First results from prototype unified retrieval
Outlook for model evaluation and improvement

4. What do CloudSat and Calipso see?
Cloudsat radar
Radar: ~D6, detects whole profile, surface echo provides integral constraint
Lidar: ~D2, more sensitive to thin cirrus and liquid clouds but attenuated
CALIPSO lidar
Target classification
Insects
Aerosol
Rain
Supercooled liquid cloud
Warm liquid cloud
Ice and supercooled liquid
Ice
Clear
No ice/rain but possibly liquid
Ground
Delanoe and Hogan (2008, 2010)

5. CloudSat and Calipso sensitivity
In July 2006, cloud occurrence in the subzero troposphere was 13.3%
The fraction observed by radar was 65.9%
The fraction observed by lidar was 65.0%
The fraction observed by both was 31.0%

6. Ingredients of a variational retrieval
Aim: to retrieve an optimal estimate of the properties of clouds, aerosols and precipitation from combining these measurements
To make use of integral constraints must retrieve components together
For each ray of data, define observation vector y:
Radar reflectivity values
Lidar backscatter values
Infrared radiances
Shortwave radiances
Surface radar echo (provides two-way attenuation)
Define state vector x of properties to be retrieved:
Ice cloud extinction, number concentration and lidar-ratio profile
Liquid water content profile and number concentration
Rain rate profile and number concentration
Aerosol extinction coefficient profile and lidar ratio
Forward model H(x) to predict the observations
Microphysical component: particle scattering properties
Radiative transfer component

7. The cost function + Smoothness constraints
The essence of the method is to find the state vector x that minimizes a cost function:
Each observation yi is weighted by the inverse of its error variance
The forward model H(x) predicts the observations from the state vector x
Some elements of x are constrained by a prior estimate
This term can be used to penalize curvature in the retrieved profile
+ Smoothness constraints

8. Ingredients developed before In progress Not yet developed
Retrieval framework
1. New ray of data: define state vector
Use classification to specify variables describing each species at each gate
Ice: extinction coefficient , N0’, lidar extinction-to-backscatter ratio
Liquid: liquid water content and number concentration
Rain: rain rate and normalized number concentration
Aerosol: extinction coefficient, particle size and lidar ratio
3a. Radar model
Including surface return and multiple scattering
3b. Lidar model
Including HSRL channels and multiple scattering
3c. Radiance model
Solar and IR channels
4. Compare to observations
Check for convergence
6. Iteration method
Derive a new state vector
Either Gauss-Newton or quasi-Newton scheme
3. Forward model
Not converged
Converged
Proceed to next ray of data
2. Convert state vector to radar-lidar resolution
Often the state vector will contain a low resolution description of the profile
5. Convert Jacobian/adjoint to state-vector resolution
Initially will be at the radar-lidar resolution
7. Calculate retrieval error
Error covariances and averaging kernel
Ingredients developed before
In progress
Not yet developed

9. Example ice cloud retrievals
Lidar observations
Delanoe and Hogan (2010)
Lidar forward model
Radar forward model
Visible extinction
Ice water content
Effective radius
Radar observations
MODIS radiance 10.8um
Forward modelled radiance

10. Evaluation using CERES TOA fluxes
Radar-lidar retrieved profiles containing only ice used with Edwards-Slingo radiation code to predict CERES fluxes
Small biases but large random shortwave error: 3D effects?
Shortwave
Bias 4 W m-2, RMSE 71 W m-2
Longwave
Bias 0.3 W m-2, RMSE 14 W m-2
Nicky Chalmers

11. CERES versus a radar-only retrieval
How does this compare with radar-only empirical IWC(Z, T) retrieval of Hogan et al. (2006) using effective radius parameterization from Kristjansson et al. (1999)?
Shortwave
Bias 48 W m-2, RMSE 110 W m-2
Longwave
Bias –10 W m-2, RMSE 47 W m-2
Bias 10 W m-2
RMS 47 W m-2
Nicky Chalmers

12. How important is lidar?
Remove lidar-only pixels from radar-lidar retrieval
Change to fluxes is only ~5 W m-2 but lidar still acts to improve retrieval in radar-lidar region of the cloud
Shortwave
Bias –5 W m-2, RMSE 17 W m-2
Longwave
Bias 4 W m-2, RMSE 9 W m-2
Nicky Chalmers

13. A-Train versus models Ice water content 14 July 2006 Half an orbit
150° longitude at equator
Delanoe et al. (2010)

14. Gridbox mean: In-cloud mean: Both models lack high thin cirrus
Met Office has too narrow a distribution of in-cloud IWC
ECMWF lacks high IWC values, remedied in new model version

15. Cascade project
How can we identify & cure errors in modelling African convection?
Unified Model simulations at a range of resolutions
Evaluate using A-Train retrievals
Also run “CloudSat simulator” to obtain radar reflectivity from model
Location of African easterly jet
Parker et al. (QJRMS 2005)
Moist monsoon flow
African easterly jet
Saharan air layer
Mid-level outflow

16. Cascade 40-km model versus CloudSat
Frequency of occurrence of reflectivity greater than –30 dBZ
Plot versus “dynamic latitude” (latitude relative to location of AEJ)
Anvil cirrus too low in model
Little sign of mid-level outflow
Unified Model
CloudSat (~01.30 LT)
CloudSat (~13.30 LT)
Thorwald Stein

17. Cascade 12-km model versus CloudSat
Fairly similar behaviour to 40-km model
Larger diurnal cycle in anvil
Unified Model
CloudSat (~01.30 LT)
CloudSat (~13.30 LT)
Thorwald Stein

18. Cascade 4-km model versus CloudSat
Note increase from 38 to 70 levels
Anvil cirrus now at around the right altitude
Slightly more mid-level cloud
Large overestimate of stratocumulus (and too low)
Unified Model
CloudSat (~01.30 LT)
CloudSat (~13.30 LT)
Thorwald Stein

19. Structure of Southern Ocean cirrus
Observations
Note limitations of each instrument
Retrievals

20. Slice through Hogan & Kew 3D fractal cirrus model
90 km
-5/3: Cloud-top turbulence & upscale cascade
Fall-streaks & wind-shear remove smaller scales lower in cloud: steeper power spectra
Hogan and Kew (QJ 2005)
Outer scale km
Slice through Hogan & Kew 3D fractal cirrus model
Southern Ocean cirrus is just like Chilbolton cirrus!

21. Tropical Indian Ocean cirrus
Burma
Tropical Indian Ocean cirrus
Stratiform region in upper half of cloud?
Turbulent fall-streaks in lower half of cloud?

22. Sum of two fractal cirrus simulations
600 km
120 km
Stratiform upper region dominated by larger scales
Turbulent lower region
Sum of two fractal cirrus simulations
Fall-streak paradigm unsuitable for cloud top
Thorwald performing spectral analysis on Cascade model

23. Unified retrieval: Forward model
From state vector x to forward modelled observations H(x)...
Ice & snow
Liquid cloud
Rain
Aerosol x
Adjoint of radar model (vector)
Adjoint of lidar model (vector)
Adjoint of radiometer model
Gradient of cost function (vector)
xJ=HTR-1[y–H(x)]
Vector-matrix multiplications: around the same cost as the original forward operations
Adjoint of radiative transfer models
yJ=R-1[y–H(x)]
Ice/radar
Liquid/radar
Rain/radar
Ice/lidar
Liquid/lidar
Rain/lidar
Aerosol/lidar
Ice/radiometer
Liquid/radiometer
Rain/radiometer
Aerosol/radiometer
Lookup tables to obtain profiles of extinction, scattering & backscatter coefficients, asymmetry factor
Radar scattering profile
Lidar scattering profile
Radiometer scattering profile
Sum the contributions from each constituent
Radar forward modelled obs
Lidar forward modelled obs
Radiometer fwd modelled obs
H(x)
Radiative transfer models

24. Scattering models
First part of a forward model is the scattering and fall-speed model
Same methods typically used for all radiometer and lidar channels
Radar and Doppler model uses another set of methods
Particle type
Radar (3.2 mm)
Radar Doppler
Thermal IR, Solar, UV
Aerosol
Aerosol not detected by radar
Mie theory, Highwood refractive index
Liquid droplets
Mie theory
Beard (1976)
Rain drops
T-matrix: Brandes et al. (2002) shapes
Ice cloud particles
T-matrix (Hogan et al. 2010)
Westbrook & Heymsfield
Baran (2004)
Graupel and hail
TBD
Melting ice
Wu & Wang (1991)

25. Radiative transfer forward models
Infrared radiances
Delanoe and Hogan (2008) model
Currently testing RTTOV (widely used, can do microwave, has adjoint)
Solar radiances
Currently testing LIDORT
Radar and lidar
Simplest model is single scattering with attenuation: b’=b exp(-2d)
Problem from space is multiple scattering: contains extra information on cloud properties (particularly optical depth) but no-one has previously been able to rigorously make use of data subject to pulse stretching
Use combination of fast “Photon Variance-Covariance” method and “Time-Dependent Two-Stream” methods
Adjoints for these models recently coded
Forward model for lidar depolarization is in progress

26. Examples of multiple scattering
LITE lidar (l<r, footprint~1 km)
CloudSat radar (l>r)
Stratocumulus
Intense thunderstorm
Surface echo
Apparent echo from below the surface

27. Scattering regimes Regime 2: Small-angle multiple scattering
Regime 0: No attenuation
Optical depth d << 1
Regime 1: Single scattering
Apparent backscatter b’ is easy to calculate from d at range r :
b’(r) = b(r) exp[-2d(r)]
Mean free path l
Regime 2: Small-angle multiple scattering
Occurs when Ql ~ x
Only for wavelength much less than particle size, e.g. lidar & ice clouds
No pulse stretching
Footprint x
Regime 3: Wide-angle multiple scattering (pulse stretching)
Occurs when l ~ x

28. Time-dependent 2-stream approx.
Describe diffuse flux in terms of outgoing stream I+ and incoming stream I–, and numerically integrate the following coupled PDEs:
These can be discretized quite simply in time and space (no implicit methods or matrix inversion required)
Time derivative Remove this and we have the time-independent two-stream approximation
Source Scattering from the quasi-direct beam into each of the streams
Gain by scattering Radiation scattered from the other stream
Loss by absorption or scattering
Some of lost radiation will enter the other stream
Spatial derivative Transport of radiation from upstream
Hogan and Battaglia (J. Atmos. Sci., 2008.)

29. Fast multiple scattering forward model
Hogan and Battaglia (J. Atmos. Sci. 2008)
New method uses the time-dependent two-stream approximation
Agrees with Monte Carlo but ~107 times faster (~3 ms)
Added to CloudSat simulator
CloudSat-like example
CALIPSO-like example

30. Multiple field-of-view lidar retrieval
To test multiple scattering model in a retrieval, and its adjoint, consider a multiple field-of-view lidar observing a liquid cloud
Wide fields of view provide information deeper into the cloud
The NASA airborne “THOR” lidar is an example with 8 fields of view
Simple retrieval implemented with state vector consisting of profile of extinction coefficient
Different solution methods implemented, e.g. Gauss-Newton, Levenberg-Marquardt and Quasi-Newton (L-BFGS)
lidar
Cloud top
100 m
10 m
600 m

31. Results for a sine profile
Simulated test with 200-m sinusoidal structure in extinction
With one FOV, only retrieve first 2 optical depths
With three FOVs, retrieve structure of extinction profile down to 6 optical depths
Beyond that the information is smeared out
Nicola Pounder

32. Optical depth from multiple FOV lidar
Despite vertical smearing of information, the total optical depth can be retrieved to ~30 optical depths
Limit is closer to 3 for one narrow field-of-view lidar
Nicola Pounder

33. Unified algorithm: first results for ice+liquid
But lidar noise degrades retrieval
Convergence!
Truth
Retrieval
First guess
Iterations
Observations Retrieval
Observations
Forward modelled retrieval
Forward modelled first guess

34. Add smoothness constraint
Smoother retrieval but slower convergence
Truth
Retrieval
First guess
Iterations
Observations Retrieval
Observations
Forward modelled retrieval
Forward modelled first guess

35. Unified algorithm: progress
Done:
Functioning algorithm framework exists
C++: object orientation allows code to be completely flexible: observations can be added and removed without needing to keep track of indices to matrices, so same code can be applied to different observing systems
Code to generate particle scattering libraries in NetCDF files
Adjoint of radar and lidar forward models with multiple scattering and HSRL/Raman support
Interface to L-BFGS quasi-Newton algorithm in GNU Scientific Library
In progress / future work:
Implement full ice, liquid, aerosol and rain constituents
Estimate and report error in solution and averaging kernel
Interface to radiance models
Test on a range of ground-based, airborne and spaceborne instruments, particularly the A-Train and EarthCARE satellites

36. Outlook
Use of radiances in retrieval should make retrieved profiles consistent with broadband fluxes (can test this with A-Train and EarthCARE)
EarthCARE will take this a step further
Use imager to construct 3D cloud field km wide beneath satellite
Use 3D radiative transfer to test consistency with broadband radiances looking at the cloud field in 3 directions (overcome earlier 3D problem)
How can we use these retrievals to improve weather forecasts?
Assimilate cloud products, or radar and lidar observations directly?
Assimilation experiments being carried out by ECMWF
Still an open problem as to how to ensure clouds are assimilated such that the dynamics and thermodynamics of the model are modified so as to be consistent with the presence of the cloud
How can we use these retrievals to improve climate models?
We will have retrieved global cloud fields consistent with radiation
So can diagnose in detail not only what aspects of clouds are wrong in models, but the radiative error associated with each error in the representation of clouds

38. Unified algorithm: state variables
Proposed list of retrieved variables held in the state vector x
State variable
Representation with height / constraint
A-priori
Ice clouds and snow
Visible extinction coefficient
One variable per pixel with smoothness constraint
None
Number conc. parameter
Cubic spline basis functions with vertical correlation
Temperature dependent
Lidar extinction-to-backscatter ratio
Cubic spline basis functions
20 sr
Riming factor
Likely a single value per profile 1
Liquid clouds
Liquid water content
One variable per pixel but with gradient constraint
Droplet number concentration
One value per liquid layer
Rain
Rain rate
Cubic spline basis functions with flatness constraint
Normalized number conc. Nw
One value per profile
Dependent on whether from melting ice or coallescence
Melting-layer thickness scaling factor
Aerosols
Extinction coefficient
One value per aerosol layer identified
Climatological type depending on region
Ice clouds follows Delanoe & Hogan (2008); Snow & riming in convective clouds needs to be added
Liquid clouds currently being tackled
Basic rain to be added shortly; Full representation later
Basic aerosols to be added shortly; Full representation via collaboration?

39. Minimizing the cost function
Gradient of cost function (a vector)
Gauss-Newton method
Rapid convergence (instant for linear problems)
Get solution error covariance “for free” at the end
Levenberg-Marquardt is a small modification to ensure convergence
Need the Jacobian matrix H of every forward model: can be expensive for larger problems as forward model may need to be rerun with each element of the state vector perturbed
and 2nd derivative (the Hessian matrix):
Gradient Descent methods
Fast adjoint method to calculate xJ means don’t need to calculate Jacobian
Disadvantage: more iterations needed since we don’t know curvature of J(x)
Quasi-Newton method to get the search direction (e.g. L-BFGS used by ECMWF): builds up an approximate inverse Hessian A for improved convergence
Scales well for large x
Poorer estimate of the error at the end

42. Comparison of convergence rates
Solution is identical
Gauss-Newton method converges in < 10 iterations
L-BFGS Gradient Descent method converges in < 100 iterations
Conjugate Gradient method converges a little slower than L-BFGS
Each L-BFGS iteration >> 10x faster than each Gauss-Newton one!
Gauss-Newton method requires the Jacobian matrix, which must be calculated by rerunning multiple scattering model multiple times

43. Radiative transfer forward models
Computational cost can scale with number of points describing vertical profile N; we can cope with an N2 dependence but not N3
Radar/lidar model
Applications
Speed
Jacobian
Adjoint
Single scattering: b’=b exp(-2t)
Radar & lidar, no multiple scattering N N2
Platt’s approximation b’=b exp(-2ht)
Lidar, ice only, crude multiple scattering
Photon Variance-Covariance (PVC) method (Hogan 2006, 2008)
Lidar, ice only, small-angle multiple scattering
N or N2
Time-Dependent Two-Stream (TDTS) method (Hogan and Battaglia 2008)
Lidar & radar, wide-angle multiple scattering N3
Depolarization capability for TDTS
Lidar & radar depol with multiple scattering
Lidar uses PVC+TDTS (N2), radar uses single-scattering+TDTS (N2)
Jacobian of TDTS is too expensive: N3
We have recently coded adjoint of multiple scattering models
Future work: depolarization forward model with multiple scattering
Radiometer model
Applications
Speed
Jacobian
Adjoint
RTTOV (used at ECMWF & Met Office)
Infrared and microwave radiances N
Two-stream source function technique (e.g. Delanoe & Hogan 2008)
Infrared radiances N2
LIDORT
Solar radiances
Infrared will probably use RTTOV, solar radiances will use LIDORT
Both currently being tested by Julien Delanoe