12 April 2013 VARSY progress meeting Robin Hogan and Nicola Pounder (University of Reading)

Brief summary of progress No plots today: Full error descriptors now implemented for liquid clouds and rain (ice already done) Solar radiance forward model: code included to describe scattering phase function with Legendre polynomials but still needs to be coupled to the LIDORT radiative transfer model Plots today: Liquid cloud retrievals using multiple scattering from single field-of- view lidar Calipso Overcoming multiple minima in the cost function for liquid cloud Possible algorithm speed-up being investigated: Levenberg-Marquardt minimization rather than quasi-Newton, plus GPU computation of Jacobian matrix Ability to simulate EarthCARE data (including Doppler and HSRL) from A-Train retrievals, then retrieve from the simulated EarthCARE data

Unified retrieval Ingredients developed Not yet developed 1. New ray of data: define state vector Use classification to specify variables describing each species at each gate Ice: extinction coefficient, N 0 ’, lidar extinction-to-backscatter ratio, riming factor Liquid: extinction coefficient and number concentration Rain: rain rate, drop diameter and melting ice Aerosol: extinction coefficient, particle size and lidar ratio 3a. Radar model With surface return and multiple scattering 3b. Lidar model Including HSRL channels and multiple scattering 3c. Radiance model Solar & IR channels 4. Compare to observations Check for convergence 6. Iteration method Derive a new state vector 3. Forward model Not converged Converged Proceed to next ray of data 2. Convert state vector to radar-lidar resolution Often the state vector will contain a low resolution description of the profile 7. Calculate retrieval error Error covariances & averaging kernel

Forward modelled backscatter Observed backscatter Liquid cloud retrieval We have found that the multiple scattering signal from Calipso can be inverted to get extinction profile for optical depth up to at least 30 Benefits from a constraint on LWC to be no steeper than adiabatic We can validate with CloudSat PIA, or assimilate PIA too Example from 1 minute (~400 km) of oceanic stratocumulus:

LWC Effective radius Optical depth CloudSat PIA Assimilate only Calipso backscatter

LWC Effective radius Optical depth CloudSat PIA Assimilate also CloudSat PIA

Will this work with EarthCARE? Simulated retrieval of optical depth for idealized adiabatic clouds, using spaceborne lidar with varying field of view (FOV) For FOV less than around 50 m, there is simply too little multiple scattering signal to retrieve extinction and optical depth Will need to rely more on radar PIA over ocean and solar radiances in the day Night-time land a problem FOV >= 55 m (e.g. Calipso) FOV <= 50 m (e.g. EarthCARE)

Why can the first guess matter? Consider a cloud with an optical depth of 50 If the first guess had an optical depth of 1 then the simulated molecular scattering below the cloud would look a bit like the measured multiple scattering Algorithm has difficulty getting over hump in cost function because increasing optical depth first reduces simulated backscatter below cloud top (leading to poorer agreement with obs) before multiple scattering builds up (leading to better agreement) Truth First guess

Possible solution Consider all possible true optical depths (but only triangular profiles so that profiles can be described uniquely by optical depth) Algorithm will converge provided first guess is outside the shaded areas Should be able to pre- analyse the profile (e.g. by integrating the backscatter with height) to tell if we are in the low or high optical depth regime, then set the first guess appropriately Previous plot considered true optical depth of 50

Potential optimization We need to speed-up the retrieval algorithm –Can we exploit parallel architectures, e.g. multicore machines or GPUs? Trade-off between minimization schemes: –Quasi-Newton (L-BFGS) Uses only the gradient of the cost function, which is fast to calculate Many iterations required –Levenberg-Marquardt (LM; more stable version of Gauss-Newton) Uses also the curvature of the cost function which is slow to calculate But few iterations required, and a little more robust (in my experience) Currently works for ice and rain, not yet for liquid Adept’s algorithm for computing the Jacobian matrix (needed by LM) is potentially parallelizable –“m” parallel threads, where “m” is number of observations (~100) –At best, the cost of an LM iteration would be the same as a quasi- Newton iteration, so LM would be much faster overall I am currently employing a programmer with GPU experience to code up a parallel Jacobian algorithm using CUDA (for NVIDIA hardware)

Example case Levenberg- Marquardt algorithm run on ice and rain region CloudSat and Calipso observations and forward model

Convergence comparison Quasi-Newton needs around five times more iterations on average (depending on convergence criterion) Levenberg-MarquardtQuasi-Newton

Convergence comparison cont. Levenberg-MarquardtQuasi-Newton Calipso CloudSat

Computational cost Computational cost (arbitrary) Proportional to number of iterations Potentially parallelizable Levenberg-Marquardt is already competitive but if Jacobian can be sped up it would be much faster than qausi-Newton Further change: perform wide-angle multiple scattering at half the vertical resolution would gain factor 4 speed-up

CloudSat EarthCARE CPR Z –Higher sensitivity CPR Z error CPR Doppler –Use Japanese random error CPR Doppler error Unified retrieval of cloud +precip …then simulate EarthCARE instruments

Calipso backscatter ATLID Mie channel –Note liquid! ATLID Mie error –Not rigorous! ATLID Rayleigh channel ATLID Rayleigh error Unified retrieval of cloud +precip …then simulate EarthCARE instruments Liquid cloud

Compare ice retrievals A-Train retrieval Pseudo- EarthCARE retrieval Assimilate Doppler and HSRL (Some difference due to lidar ratio not being carried between retrieval and simulation) Extinction Number concentration Extinction Number concentration

Compare liquid clouds and rain Liquid water content Rain rate Liquid water content Rain rate A-Train EarthCARE –Poor LWC: not enough lidar multiple scattering!

Remaining algorithm development Minimization –Parallelize Jacobian calculation on GPU and compare speed of Levenberg- Marquardt to quasi-Newton Forward models –Finish implementation of LIDORT solar radiance model Ice clouds –Add “riming” factor –Add Baran phase functions where appropriate Liquid clouds –Test impact of solar radiances on retrievals –Test size retrieval from two solar wavelengths Rain –Test impact of various observations (PIA, radar multiple scattering) Aerosols –Implement an aerosol retrieval scheme (contract extension)