Impact of Observations – recent studies Jean-Noël Thépaut credits: ECMWF staff, including Tony McNally, Stephen English, Alan Geer, Cristina Lupu (strong NWP and DA bias…) (largely inspired from a recent seminar at ECMWF  Tony McNally)

Overview What do we mean by impact (analysis / forecasts) What factors influence impact ? What diagnostics are available to measure impact and are they reliable ? An assessment of current satellite impact on NWP forecasts Final note on the forthcoming OSE/OSSE workshop

WMO Integrated Global Observing System Courtesy: WMO ECMWF Preview – DTS TWP - ICE Supported by field campaign experiments, Data targeting studies, etc. WMO plays a crucial role of course to coordinate, integrate and sustain the global observing system, with a number of commissions, expert teams and processes in place to ensure that the design of the global observing system has a strong bottom-up dimension. And in addition, the science is supported by a number of continual field campaign experiment, data targetting studies, etc.

(big Volume, big Velocity, big Variety) 5.5 more instruments per year THIS IS BIG DATA! (big Volume, big Velocity, big Variety) This plot represents the evolution of the number of satellite data products monitored at ecmwf for the past 18 years. Obviously the days where one specialist looked after assimilation from 1-2 instruments are long since over… The volumes involved are not necessarily linear but what is clear anyhow is that we are talking big data here, with big data volumes, big velocity in terms of retrieving and archiving requirements, and big variety of products

Cloud Radar Reflectivity Models and observation operators have become much more realistic and accurate CLOUDSAT Continual improvement of ECMWF short-range precipitation forecasts with respect to ground-based radar data. Cloud Radar Reflectivity These improvements open new opportunities to explore more observations A second and third examples show on the left the model representation of cloud radar reflectivity and the associated cloudsat measurements. Here again the similarity is becoming excellent, and on the right, the plot shows the improvement of short-range precipitation forecasts over the last twelve years as compared with a ground based radar network (NEXRAD) over the US. Being able to depict so accurately observables for quantities such as moisture, clouds an precips is of course relevant for extreme weather prediction

Overview What do we mean by impact (analysis / forecasts) What factors influence impact ? What diagnostics are available to measure impact and are they reliable ? An assessment of current satellite impact on NWP forecasts

What do we mean by impact ? Traditionally: Range (12hrs, 5 days, 10 days…) Parameter (gp height, wind, temperature, humidty …) Altitude (surface, 500hPa, 1hPa) Region (global, NH, SH, Tropics, Europe) Note: Observation impact has a sign ….it can be good and bad!

Overview What do we mean by impact (analysis / forecasts) What factors influence impact ? What diagnostics are available to measure impact and are they reliable ? An assessment of current satellite impact on NWP forecasts How satellite data impact model development

Factors that determine impact ? Observation quality Observed quantity (important ? already known?) Observation usability (ambiguity) Observation spatial coverage Observation time Tuning of the assimilation system (correct specification of B, R, BC, QC)

Factors that determine impact ? Observation quality Observed quantity (important ? already known?) Observation usability (ambiguity) Observation spatial coverage Observation time Tuning of the assimilation system (correct specification of B, R, BC, QC)

Mechanism: Dynamical adjustments in 4DVAR Assimilation of a single 183 ±1 GHz observation Start Time of observation Assimilation window UTH increment (200-500 hPa mean RH) Humidity reduction at observation time generated by changes in wind (and other dynamical variables) 1000km away, 9h earlier! Zonal wind increment at 400 hPa

No impact of USPS over USA !

Figure 2. Orbits of the EPS and USPS in the 00z (left) and 12z (right) 4DVAR 12 hour assimilation window. Those coloured red (for USPS) and blue (for EPS) are observed in the lattermost 3 hours of the assimilation window.

Factors that determine impact ? Observation quality Observed quantity (important ? already known?) Observation usability (ambiguity) Observation spatial coverage Observation time Tuning of the assimilation system (correct specification of B, R, BC, QC)

Observation error specification: Impact on FG-departures for other observations AMSU-A, tropics Radiosondes - T, tropics GPSRO, global IASI inter-channel error correlation matrix A quick illustration here to show that observation error specification is extremely important. This is a study showing the impact of taking into account the error correlation between channels from the IASI interferrometer onboard METOP, (left panel) on the fit of the model first-guess to other independent observations. This impact is significant and general. < 100% : improved model first-guess fit to observations

Sandy: impact of background error specifications No Polar: EDA-based background error covariances Control No Polar: Brute force Sandy: impact of background error specifications If we withdraw the observations without informing the asimilation system, the 5 and 4 day forecasts become very poor (admittedly a very gross experiment). Point to illustrate, is that by specifying the appropriate background error, the system recovers partly the forecast by making a better use of other observations Four day forecasts of surface pressure launched from 26th October (left) and five day forecasts from the 25th October (right) for the control (grey), NOPOLAR (red) and NOPOLAR-EDA (blue). Contours at 10hPa intervals with shading below 970hPa).

Overview What do we mean by impact (analysis / forecasts) What factors influence impact ? What diagnostics are available to measure impact and are they reliable ? An assessment of current satellite impact on NWP forecasts

Diagnostics Available Observing System Experiments (OSE) Denial or addition experiments Periodic statistical evaluations Case studies Adjoint Sensitivity Diagnostics (ASD) Impact assessed without denial Case studies ?

Pros and Cons of OSE Extremely (prohibitively?) expensive to run long periods (needed for small signals) Adding or denying a data type may require background errors to be retuned* Verifying short-range forecasts is less reliable The only measure of medium-range observation impact They give the only clear definitive answer to the question “what if I did not have this satellite ?”

Diagnostics Available Observing System Experiments (OSE) Denial or addition experiments Periodic statistical evaluations Case studies Adjoint sensitivity Diagnostics (ASD) Impact assessed without denial Case studies ?

Forecast impact of all-sky assimilation of microwave WV Compared to clear-sky assimilation Change in vector wind RMS error, NH, 500 hPa SH Tropics NH All-sky MW WV – No MW WV Clear-sky MW WV – No MW WV

OSEs for surface-sensitive MW sounder data over land/sea-ice Experiments over 8 months: 2 June – 30 Sept 2014; 2 Dec 2014 – 31 March 2015 Base: No surface-sensitive MW sounder data over land and sea-ice, otherwise as operations Base + MW seaice: Add surface-sensitive MW sounders over sea-ice Base + MW seaice + MW WV land: Add MW humidity sounders over land Base + WV seaice + MW land: Add surface-sensitive MW temperature sounders over land (ie, data usage as in operations) MW humidity sounder channels: ATMS ch18-22 (clear-sky) – 1 instrument MHS ch 3-5 (all-sky) – 4 instruments SSMIS ch 9-11 (all-sky) – 1 instrument Surface-sensitive MW temperature sounder channels: ATMS ch 6-8 (clear-sky) - 1 instrument AMSU-A ch 5-7 (clear-sky) - 5 instruments

Impact of surface-sensitive MW sounders over sea-ice and land, respectively Base + MW seaice + MW land Base + MW seaice vs Base

Impact of surface-sensitive MW humidity and temperature sounders over land Base + MW seaice + MW land Base + MW seaice + MW WV land vs Base + MW seaice

Giovanna De Chiara

Impact of observations in coupled systems: A case study Laloyaux et al 2015b Tropical Cyclone Phailin, Bay of Bengal Assimilation of scatterometer wind data

Impact of observations in coupled systems: A case study Laloyaux et al 2015b Tropical Cyclone Phailin, Bay of Bengal Assimilation of scatterometer wind data Verification against Argo data Ocean temperature at 40m depth: Impact of surface wind observations uncoupled coupled 2K scatt impact Argo Time series of ocean temperature observations at a depth of 40 metres by the Argo float 2901335(Observations) with (a) the temperature analyses produced by the CERA system with scatterometer data assimilation (CERA-SCATT) and without scatterometer data assimilation (CERA-NOSCATT); and (b) the temperature analyses produced by the UNCPL system with scatterometer data assimilation (UNCPL-SCATT) and without scatterometer data assimilation (UNCPL-NOSCATT). 5 days

Diagnostics Available Observing System Experiments (OSE) Denial or addition experiments Periodic statistical evaluations Case studies Adjoint Sensitivity Diagnostics (ASD) Impact assessed without denial Case studies ?

Adjoint Sensitivity Diagnostics E(36hr) E(24hr) adjoint of forecast model

Pros and Cons of ASD Can only operate a short-range where verification is least reliable Problems relating TE metric to parameters and consistency of adjoint model (dry/wet) Poor observation error tuning can produce misleading results* Very affordable (compared to OSE) Allows detailed evaluation of individual channel impacts

Forecast sensitivity to observations (Oct 2015) (mostly all-sky) (all-sky)

The WMO 6th Workshop on the Impact of Various Observing Systems on NWP Shanghai, China May 10-13 2016 organised by the Inter Programme Expert Team on the Observing System Design and Evolution (IPET-OSDE) Will review recent changes and trends: changes and developments have affected the global observing system more efforts have been devoted to meso-scale observing and assimilation systems. Trend toward using techniques other than OSEs to document data impact, such as adjoint- and ensemble-based forecast sensitivity observation impact (FSOI and EFSOI) estimates of analysis uncertainty.