Assessing the impact of observations on numerical weather forecasts Ron Gelaro NASA Global Modeling and Assimilation Office JCSDA Summer Colloquium, 4 August

Data Assimilation Use available observations together with a model forecast to provide the best estimate of the state of a physical system …maximize the utility of BOTH 2 4-D data assimilation with a 6-hr update cycle Observations provide information about “reality” but are disparate and irregular in space and time Models provide regular, physically consistent information about the system, but are prone to systematic errors Graphic, ECMWF

Surface-based ~11,000 land stations observing at least 3-hrly ~4,000 VOS with ~1,000 reports daily ~1,200 drifting buoys (~14K SLP and 27K SST daily) Upper Air ~1,300 stations with >1,500 reports daily ~300,000 aircraft reports daily 3 Components of the WMO Integrated Observing System The World Meteorological Organization(WMO ) Satellites ~268 Earth observation satellites ~413 instruments Tens of millions of obs daily

Observation Count (millions) IASI-A IASI-B Data Assimilation in the Era of Hyperspectral Satellites Observations analyzed per 6-hr cycle in the GEOS-5 data assimilation system AIRS

5 Observations used in the GEOS-5 analysis on 10 Dec UTC 6-hr assimilation time window Will’s obs movie was here....

With millions of observations assimilated every analysis cycle, how do we quantify the value provided by all these data? Which observation types provide the largest total impacts, or largest impact per observation? …This talk focuses on using the adjoint (transpose) of a data assimilation system but other methods are commonly used such as data-denial or observing system experiments (OSEs) How do impacts vary by location, channel or other attribute? Do all observations provide benefit? 6

Observing System Experiments – OSEs (Data Denial Experiments) Subsets of observations are removed (or added) to the data assimilation system to assess their impact Long history, run intermittently at most NWP centers Valid for any forecast range or measure Costly, requires re-running the data assimilation system for each subset of observations examined Observing System Experiments Kelly et al. (2004), ECMWF Anomaly Correlation NH ACC 500 hPa Geop Ht SH ACC 500 hPa Geop Ht 7 skill time

Consider a model forecast: Note that transforms a perturbation in observation space to a perturbation in model (state) space System Definitions 8 A quadratic measure of forecast error, where is a norm and is a verification state: An atmospheric analysis that produces the best (linear) estimate of : where is the background forecast, is a weighting (gain) matrix, are the observation-minus-background departures, and is the analysis increment.

observations assimilated 9 0 h -6 h +24 h Time Fcst Error Observation Impact Measure following Langland and Baker (2004) The difference measures the collective impact at 24 h of all observations assimilated at 0 h. (model space) Can we measure their individual contributions? (observation space)

Apply the definition of an adjoint (transpose) Recall the definition of the analysis increment Introduce the Adjoint Operator following Errico (2007) Then for any vector in model space, there is a corresponding vector in observation space such that 10 where is a vector and is the inner-product. model space observation space

Use Taylor series to express changes in e due to changes in : Express δe in Observation Space following Errico (2007) adjoint forecast model adjoint analysis scheme Summation of individual observation impacts 11 Now substitute definitions of... and do algebra to obtain the analogous expression in observation space:

Properties of the Impact Estimate 3 rd order approximation provides sufficient accuracy (Gelaro et al. 2007) where The vector is computed once for all observations simultaneously (it can be thought of as a set of weights acting on the departures ) The impact of any subset of observations, S, can be easily quantified using a partial sum δe 0) implies the observation improves (degrades) the forecast Only valid for forecasts up to 2-3 days (for synoptic scale motions) due to the tangent linear assumption required for M T 12

13 0 h -6 h +24 h Time Fcst Error Forward and Adjoint Model Integrations This schematic should not to be taken too literally! Note that the initial conditions for the adjoint integrations are: respectively, which are difficult to depict graphically here. and,

Adjoint Analysis System Adjoint Forecast Model Output: Forecast Sensitivity to Observations and Background Forecast Sensitivity to Initial State Input: Forecast (measure) Forward Data Assimilation-Forecast Procedure Adjoint Data Assimilation-Forecast Procedure Observation impact Analysis System invisible Forecast Model invisible Input: Observations and Background Initial State Output: Forecast Analysis System Forecast Model Operational Procedure for Forward and Adjoint DAS The computational cost of the adjoint system is roughly the same as that of the forward data assimilation system Km KTKT MTMT 14

GEOS-5 atmospheric data assimilation system: provides real time analyses and forecast support for NASA instrument teams, field campaigns and other science users GEOS-5 AGCM (~¼° L72) + GSI analysis (~½° L72) 6-h assimilation cycle, 3D-EnVar Hybrid (1° x 32 members) NCEP GDAS observation set (~4.5 million obs/6h) 5-day forecasts at 00z and 12z Adjoint-based observation impacts: computed daily for the forecast initiated at 00z Global 24-h forecast error measure Moist total energy norm (u, v, T, p s, q J/kg) Adjoint physics include PBL, moist convection, cloud microphysics (Holdaway et al. 2014, 2015) 15 GMAO GEOS-5 Observation Impact Monitoring

Time Series of Observation Impacts 3-Month Time Series for 24h Forecasts Initialized at 00z Radiosonde u,v,T,q 16 Aircraft u,v,T MetOp-A AMSU-A Tb Drifting Buoy Ps more beneficial Negative values are beneficial; color shading denotes magnitude, units are J/kg

Seasonality of Observation Impact 1-Year Time Series for 24h Forecasts Initialized at 00z 17 NOAA-18 AMSU-A Tb Negative values are beneficial; color shading denotes magnitude NOAA-18 AMSU-A Tb Northern Hemisphere Southern Hemisphere Jan Jul

Ranked Summary of Observing System Impacts 1-Year Average for 24h Forecasts Initialized at 00z Total ImpactImpact Per Observation AMSU-A radiances have the largest impact globally; satellite data very important overall but conventional data (radiosondes, aircraft) also still important Conventional data that are few in number can have large individual impact; dense satellite data tend to have small individual impacts beneficial Color shading denotes observation count 18

Targeted Dropsondes Impact on a Per-Observation Basis Dropsonde TempsMetOp-A IASI Brightness Temps 19 0 Rejected1 Passive21 Used 371,731 Rejected3,866,967 Passive734,270 Used Impact per observation 02 June J Kg -1 GEOS-5 Operational Data Coverage 02 June UTC GEOS-5 Observation Impact Time Series

Ranked Summary of Observing System Impacts by Hemisphere 1-Year Average for 24h Forecasts Initialized at 00z Total Impact - N.HemTotal Impact - S.Hem Radiosondes have largest impact, but other conventional and satellite data also have large impact (AMSU-A, aircraft, IASI) Satellite data dominate, especially AMSU-A and IASI. Contribution from radiosondes (and other data types) is still significant. 20

Raob T-500 AMSU-A Ch.6 Tb Map Views of Observation Impacts 3-Month Average (MJJ 2015) for 24h Forecasts Initialized at 00z Aircraft u-250 Drift Buoy Ps Global sums are beneficial (negative), but many obs degrade the forecast! improve degrade 21 Average values in 2° lat-lon bins (10 -6 J/kg)

beneficial NOAA-19 AMSU-A Ch.5 Scatter of Impact versus δ y The numbers of observations that improve or degrade the forecast are both large Most of the total error reduction comes from a large number of observations with small or moderate individual impacts non-beneficial What fraction of the assimilated observations improves the forecast? Fraction of Beneficial Observations For almost all data types, only a small majority of observations (50-55%) improve the forecast! …a robust result for modern DA systems (Gelaro et al. 2010) 22

How can ‘good observations’ have a negative impact? More accurate backgrounds More accurate observations Ratio (Obs Error/Bkg Error) Probability of Degrading the Background Forecast in a Single-Ob Scalar Analysis The (fundamental) statistical nature of data assimilation Reliance on statistics of background and obser- vation errors implies a distribution of positive and negative impacts, regard- less of data quality Assimilating an observation reduces background error variance, but this does not imply the analysis is closer to the true state on a case- by-case basis Monte Carlo calculation by Mike Fisher, ECMWF 23

Pressure (hPa) Weight AMSU-A AIRS Satellite Radiance Observations The introduction of hyper-spectral sounders has increased the number of available channels (observations in a single profile) from O(10) to O(1000) 24 Weighting functions (channel sensitivity profiles) for AMSU-A and AIRS instruments 15 channels c channels c. 2002

MetOp-B AMSU-A (Microwave) Channels 4-9 are high quality and have largest positive impact. Channels 1-3, 15 have large surface sensitivity, not used at GMAO MetOp-B IASI (Hyperspec IR) Channels sensitive to tropospheric temper- ature and have significant positive impact. Other channels sensitive to surface and water vapor more difficult to use. 25 Examples of Impact of Satellite Radiances by Channel 3-Month Average for 24h Forecasts Initialized at 00z

Example Impact Map of Satellite Radiances by Channel 3-Month Average (MJJ 2015) for 24h Forecasts Initialized at 00z Average values in 2° lat-lon bins (10 -6 J/kg) MetOp-B IASI Ch.249 Sensitive to tropospheric temperature has significant overall beneficial impact MetOP-B IASI Ch.141 Also sensitive to tropospheric temperature but has a non- beneficial overall impact during this period...(?) 26

ADJ: measures the impacts of observations in the context of all other observations present in the assimilation system OSE: measures the impact of observations by their removal from the system; the system is changed (degraded) in each experiment ADJ: measures the impact of observations in each analysis cycle separately and against the control (fully observed) background OSE: measures the impact of removing information from both the background and analysis in a cumulative manner ADJ: measures the response of a single forecast metric to all observational “perturbations” to the system OSE: measures the effect of a single observational perturbation on all possible forecast metrics   Comparing/Interpreting Adjoint and OSE Results A few things to keep in mind Both OSEs and the ADJ method measure aspects of the net effect of observations on the forecast, however... 27

Complementary Use of OSEs and the Adjoint Method We are also interested in dependencies and redundancies between observing systems as observations are added or removed from the assimilation system These responses can be examined through the combined use of OSEs and the ADJ method (Gelaro and Zhu 2009) 28

29 10 Dec 2010 – 31 Jan z N.Hem Obs S.Hem Obs Compensating effects of removing conventional observing systems are largest in the NH Removal of raobs increases fractional impact of aircraft and AMSU-A (>60%); removal of aircraft increases fractional impact of raobs (>30%) Relative Impacts with Selected Conventional Data Types Removed

30 10 Dec 2010 – 31 Jan z Compensating effects of removing satellite radiances are largest in the SH Removal of AMSU-A doubles the fractional impact of hyperspectral IR; removal of hyperspectral IR increases the fractional impact of AMSU-A (>35%) Relative Impacts with Selected Satellite Data Types Removed N.Hem Obs S.Hem Obs

Closing Remarks Adjoint data assimilation system provides an accurate and efficient tool for estimating observation impact on short-term forecasts computed with respect to all observations simultaneously permits arbitrary aggregation of results by data type, location, etc. Well suited for routine monitoring of the observing system: now used by Navy/FNMOC, GMAO, ECMWF, Met Office, others... Satellite observations, especially radiance data, are critical for global NWP, but conventional data remain very important ADJ method complements, not replaces, OSEs as tools for assessing observation impact…metrics, interpretations differ Used together, the ADJ method and OSEs illuminate the complex, complementary nature of how observations are used by the assimilation system Only a small majority (50-54%) of observations improves the forecast, and most of the overall benefit comes from a large number of observations having small-moderate impacts: …implications for data volumes, targeted observing strategies, etc 31

32 Errico, R. M Interpretation of an adjoint-derived observational impact measure. Tellus 59A, 273–276. Gelaro R., Langland R. H., Pellerin S., Todling R., 2010: The THORPEX observation impact intercomparison experiment. Mon. Weather Rev. 138, 4009–4025. Gelaro, R. and Y. Zhu, 2009: Examination of observation impacts derived form observing system experiments (OSEs) and adjoint models. Tellus, 61A, 179–193. Gelaro, R., Y. Zhu and R. M. Errico, 2007: Examination of various-order adjoint-based approximations of observation impact. Meteorologische Zeitschrift, 16, Holdaway, D., R. Errico, R. Gelaro and J. G. Kim, 2014: Inclusion of linearized moist physics in NASA’s Goddard Earth Observing System data assimilation tools. Mon. Wea. Rev., 142 (1), 414–433. Holdaway, D., R. Errico, R. Gelaro, J. G. Kim and R. Mahajan, 2015: A linearized prognostic cloud scheme in NASA’s Goddard Earth Observing System data assimilation tools. Mon. Wea. Rev., in press. Citations I

33 Kelly, G., McNally, T., Thépaut, J.-N. and Szyndel, M OSEs of all main data types in the ECMWF operation system. In: Proceedings of ThirdWMOWorkshop on the Impact of Various Observing Systems on NumericalWeather Prediction, Alpbach, Austria, WMO/ TD No. 1228, 63–94. Langland, R. H. and Baker, N Estimation of observation impact using the NRL atmospheric variational data assimilation adjoint system. Tellus 56A, 189–201. Citations II