1. Assessing the Impact of Aircraft Observations on Model Forecasts
Geoff Manikin (NOAA/NCEP/EMC)
Materials from
Andrew Collard
P. Pauley, R. Langland (NRL)
C. Hill
J. Jung (U. Wisconsin)
R. Gelaro (GMAO)
S. Larouche, R Sarrazin (Environment Canada)
C. Parrett, B. Ingleby (Met Office)
H. Bénichou, N. Boullot (Météo-France)

2. Introduction
The ultimate purpose of assimilating any kind of observation into NWP models is to improve the model forecast.
Robust methods for determining the impact of the observations are therefore required. Operational centers typically have a wide range of tools available for this.
The initial aim in any data assimilation system is to use the observations in such a way as to improve the accuracy in the analysis. Sometimes these improvements are not directly translated into increases in forecast skill due to the characteristics of the forecast model itself.
In general this presentation is considering aircraft observations of temperature, humidity and wind vector together.

3. for GDAS 00 UTC Assimilation Cycle
Conventional Data Received by NCEP
Average Number of Observations Received Daily
for GDAS 00 UTC Assimilation Cycle
300
Total AMDAR: % %
200
COUNT
× 1000
100
2003
2007
2011
2015
YEAR
Source: APSDEU-14 / NAEDEX-26 – NCEP Update – October 2015 3

4. Data Denial
Data denial or Observation System Experiments (OSEs) are simply a way of investigating the impact of an observation or change by running full forecast experiments with and without the element to be tested.
OSEs are expensive to run, particularly at full operational resolution, and they need to be run for many forecast cycles (60 days is a typical number for global forecast systems) before statistically significant results are obtained.
Individual case studies are generally not trusted as a way of demonstrating forecast impact because of the dominance of statistical fluctuations.
Forecast impact scores are generally presented with error bars indicating statistical significance.
Score are normally given in terms of differences between forecasts and “truth” in terms of RMS error or anomaly correlation coefficients

5. Forecast Sensitivity to Observations (FSO)
Observations move the forecast from the background trajectory to the trajectory starting from the new analysis xb xg
t= -6 hrs eg
30-hr fcst xt xf xa
t= 24 hrs
t=0 ef
In this context, “OBSERVATION IMPACT” is the effect of observations on the difference in forecast error norms C(ef-eg)
24-hr fcst
“Background”
“Truth”
Earlier papers by Doerenbecher and Bergot (2001) and Fourrié et al. (2002) used similar impact functions – involving the innovation vector and observation sensitivity gradient
“Analysis”
6 hr assimilation window
Langland and Baker (Tellus, 2004), Gelaro et al (2007), Morneau et al. (2006)

6. Advantages and Disadvantages of FSO
Can infer the impact of observations to whatever level of detail is required (e.g. ob by ob, channel by channel) without having to re-run the full system repeatedly.
Useful for determining relative impact of observations and for quality control of bad observations.
Allows the impact of observations on the forecast to be monitored on a daily basis.
Disadvantages
Limited to short-range forecasts
- sensitivity to the accuracy of the verifying analysis
Impact is always in the context of the total observing system as used
- forecast impacts of an observation type may change as other
observations are added/removed.
Impact is insensitive to the information contributing to the forecast skill that was assimilated before the current analysis (and therefore contributing to the background state).

7. GLOBAL NWP DATA IMPACTS

8. ENSEMBLE FSO at NCEP Aircraft obs have provided the
greatest reduction of NWP fcst
error due to their great number,
wide spaital distribution, and
overall good quality
Singular aircraft observations are highly impactful due to their unique placement in time and space
Radiosonde and GPSRO data provide more impact due to their more balanced presence in the vertical space.
LOW
MID
HIGH
LOW
MID
HIGH
RAOB impact greatest in
lower/mid troposphere
Aircraft obs impact greatest
in upper troposphere

9. RMSE differences for Z500 Impact of aircraft reports
Impact of radiosonde data 0h
24h
48h
Environment Canada

10. Impact of the main global observing systems on NWP in terms of gain in predictability
Z500 over Northern Hemisphere
BLACK – CONTROL
BLUE – NO RAOBS
RED – NO AIRCRAFT
Northern Hemisphere Extra-Tropics
12 hours
6 hours
A few hours
Case impact
Neutral
SSMI
AMV
SCAT
GPS-RO
AMSU/A
IASI
AIRS
Buoys
Aircraft
Radiosonde
~6 hours
Gain in predictability of the main global observing systems
Source: 4th WMO workshop on the impact of various observations on NWP (2008)
Current contributions to some parts of the existing observing system to the large-scale forecast skill at short and medium-range. Green colour means the impact is mainly on the mass and wind field. Blue colour means the impact is mainly on humidity field. The contribution is primarily measured on large-scale upper-air fields. The read horizontal bars give an indication of the spread of results among the different impact studies sofar available (for several observing systems the quantity of impact studies is too small and the spread is not quantified).
Results from data denial experiments
with the MSC global forecast system
and 3D-Var, January 2009
Environment Canada

11. U250 Forecast impacts (FI%) over regions of North America
Environment Canada

12. Impact at ECMWF: Bouttier & Kelly 2001
N HEMIS
S HEMIS
EUROPE
TROPICS

13. GMAO GEOS-5 24h Adjoint-Based Observation Impact January 2014 00Z
Total Impact
Impact Per Ob
Observation Count
% Benefical Obs
High/Lo=Above/below 400hPa
ASDAR=Non-US AMDAR

14. Global NWP Data Impacts: FSO at Météo-France

15. Global NWP Data Impacts: FSO at UKMET
Observation impacts at the Met Office – FSO results Contributions to the total observation impact on a moist 24-hour forecast-error energy-norm per day, averaged over the period 01 Apr - 31 Jul 2013 for the Met Office Global Model

16. Conclusions
Positive impact from aircraft observations has been demonstrated in short-range forecasts at global and regional scales
Forecast sensitivity experiments consistently show aircraft observations as one of the most important data types
FSO experiments show greater impact from aircraft observations higher in the atmosphere