1. Results from the THORPEX Observation Impact Inter-comparison Project
Ron Gelaro1, Rolf Langland2, Simon Pellerin3, Ricardo Todling1
1NASA Global Modeling and Assimilation Office (GMAO)
2Naval Research Laboratory (NRL)
3Environment Canada (EC)
Third THORPEX International Science Symposium
Sept 2009 – Monterey, CA

2. Background / Motivation
A goal of THORPEX is to improve our understanding of the ‘value’ of observations provided by the current global network
optimize the use of current observations
inform the design, deployment of new obs systems
DAOS WG proposed, as a starting point, a comparison of observation impacts in several forecast systems, facilitated by the emergence of new (adjoint-based) techniques
Experiments for a ‘baseline’ observation set were designed by NRL, NASA/GMAO, EC, ECMWF and Météo France
…this talk presents results from 3 systems: NRL, GMAO, EC
Acknowledgements: Carla Cardinali (ECMWF), Pierre Gauthier (EC, UQAM), Florence Rabier (Météo France)

3. The Challenge
With millions of observations assimilated every analysis cycle, how do we quantify the value provided by these data?
QUESTIONS:
How similar or different are the impacts of observations in one forecast system vs. another?
Which observation types have the largest total impacts, and impacts per-observation?
How do observation impacts vary as a function of location, channel, or other relevant attribute?
How is it possible to determine if particular observations or groups of observations are adding value to a weather forecast?

4. ◄ Methods for Evaluating Observation Impact
DATA DENIAL / ADDITION EXPERIMENTS (OSEs)
Typically done for a few sets of observation types
Not feasible to examine impact of ALL observations
Valid for any forecast range or measure
ADJOINT-BASED OBSERVATION IMPACT
Can evaluate impacts of all observations on a selected measure of short-range forecast error
Computationally efficient and accurate, but subject to assumptions and limitations in adjoint model
Method described in Langland and Baker (Tellus 2004) and subsequent papers by Errico 2007; Gelaro et al. 2007 ◄
Methods compared by Gelaro and Zhu (2009), Cardinali (2009)

5. Definition of Observation Impact
Energy-weighted forecast error norms
Observation impact is quantified as the difference in forecast error norms
It may be estimated as a sum of contributions from individual obs using information from the model and analysis adjoints
observations assimilated  
“Model error” does not affect this estimate of observation impact
t  6h
t = 0
t + 24h
Assimilation of observations moves the model state from the background trajectory to the new analysis trajectory 5

6. Calculation of Observation Impact
Based on LB04, or an extension of that method
adjoint analysis scheme
innovations
where
adjoint model
Impact computed for all observations simultaneously
The impact of any subset of observations, S, can be easily quantified using a partial sum

7. Experiment for the Baseline Observation Set
January 2007, every 6 hours, 124 assimilation times
Baseline set of observations assimilated (next slide)
24h global forecast error - Dry total energy norm
Dry physics in adjoint model
NRL – NOGAPS forecast and adjoint T239L30 NAVDAS 3D-Var analysis and adjoint 0.5°
GMAO – GEOS-5 forecast 0.5°, adjoint 1.0° GSI 3D-Var analysis and adjoint 0.5°
EC – Meso-Strat GFS forecast 0.4°, adjoint 1.5° 4D-Var analysis and adjoint 1.5°

8. Observations in Baseline Comparison
Conventional observations
Raobs, pibals, dropsondes
Commercial aircraft
Land surface
Ship surface
Buoys
Satellite observations
AMSU-A (3 satellites)
Geo-satellite winds
Vis, IR and WV
MODIS polar winds
SSM/I surface wind speed
QuikScat surface wind
 Each center used their usual data selection and QC procedures, error statistics and observation operators
 NRL and GMAO assimilated SSM/I speeds but no Profiler winds; EC did the opposite

9. Time series of forecast error norms
Global domain: All assimilation times Jan 2007
GEOS-5
NOGAPS
Forecast error on Background Trajectory,
Forecast error on Analysis Trajectory,
Difference of nonlinear forecast error norms
Adjoint-based estimate of global observation impact

10. Daily average observation impacts
Global domain: UTC assimilations Jan 2007
GEOS-5
NOGAPS 
FCST ERROR REDUCTION
EC-MSGFS
AMSU-A, Raob, Satwind and Aircraft have largest impact in all systems
Smaller impact of Satwind in GEOS-5 than in other systems
FCST ERROR REDUCTION

11. Impacts per-observation
Global domain: UTC assimilations Jan 2007
GEOS-5
NOGAPS
>30! 
FCST ERROR REDUCTION
EC-MSGFS
GEOS-5 has smaller impact per-ob, because more observations are assimilated (next slide) …recall TOTAL impacts are similar
Very large impact per-ob for Ships in EC system is an outlier
FCST ERROR REDUCTION

12. Global domain: 00+06 UTC assimilations Jan 2007
Observation Counts
Global domain: UTC assimilations Jan 2007
GEOS-5
NOGAPS  
EC-MSGFS
GEOS-5 assimilates more obs overall, especially AMSU-A, Aircraft, QuikScat
NOGAPS assimilates more Satwinds

13. Fraction of obs that reduce forecast error
Global domain: UTC assimilations Jan 2007
GEOS-5
NOGAPS
EC-MSGFS
All observation types (except SSMI speeds in GEOS-5) are in the range of 50% to 54% beneficial

14. Scatter of observation impact vs. innovation
Global domain: 00 UTC assimilation 21 Jan 2007
GEOS-5
NOGAPS
N18 AMSUA Ch 7
N18 AMSUA Ch 7
FCST ERROR INCREASE
FCST ERROR REDUCTION
Most total forecast error reduction comes from observations with moderate-size innovations – not from outliers with very large positive or negative innovations
Raob T hPa
Raob T hPa
FCST ERROR INCREASE
FCST ERROR REDUCTION

15. Observation impacts: AMSU-A channels 4 -11
Global domain: UTC assimilations Jan 2007
GEOS-5
NOGAPS  
EC-MSGFS
Large impact from channels 5, 6 and 7 in all systems
Small impact of channel 8 in NOGAPS; significant impact of channel 9 in EC-MSGFS

16. Observation impacts: NOAA-18 channel 7
Global domain: UTC assimilations Jan 2007
GEOS-5
NOGAPS
Observations that produce large forecast error reductions
Observations that produce forecast error increases in both models …land or ice surface contamination of radiance data?

17. Summary of Comparison Results to Date
Comparison experiments for GMAO, NRL and EC systems completed for baseline set of observations
Despite differences in algorithms, RTMs and data handling, overall quantitative results similar for all systems; but details of impact differ (e.g., impact-per-ob, channels)
Largest impacts provided by AMSU-A and raobs (GMAO), AMSU-A and satwinds (NRL, EC); aircraft also has large impact in all systems
Only a small majority (50-55%) of assimilated observations improve the forecast…targeting ramifications?
Common problem areas with AMSU-A noted; handling of surface properties a likely cause
Future study to include other models(?) and more recent observation types…