1. Steps towards evaluating the cost-benefit of observing systems
Rebecca Reid and John Eyre Met Office, UK
WMO Impact Studies Workshop Shanghai, China
10-13 May 2016

2. Outline
Motivation
Approach
Some results
Discussion

3. Acknowledgements Richard Marriott Andrew Lorenc
Larry Morgan Mike Thurlow
Helen Buttery Malcolm Kitchen
Mike Molyneux Steve Stringer
Jim Trice Colin Chatters
Biz Sykes Jon Turton
Sarah North Dave Edwards

4. Motivation

5. The role of observations for National Meteorological Services
NMS
weather
and climate
services
observations
people
computing
“… behind every weather, water and climate condition forecast, every disaster mitigated, and every prediction debated, are the observational data”
(WMO RA-V, 15th Session, May 2010, General summary)
© Crown copyright 2007

6. What will WIGOS deliver?
Observing systems
“Applications”
Services
sondes
NWP
Weather
Water
Climate
IR+MW
climate monitoring
GPS-RO
other
other
strong links and
“clean” observational data sets
© Crown copyright 2007

7. Approach

8. The cost-benefit chain
benefit per impact
M$ cost
M$ benefit
impact per cost
observing system
Application Areas
services
users
direct users of observations
indirect users of observations
© Crown copyright 2007

9. Approach Long-term aim:
Quantitative evaluation of cost-benefit of observing systems
Short-term approach:
Measure Impacts of observations in global NWP using Forecast Sensitivity to Observations Impact (FSOI) metric
Costs of observing systems operated by the Met Office are known; costs of global observing systems estimated
These data used to calculate Impact per Cost for each observing system

10. Data assimilation 4D-Var
FSOI in Met Office global NWP
Analysis
Observations
Forecast
“error”
T+24 M
Data assimilation 4D-Var
Forecast
T+24 M
Forecast
T+30
Forecast “error”
T+30
Change in forecast error due to each observation !!
Change in forecast error (due to obs)
Adj.M
Change in forecast error due to analysis increments
© Crown copyright 2007

11. FSOI: data for January 2016 total impact, by observation type
© Crown copyright 2007

12. FSOI: data for April-July 2013 total impact, by observation type
© Crown copyright 2007

13. FSOI: data for April-July 2013 impact per observation
© Crown copyright 2007

14. FSOI: data for January 2016 impact per observation
© Crown copyright 2007

15. Impact per cost in global NWP
Observation categories – as WMO Vision for the GOS
Platforms … and their costs
Instruments … and their cost-fractions
Data types
Cost, for each obs-type
Impact, for each obs-type
 Impact per cost, for each obs-type
For more detail: Eyre J and Reid R. Cost-benefit studies for observing systems.
Met Office Forecasting Research Technical Report No.593, August 2014.
© Crown copyright 2007

16. Costs of observing systems for observations used in global NWP
Current estimates and guesses
Space-based
EUMETSAT programmes – estimates - to be improved
Other agencies – guesses – based on similar EUMETSAT programmes
Surface-based
Costs for networks funded or part-funded by UK
Numbers of observations: global total, UK-funded total
 UK costs scaled up to estimate global costs
For more detail: Eyre J and Reid R. Cost-benefit studies for observing systems.
Met Office Forecasting Research Technical Report No.593, August 2014.
© Crown copyright 2007

17. Network costs – Met Office, UK
Observation type
Direct costs (£K)
Services received (£K)
Total annual cost (£K)
Drifting buoy 35 10 45
Moored buoy
600
325
925
Ship
200
330
530
SYNOP station
750
3350
4100
Wind profiler
100 65
165
GPS water vapour 80
115
Radiosonde (land station)
550
500
1050
Observation type (programme)
Total annual contribution (£K)
Aircraft (E-AMDAR)
285
Ship radiosondes (E-ASAP)
200

18. Impact per cost (for Met Office global NWP)
Met Office costs v. impact of Met Office observations
Select the exact observations that we pay for
What about programme contributions and “free” data?
Met Office costs v. impact of global observations
Takes into account benefit from collaboration
“Cost to the world” v. impact of global observations
Guess “cost to the world” by scaling up known costs 18

19. Results

20. impact of Met Office observations
Met Office cost v.
impact of Met Office observations
*Two versions of same graph
*Three variables
*Note the bar size represents a percentage 20

21. impact of global observations
Met Office cost v.
impact of global observations 21

22. impact of global observations
‘Cost to the world’ v.
impact of global observations
Metop-A+B AMSU-A
Metop-A+B IASI
Satellites
aircraft
Ground-based
drifting buoys

23. Discussion

24. Limitations of FSOI approach
This FSOI metric measures:
impact of observations on 24-hour forecast
impact of observations in the presence of all other observations
Other uses of observations in global NWP:
Non-assimilated fields (SST, sea-ice, …)
Verification
Quality control of other observations
Bias correction of other observations
Use of observations for other applications
Nowcasting
Climate monitoring …
Shorten these slides

25. Other considerations
These results could guide further, more detailed studies of particular observation types
This analysis is presented as a first step towards evaluating “cost:benefit” of observing networks, bearing in mind:
limitations of the metric
limitations of cost estimates

26. Extension to other Applications Areas and to services
Global NWP has appropriate metrics – FSOI, OSE
Metrics needed for other Application Areas (Aas):
High-resolution NWP possible, but more challenging
Other AAs (Nowcasting, climate monitoring, …)
… likely to be less objective, but don’t worry – better than nothing!
From AAs to services
Challenging
Link to work quantifying benefits of NMSs
© Crown copyright 2007

27. Conclusions - next steps
Improve estimates/guesses of observing system costs,
Generalise the results to more than one NWP centre and more than one period
Promote development of appropriate metrics for other Application Areas
Extend impact-per-cost assessments to other parts of the cost-benefit chain,
… and eventually to an integrated assessment of cost-benefit over many applications and services
© Crown copyright 2007

28. Thank you! Any questions?

29. Observing networks – marine
Observation type
Met Office stations
Drifting buoys
The Met Office is responsible for ~25 annually (there are around 1250 worldwide)
Moored buoys
Network of 11 stations in UK waters
Ships
Approximately 250 manual VOS and 40 AWS equipped ships were supported by the Met Office and operational during the period of study
Moored buoys
Drifting buoys
Ships

30. Observing networks – land
Observation type
Met Office stations
SYNOP stations
Approximately 290 land stations in UK network
Wind profilers
7 stations in UK network
GPS water vapour
Approximately 120 stations in UK network
Radiosondes (land stations)
6 stations in UK network (excluding defence stations providing irregular soundings)
SYNOP
Wind profilers
GPS water vapour
Radiosondes

31. Observing networks – programmes
E-AMDAR:
Aircraft observations
E-ASAP: Ships providing radiosonde ascents