Presentation is loading. Please wait.

Presentation is loading. Please wait.

EQC for Seasonal Forecasts

Published byPriscilla Harmon Modified over 6 years ago

Similar presentations

Presentation on theme: "EQC for Seasonal Forecasts"— Presentation transcript:

1 EQC for Seasonal Forecasts
C3S_51 Lot 3: EQC for Seasonal Forecasts Francisco Doblas-Reyes1,2 1 Barcelona Supercomputing Center (BSC) 2 Institució Catalana de Recerca i Estudis Avançats (ICREA)

2 Challenges of the EQC for seasonal forecasts
Adaptation: provide information for all kind of services. Consistency: build trust, ensuring a high degree of coherence across products. Innovation: transfer recent developments from research to operations. Efficiency: information should respond to users’ queries with a delay as short as possible. Targets: define data, products, verification, guidance, etc.

3 Some important elements:
C3S_51 Lot 3: What C3S_51 Lot 3 aims at developing a strategy for the evaluation and quality control (EQC) of the multi-model seasonal forecasts provided by the Copernicus Climate Change Service (C3S) to respond to the needs identified among a wide range of stakeholders. Some important elements: Users are consulted to identify needs in terms of products and EQC information Forecast data are quality controlled and forecast products (not forecast systems) are verified A quality assurance template is required to summarise the information A framework is designed that serves as a reference to the community

4 C3S_51 Lot 3: The strategy C3S_51 Lot 3 aims at developing a strategy for the evaluation and quality control (EQC) of the multi-model seasonal forecasts provided by the Copernicus Climate Change Service (C3S) to respond to the needs identified among a wide range of stakeholders. Quality assurance Assessment of user needs Technical assessment and KPIs Scientific assessment Data sources, standards and conventions Forecast quality prototype

5 C3S_51 Lot 3: Who Partner Nature Role BSC-CNS (ES) Main contractor
Coordination, data inventory and EQC framework and prototype Univ. Leeds (UK) Subcontractor Assess user requirements Meteoswiss (CH) Scientific quality assessment and gap analysis Predictia (ES) CDS requirements and development of the prototype Univ. Exeter (UK) Expert statistical advice IFCA-CSIC (ES) Downscaling

6 Assessing forecast quality Preferences for forecast quality format
C3S_51 Lot 3: The users Assessing forecast quality Beware of user fatigue Preferences for forecast quality format M. Soares (Univ. of Leeds)

7 C3S_51 Lot 3: Standards and metadata
All necessary metadata have been defined for the seasonal forecasts using all standards available (CHFP, SPECS). Similar standards could not be found for the reference observations. But beware Additional checks still necessary (data checker). xkcd.com

8 C3S_51 Lot 3: Scientific assessment
Forecast quality assessment Verification procedure of existing forecast systems based on scoring rules (e.g. RPS for multi-category probabilities, CRPS for ensembles) Products, and not data, are verified; forecast products are scrutinised Impact of lagged ensembles and adequate product generation Calibration All bias correction and recalibration methods effectively remove bias Added value of sophisticated methods (e.g. EMOS) small to inexistent due to limited hindcast length (and low skill) Multi-model combination No forecast system consistently outperforms others Multi-model combination is beneficial Avoid the temptation of identifying inadequate data sources to e.g. discard “bad” forecast systems. These points address questions raised by users that lead to new products In addition: Calibration is more important than combination for ensemble forecasts Construction of ‘best performing’ lagged ensemble: two most recent initializations appear to be enough for short lead times

9 C3S_51 Lot 3: Forecast quality assessment and calibration
Skill of JJA temperature from ECMWF SEAS5 + recalibration* Better than constant climatological forecast Worse CRPS of JJA near-surface temperature from ECMWF SEAS 5 initialized in May calibrated with the climate-conserving recalibration (CCR) and verified against ERA Interim for * CRPS of JJA near-surface temperature from ECMWF SEAS 5 initialized in May calibrated with the climate-conserving recalibration (CCR) and verified against ERA Interim for S. Hemri (Meteoswiss)

10 C3S_51 Lot 3: Forecast quality assessment and multi-model
Skill of JJA temperature from C3S multi-model* Better than constant climatological forecast Worse CRPS of JJA near-surface temperature from ECMWF SEAS 5 initialized in May calibrated with the climate-conserving recalibration (CCR) and verified against ERA Interim for * CRPS of JJA near-surface temperature from ECMWF SEAS 5, Meteo France System 5, MetOffice GloSea5 initialized in May CCR + weighted (RMSE) averaging of forecast PDF and verified against ERA Interim for S. Hemri (Meteoswiss)

11 C3S_51 Lot 3: Timeliness and provenance
Computing performance and product provenance are key. Forecast verification (ROC area) with performance analysis and workflow definition using SpecsVerification and easyVerification. J. Bedia, D. San Martín (Predictia)

12 C3S_51 Lot 3: Timeliness and provenance
Computing performance and product provenance are key. Users need to generate products from the forecasts. Users require bias adjustment. Computational cost estimates are fundamental for user satisfaction. R. Manzanas (IFCA-CSIC)

13 C3S_51 Lot 3: Timeliness and provenance
Prepare products for reproducibility, traceability, reuse and context METACLIP ensures the reproducibility of objects (NetCDF file, image) with human and machine-readable solution It uses an RDF-based semantic metadata model that builds the vocabularies on existing initiatives (e.g. VALUE, PROV) and stores information in JSON files. “De facto” the IPCC WGI standard for AR6. Visualisation of provenance information J. Bedia, D. San Martín (PREDICTIA)

14 Quality assurance template and reports
C3S_51 Lot 3: Where Quality assurance template and reports The quality information of the climate forecast datasets should be input into a quality assurance template to produce reports that are made available along with the datasets in the CDS. Reports include dataset documentation, quality flags and an independent assessment.

15 C3S_51 Lot 3 The contract has documented a metadata standard to feed the common data model and created a provenance solution for products provided comprehensive forecast quality assessment of products illustrated the benefits of bias adjustment and multi-model solutions demonstrated that scalability and computational efficiency of the solutions should be taken into account to ensure timeliness identified relevant user requirements Relevant gaps: 1) guidance and context for forecast products, 2) metadata, vocabularies and provenance governance, 3) observational uncertainty, 4) criteria to identify relevant forecast products …

Download ppt "EQC for Seasonal Forecasts"

Similar presentations

Measuring the performance of climate predictions Chris Ferro, Tom Fricker, David Stephenson Mathematics Research Institute University of Exeter, UK IMA.

Measuring the performance of climate predictions Chris Ferro, Tom Fricker, David Stephenson Mathematics Research Institute University of Exeter, UK IMA.
ECMWF long range forecast systems

ECMWF long range forecast systems
Enhancing Data Quality of Distributive Trade Statistics Workshop for African countries on the Implementation of International Recommendations for Distributive.

Enhancing Data Quality of Distributive Trade Statistics Workshop for African countries on the Implementation of International Recommendations for Distributive.
Short Course on Introduction to Meteorological Instrumentation and Observations Techniques QA and QC Procedures Short Course on Introduction to Meteorological.

Short Course on Introduction to Meteorological Instrumentation and Observations Techniques QA and QC Procedures Short Course on Introduction to Meteorological.
A Regression Model for Ensemble Forecasts David Unger Climate Prediction Center.

A Regression Model for Ensemble Forecasts David Unger Climate Prediction Center.
WMO UNEP INTERGOVERNMENTAL PANEL ON CLIMATE CHANGE NATIONAL GREENHOUSE GAS INVENTORIES PROGRAMME WMO UNEP IPCC Good Practice Guidance Simon Eggleston Technical.

WMO UNEP INTERGOVERNMENTAL PANEL ON CLIMATE CHANGE NATIONAL GREENHOUSE GAS INVENTORIES PROGRAMME WMO UNEP IPCC Good Practice Guidance Simon Eggleston Technical.
Climate Forecasting Unit Second Ph’d training talk Prediction of climate extreme events at seasonal and decadal time scale Aida Pintó Biescas.

Climate Forecasting Unit Second Ph’d training talk Prediction of climate extreme events at seasonal and decadal time scale Aida Pintó Biescas.
Climate Forecasting Unit Prediction of climate extreme events at seasonal and decadal time scale Aida Pintó Biescas.

Climate Forecasting Unit Prediction of climate extreme events at seasonal and decadal time scale Aida Pintó Biescas.
Barcelona, 2015 Ocean prediction activites at BSC-IC3 Virginie Guemas and the Climate Forecasting Unit 9 February 2015.

Barcelona, 2015 Ocean prediction activites at BSC-IC3 Virginie Guemas and the Climate Forecasting Unit 9 February 2015.
Quality assurance / Quality control system for the Greek GHG emissions inventory Yannis Sarafidis, Elena Georgopoulou UNFCCC Workshop on National Systems.

Quality assurance / Quality control system for the Greek GHG emissions inventory Yannis Sarafidis, Elena Georgopoulou UNFCCC Workshop on National Systems.
Interpretation of climate change using maps of climate analogues in the DRIAS climate services project Julien Lémond 1-2, S. Planton 1, M. Ha-Duong 3,

Interpretation of climate change using maps of climate analogues in the DRIAS climate services project Julien Lémond 1-2, S. Planton 1, M. Ha-Duong 3,
1 Designing Effective Programs: –Introduction to Program Design Steps –Organizational Strategic Planning –Approaches and Models –Evaluation, scheduling,

1 Designing Effective Programs: –Introduction to Program Design Steps –Organizational Strategic Planning –Approaches and Models –Evaluation, scheduling,
ENSEMBLES RT4/RT5 Joint Meeting Paris, 10-11 February 2005 Overview of the WP5.3 Activities Partners: ECMWF, METO/HC, MeteoSchweiz, KNMI, IfM, CNRM, UREAD/CGAM,

ENSEMBLES RT4/RT5 Joint Meeting Paris, February 2005 Overview of the WP5.3 Activities Partners: ECMWF, METO/HC, MeteoSchweiz, KNMI, IfM, CNRM, UREAD/CGAM,
12/10/15.  It is a Cross Life Cycle Activity (CLCA) that may be performed at any stage ◦ In fact, some part of it (e.g. risk analysis and management)

12/10/15.  It is a Cross Life Cycle Activity (CLCA) that may be performed at any stage ◦ In fact, some part of it (e.g. risk analysis and management)
Thoughts on Stewardship, Archive, and Access to the National Multi- Model Ensemble (NMME) Prediction System Data Sets John Bates, Chief Remote Sensing.

Thoughts on Stewardship, Archive, and Access to the National Multi- Model Ensemble (NMME) Prediction System Data Sets John Bates, Chief Remote Sensing.
Business Analysis. Business Analysis Concepts Enterprise Analysis ► Identify business opportunities ► Understand the business strategy ► Identify Business.

Business Analysis. Business Analysis Concepts Enterprise Analysis ► Identify business opportunities ► Understand the business strategy ► Identify Business.
Work Package 6 L2C Kick-off meeting Fontainebleau, March 7th 2006.

Work Package 6 L2C Kick-off meeting Fontainebleau, March 7th 2006.
United Nations Oslo City Group on Energy Statistics OG7, Helsinki, Finland October 2012 ESCM Chapter 8: Data Quality and Meta Data 1.

United Nations Oslo City Group on Energy Statistics OG7, Helsinki, Finland October 2012 ESCM Chapter 8: Data Quality and Meta Data 1.
Software Quality Assurance SOFTWARE DEFECT. Defect Repair Defect Repair is a process of repairing the defective part or replacing it, as needed. For example,

Software Quality Assurance SOFTWARE DEFECT. Defect Repair Defect Repair is a process of repairing the defective part or replacing it, as needed. For example,
The Role of International Standards for National Statistical Offices Andrew Hancock Statistics New Zealand Prepared for 2013 Meeting of the UN Expert Group.

The Role of International Standards for National Statistical Offices Andrew Hancock Statistics New Zealand Prepared for 2013 Meeting of the UN Expert Group.

Similar presentations

Ads by Google