1. HARMO13, 1-4June 2010, Paris, France1 http://ies.jrc.ec.europa.eu/ http://www.jrc.ec.europa.eu/ Institute for Environment and Sustainability Procedure for Air Quality Models Benchmarking P. Thunis, E. Georgieva, S. Galmarini FAIRMODE WG2 – SG4 activity

2. HARMO13, 1-4June 2010, Paris, France 2 Outline Objectives & Background Key elements of the procedure The Benchmarking service Usage of the procedure Workplan Contributions & links to other SG

3. HARMO13, 1-4June 2010, Paris, France 3 Objectives Develop a procedure (sequence of operations) for the benchmarking of AQ models in order to evaluate their performances and indicate ways for improvements. Support both model users & model developers in the implementation of the AQD (Assessment & plans) Provide an aid to policy bodies in their judgment on the quality of model results

4. HARMO13, 1-4June 2010, Paris, France 4 Background (I) - First Proposal for a Benchmarking Tool as a diagnostic instrument for checking quality of model results : WG2 Meeting November 2009 - Document “ Procedure for AQ models Benchmarking” sent out to SG4 participants April 2010 (uploaded on FAIRMODE web page) - Only a few feed-backs up to now…

5. HARMO13, 1-4June 2010, Paris, France 5 Background (II) Application types: AQ assessment and Planning Models included: from regional to local scale –But should provide data with sufficient spatial and temporal resolution. Focus on pollutants considered in the AQ Directive (NO2, PM and O3) depending on the spatial scale addressed.

6. HARMO13, 1-4June 2010, Paris, France 6 Background (III) REFERENCES BOOT software (Chang and Hanna, 2005) ASTM Guidance (ASTM, 2000) USA-EPA AMET package (Appel and Gilliam, 2008) EPA Guidance (2007, 2009) AIR4EU conclusions (Borrego et al. 2008) CityDelta and EuroDelta projects ENSEMBLE platform (Galmarini S. et al. 2001, 2004). PM model performance metrics (Boylan and Russell 2006) Summary diagrams (Jolliff et al. 2009) SEMIP project Mesoscale Model Evaluation – COST728 (Schluenzen & Sokhi, 2008)

7. HARMO13, 1-4June 2010, Paris, France 7 Key elements of the procedure (I) DELTA: Evaluation tool based on City- & Euro- Delta intercomparison exercises ENSEMBLEJRC web based multi-model evaluation and inter- comparison platform used by several modeling communities (e.g. Galmarini S. et al. 2001, 2004a and b). Data Extraction JRC based (AirBase, Emissions, BC), links to other projects data (GMES, EC4MACS…) Benchmarking Service JRC based (performance indicators, criteria and goals, summary reports)  to be developed

8. HARMO13, 1-4June 2010, Paris, France 8 Key elements of the procedure (II) Model results DELTA JRC USER Data Extraction Facility Unofficial Working Report BENCHMARKING service Official Reports Unofficial Working Report

9. HARMO13, 1-4June 2010, Paris, France 9 The benchmarking service (I) Main elements Decomposition of the evaluation in temporal and spatial segments Definition of a core set of statistical indicators and summary diagrams. Definition of criterias, goals and observation uncertainty Selection of tests (model vs. observations, model vs. model, model sensitivities) Summary model performance reporting (automatic)

10. HARMO13, 1-4June 2010, Paris, France 10 The benchmarking service (II) Core set of statistical indicators RCorrelation BBias SDStandard deviation RMSERoot Mean Square Error RMSEsSystematic RMSE RMSEuUnsystematic RMSE CRMSECentered RMSE IOAIndex of Agreement MFBMean Fractional Bias MFEMean Fractional Error RDE Relative Directive Error RPE Relative Percentile Error

11. HARMO13, 1-4June 2010, Paris, France 11 Cos -1 R CMRSE The benchmarking service (II) Summary Diagrams

12. HARMO13, 1-4June 2010, Paris, France 12 Criteria: Model acceptable performance for a given type of application (e.g. PM: MFE=75%, MFB=+/-60%) Goal: Best performance a model should aim to reach given its current capabilities (e.g. PM: MFE=50%, MFB=+/-30%) Observation Uncertainty: Accuracy level of the measurements The benchmarking service (III)

13. HARMO13, 1-4June 2010, Paris, France 13 The benchmarking service (IV) Performance summary report

14. HARMO13, 1-4June 2010, Paris, France 14 Usage of the procedure Model results DELTA JRC USER Data Extraction Facility Unofficial Working Report BENCHMARKING service Official Reports Unofficial Working Report

15. HARMO13, 1-4June 2010, Paris, France 15 Work Plan - Discussion and consensus on overall methodology (FAIRMODE meeting 09/2010) - Development of the DELTA and benchmarking service prototypes (Dec 2010) - Testing of the prototypes on existing datasets (2011) - Development of the JRC Web facilities (data extraction, links ENSEMBLE-Benchmarking service…) - Set-up of a joint exercise for testing of the whole system (2012)

16. HARMO13, 1-4June 2010, Paris, France 16 Contributions Discussion and definition of the benchmarking protocol elements (species, statistics, goals and criterias…) for model performance reporting Links to other SGs Definition of the joint activities