1. FAIRMODE SG4: Benchmarking & modelling quality objectives P. Thunis, JRC Antwerp, 04-2013

2. Model Quality Objectives Wed.14:00-15:00Uncertainty based MQO: overview & discussionP. Thunis 15:00-15:30UK Feedback on MQOD. Brookes & B. Stacey 15:30-15:50DELTA tool: Version 2 updatesP. Thunis DELTA tool experience 15:50-16:10Exp. In model inter-comparison exercise in BelgiumB. Maiheu 16:10-16:30UK FeedbackJ. Stedman 16:30-16:50CALIOPE forecast evaluationJ. Baldasano Thur.09:00-09:20TCAM model evaluationC. Carnevale 09:20-09:40CERC’s experience & FeedbackJ. Stocker 09:40-10:00Aveiro Univ. experience & feedbackA.Miranda 10:00-10:20The Fairmode deltatool: a MS experienceE. Trimpeneers 10:20-10:40Austria FeedbackD. Oettl Benchmarking activities related to SG4 10:40-11:10ATMOSYS model evaluation tool (On-line DELTA tool)N. Veldeman

3. Model Quality Objectives

4. Antwerp, 04-2013 P. Thunis Why do we need MQO? Tell whether a model is fit for purpose Support model users in their evaluation work (e.g. is R=0.6 good?) Facilitate experience exchanges among modelers based on a common performance scale

5. Antwerp, 04-2013 P. Thunis Little insight into performances (timing of events not considered, only in the LV neighborhood) Model can perform well for the wrong reason  ?? Planning applications ?? Current MQO (Guidance document) RDE=10% Why a new MQO?

6. Antwerp, 04-2013 P. Thunis Model: CTM TCAM Area: Po-Valley IT Year: 2009 Model: PM10=50ug/m3 Area: Everywhere Year: Any time

7. Antwerp, 04-2013 P. Thunis Current MQO (Guidance document) MQO current status

8. Antwerp, 04-2013 P. Thunis Part I: Derivation of the MQO Part II: Setting a value for U I.Performance criteria to evaluate air quality modelling applications. P. Thunis, A. Pederzoli, D. Pernigotti, 2012, Atmos. Env., 59, 476-482. I.Model quality objectives based on measurement uncertainty. Part I: Ozone. P. Thunis, D. Pernigotti, M. Gerboles, 2013, Submitted to Atmos. Env. II.Model quality objectives based on measurement uncertainty. Part II: NO2 and PM10, D. Pernigotti, P. Thunis, M. Gerboles, C. Belis, 2013, Submitted to Atmos. Env. Acknowledgments: AEA

9. DERIVATION OF THE MQO Antwerp, 04-2013 P. Thunis

10. Event A: Measured Event B: Modeled (1) (2)

11. Antwerp, 04-2013 P. Thunis Accuracy Precision 2 conditions imposed on model results systematic errorrandom error

12. Antwerp, 04-2013 P. Thunis Condition 1: Accuracy Condition on distance between distributions

13. Antwerp, 04-2013 P. Thunis Condition 2: Precision Condition on model distribution spread (σ)

14. Antwerp, 04-2013 P. Thunis Conditions (1) & (2): Accuracy and Precision Conditions on distance between distributions model distribution spread (σ) Constrained systematic and random errors

15. Antwerp, 04-2013 P. Thunis Year Hour/day MQO What value for U? Annual & hourly/daily MQO

16. SETTING A VALUE FOR U Antwerp, 04-2013 P. Thunis

17. Approach 2: uncertainty varies according to concentration (estimates based on monitoring) Concept of maximum uncertainty Approach 1: uncertainty assumed constant & equal to LV uncertainty over the whole range of concentrations

18. Antwerp, 04-2013 P. Thunis What value for U? p np

19. Antwerp, 04-2013 P. Thunis Applications  O3  NO2  Hourly  Yearly  PM10  Daily  Yearly

20. Antwerp, 04-2013 P. Thunis O3  GUM approach

21. Antwerp, 04-2013 P. Thunis Hourly O3 RV=120 ug/m3

22. Antwerp, 04-2013 P. Thunis O3 knownFit U kRV 1.441200.0900.620 Low range (~20ug/m3) LV range (120 ug/m3) High range (~200 ug/m3) MQO 120%26%20% U MQO

23. Antwerp, 04-2013 P. Thunis P. Thunis Hourly NO2: GUM approach 8760 hours 1000 AIRBASE stations 95th percent

24. Antwerp, 04-2013 P. Thunis Hourly NO2 knownFit U kRV 22000.120.02 Low range (~10ug/m3) LV range (200ug/m3) High range (>200 ug/m3) MQO 144%48% U MQO

25. Antwerp, 04-2013 P. Thunis Yearly NO2: GUM approach 1000 AIRBASE stations 95th percent

26. Antwerp, 04-2013 P. Thunis Yearly NO2 knownFit U kRVNpNp N np 22000.120.024.76.7 Low range (~10ug/m3) LV range (40ug/m3) High range (100 ug/m3) MQO 57%26%23% U MQO

27. Antwerp, 04-2013 P. Thunis U 2 PM 10 PM 10 2 30 stations x 10 days Gravimetric vs. gravimetric Daily PM10  GDE approach

28. Antwerp, 04-2013 P. Thunis Daily PM10 knownFit U kRV 2500.1380.027 Low range (~10ug/m3) LV range (120 ug/m3) High range (~100 ug/m3) MQO 70%56%54% U MQO

29. Antwerp, 04-2013 P. Thunis MethodSourceNb of stations GravimetricJRC intercomparison30 (10 days)13.8 % TEOMJRC intercomparison26 (10 days)19.0 % Beta-RayAIRBASE5 stations19.0 % Uncertainties for other measurement techniques

30. Antwerp, 04-2013 P. Thunis U 2 PM 10 PM 10 2 5 stations Beta-ray & Gravimetric Yearly PM10  GDE approach

31. Antwerp, 04-2013 P. Thunis Yearly PM10 KnownFit U kRVNpNp N np 2500.1380.027401 MQO Low range (~10ug/m3) LV range (40ug/m3) High range (100 ug/m3) 46%14%10% U MQO

32. Antwerp, 04-2013 P. Thunis MQO “Tuning”

33. Antwerp, 04-2013 P. Thunis M M OO Data Assimilation Data fusionNo assimilation O M O M

34. Antwerp, 04-2013 P. Thunis MQO “Tuning”

35. Antwerp, 04-2013 P. Thunis [C] NO2 Station typeRepresentative areaCell resolution UrbanA few km2~ 1-3 km SuburbanSome tens of km2~ 3-10 km RuralSome hundred of km2~ 10-35 km Rural background1000-10000km2~ 35-100km Traffic?? 1-3km

36. Antwerp, 04-2013 P. Thunis Conclusions (I) The most important feature of this MQO is the link to measurement uncertainty, i.e. the need for a better understanding of measurements during model evaluation. The stringency of the current MQO should not be seen as an obstacle since possible expert tuning can be made (based on SG4 current & future experience exchanges) Approach available for O3, NO2, PM10. Can be generalised to other variables (e.g. Meteo, NOx) and adapted to other modelling approaches (e.g. data assimilation) MQO are in testing phase. They should not be used as reference for the time being.

37. Antwerp, 04-2013 P. Thunis Conclusions (II) Uncertainty estimates are quite robust for NO2 values and daily PM10 (representative sample). But sample size is reduced for yearly estimates, especially PM10. Values assigned to Np and Nnp probably need further investigation (e.g. more years). Around the LV (for hourly/daily) the new MQO values are close to the 50% AQD model uncertainty for both PM10 and NO2. For O3 and PM10 Biases are also close to performance criteria proposed in literature previously (Boylan and Russel, 2005, Chemel et al., 2010)

38. Antwerp, 04-2013 P. Thunis Conclusions (III) O3 MQO probably too strict (k=1.4  k=1.6) implying (MQO=26%  MQO=30% around the Limit value) Data & experience exchanges would help assigning “final” values to the MQO parameters. Each parameter can be tested by any user (configuration file).

39. GRAPHICAL VIEW AND PERFORMANCE CRITERIA Antwerp, 04-2013 P. Thunis

40. The Target diagram Right - Left X - Y

41. Antwerp, 04-2013 P. Thunis All error inMPC Bias R Std. Dev. Bias=0 R=1 Complementary Performance criteria for hourly/daily resolution Each station type for a given location and for a given period of time will have a specific signature in terms of mean and standard deviation, leading to specific MQO and MPC. This is implicitly accounted for through the formulation.

42. Antwerp, 04-2013 P. Thunis 8h max O3Hourly NO2Daily PM10 BiasRSDBiasRSDBiasRSD Rural 37%0.40107%159%>0200%67%0.33116% Urban 41%0.5197%79%0.29117%65%0.33116% Traffic 44%0.42106%65%0.32116%64%0.26124% Po-Valley 37%0.8063%68%0.42107%66%0.46104% Paris 42%0.49100%77%0.32116%63%0.24124% Krakow 40%0.6386%91%>0139%69%0.5496% Model Performance Criteria for bias, R and SD in terms of location and station type (based on EU Airbase 2009 measurement data)

43. Antwerp, 04-2013 P. Thunis Conclusions MPC are necessary but not sufficient conditions Both the MQO and each MPC have been assigned a specific diagram in the Delta tool

44. BENCHMARKING REPORTS Antwerp, 04-2013 P. Thunis

45. BIAS < 0 BIAS > 0 RSD

46. DELTA tool Version 3.3 FAIRMODE – SG4

47. Antwerp, 04-2013 P. Thunis Main Changes in Version 3.3 Split hourly/daily  yearly 90%  75% data availability threshold Input formatting netcdf  xls leap year Check-IO: a necessary step

48. Antwerp, 04-2013 P. Thunis Everyone can test new MQO parameters Configuration file: goals & criteria 0; PM10; ALL; OU; PMEAN; 27.8; 0.027; 40; 1; 50

49. Antwerp, 04-2013 P. Thunis Planning time decomposition

50. Antwerp, 04-2013 P. Thunis Other: Specific features for inter-comparing models Linux version in development

51. Future activities FAIRMODE – SG4

52. Antwerp, 04-2013 P. Thunis NO2 PM10 O3 Pollutant Uncertainty budgets Data Assimilation90% threshold [C] NO2 Sp. Repres JRC-AEA JRC-Univ Bresc. Assessment of MQO