Download presentation
Presentation is loading. Please wait.
1
Benchmark database Victor Venema, Olivier Mestre, Enric Aguilar, Ingeborg Auer, José A. Guijarro, Petr Stepanek, Claude.N.Williams, Matthew Menne, Peter Domonkos, Julien Viarre, Gerhard Müller-Westermeier, Tamás Szentimrey, Monika Lakatos, Dubravka Rasol, Elke Rustemeier, Gregor Vertacnik, Kostas Kolokythas, Tania Marinova, Fiorella Acquaotta, Sorin Cheval, Lars Andersen, Tanja Likso, Matija Klancar, Michele Brunetti, Christine Gruber, Marc Prohom Duran, Theo Brandsma
2
Content Introduction to benchmark dataset Preliminary results
Changes since Tarragona Preliminary results Conclusions, further analysis, articles, future
3
Deadline Deadline: no more updated contributions
Now: I can tell much more about results EGU, 3 – 7 Mai; presentation benchmark We can organise another comparison in few years Non-blind comparisons Partial release of benchmark Release a few networks Find errors and update contribution for other networks New work for everyone and a delay
4
Benchmark dataset Real (inhomogeneous) climate records Synthetic data
Most realistic case Investigate if various HA find the same breaks Synthetic data For example, Gaussian white noise Insert know inhomogeneities Test performance Surrogate data Empirical distribution and correlations Compare to synthetic data: test of assumptions
5
Creation benchmark Start with homogenised data
Multiple surrogate and synthetic realisations Mask surrogate records Add global trend Insert inhomogeneities in station time series Published on the web Homogenize by COST participants and third parties Analyse the results and publish
6
1) Start with homogeneous data
Monthly mean temperature and precipitation Working on daily data (WG4; Wednesday) Homogeneous, no missing data Longer surrogates are based on multiple copies Generated networks are 100 a
7
2) Multiple surrogate realisations
Temporal correlations Station cross-correlations Empirical distribution function Annual cycle removed before, added at the end Number of stations, 5, 9 or 15 Cross correlation varies as much as possible
8
4) Add global trend Stochastic signal Fourier filtered noise
E(k) ~ k-4
9
5) Insert inhomogeneities in stations
Independent breaks Determined at random for every station and time Number breaks per 100 a Uniform distribution between [2, 8] Temperature Additive, Size: Gaussian distribution, σ=0.8°C Size seasonal cycle: Gaussian distribution, σ=0.4°C Seasonal cycle has minimum in summer or winter Rain Multiplicative Size: Gaussian distribution, <x>=1, σ=15% Size seasonal cycle: Gaussian distribution, σ =7.5%
10
5) Insert inhomogeneities in stations
Correlated break in network One break in 10 % of networks In 30 % of the station simultaneously Position random At least 10 % of data points on either side
11
Example correlated break
12
5) Insert inhomogeneities in stations
Outliers Size Temperature: < 1 or > 99 percentile Rain: < 0.1 or > 99.9 percentile Frequency 1 per station per 100a
13
5) Insert inhomogeneities in stations
Local trends (only temperature) Linear increase or decrease in one station Duration: between 30 and 60a Maximum size: Gaussian distribution, σ=0.8°C Frequency: once in 10 % of the stations
14
Example local trends
15
Contributions Last year 17 contributions; 11 contributors
Directory Algorithm Version Contributors 1 h001 MASH Kolokythas Kostas Kolokythas 2 h002 Prodige Breaks Dubravka Rasol, Elke Rustemeier, Olivier Mestre 3 h003 USHCNv2 52x Claude.N.Williams, Matthew Menne 4 h004 52d 5 h005 cx8 6 h006 C3SNHT Enric Aguilar 7 h007 RhTestV2 Absolute Julien Viarre 8 h008 Relative 9 h009 Marinova Tania Marinova h010 Climatol José A. Guijarro a h011 Hungary Szentimrey Tamás, Monika Lakatos b h012 SNHT DWD Müller-Westermeier Gerhard c h013 Trendy d h014 AnClim Likso Tanja Likso e h015 Domonkos Peter Domonkos f h016 iCraddock Vertacnik Gregor Vertacnik g h017 Klancar Matija Klancar h h018 Stepanek Petr Stepanek i h019 Andersen Lars Andresen j h020 Acquaotta Fiorella Acquaotta k h021 Craddock Brunetti Michele Brunetti l h022 Basic Sorin Cheval m h023 Light n h024 Strict o h025 No meta p h026 PRODIGEm monthly Contributions Last year 17 contributions; 11 contributors Now 26 contributions; 18 contributors (not counting multiple people working on one contributions)
16
No. homogenised networks - algorithm
Table 1. Number of homogenised networks per algorithm Homogenisation alg. All networks Real netw. Surrogate netw. Synthetic netw. USHCNv2 52x 46 6 20 USHCNv2 52d USHCNv2 cx8 PRODIGE Acquaotta 3 PRODIGE regular 40 PRODIGE trendy PRODIGE monthly MASH SL 62 2 MASH Marinova 8 7 1 MASH Kolokythas 16 10 5 MASH Basic 4 MASH Light MASH Strict MASH No meta iCraddock Vertacnik 12 9 iCraddock Klancar Craddock Brunetti AnClim Stepanek 92 AnClim Andresen AnClim Likso RhTestV2 abs RhTestV2 rel C3SNHT SNHT DWD Climatol Domonkos No. homogenised networks - algorithm
17
No. homogenised networks – input data - surrogate
Table 2. Number of homogenised networks per network Network No networks precip sur 11 temp sur 23 precip sur 10 temp sur 24 precip sur temp sur 22 precip sur temp sur 19 precip sur temp sur 18 precip sur temp sur 17 precip sur temp sur 16 precip sur temp sur precip sur temp sur precip sur temp sur precip sur temp sur 15 precip sur temp sur 14 precip sur temp sur precip sur temp sur precip sur temp sur precip sur temp sur precip sur temp sur precip sur temp sur precip sur temp sur precip sur temp sur No. homogenised networks – input data - surrogate
18
No. homogenised networks – input data - synthetic
Table 2. Number of homogenised networks per network Network No networks precip syn 2 temp syn 9 precip syn temp syn precip syn temp syn precip syn temp syn 7 precip syn temp syn 8 precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn precip syn temp syn No. homogenised networks – input data - synthetic
19
Mean no. outliers per station
Table 21. Mean number of outliers per station for every algorithm Homogenisation alg. All networks Real netw. Surrogate netw. Synthetic netw. USHCNv2 52x 0.3 0.1 USHCNv2 52d USHCNv2 cx8 PRODIGE Acquaotta 16.2 NaN 12.1 20.3 PRODIGE regular 2.5 PRODIGE trendy PRODIGE monthly MASH SL 2.9 2.1 3 MASH Marinova 2.3 0.7 MASH Kolokythas 1 0.8 1.3 MASH Basic 5.2 MASH Light 6.3 MASH Strict MASH No meta 5.6 iCraddock Vertacnik 0.5 0.6 iCraddock Klancar Craddock Brunetti AnClim Stepanek 1.6 1.8 AnClim Andresen AnClim Likso RhTestV2 abs RhTestV2 rel C3SNHT SNHT DWD Climatol 1.5 1.7 1.1 1.9 Domonkos Mean no. outliers per station
20
Mean no. breaks per station
Table 22. Mean number of breaks per station. * No. MASH breaks divided by 12. Homogenisation alg. All networks Real netw. Surrogate netw. Synthetic netw. USHCNv2 52x 2.1 1.2 2.4 2 USHCNv2 52d 1.6 0.9 1.8 USHCNv2 cx8 1.7 1.4 1.9 PRODIGE Acquaotta 4.2 NaN 5.3 3.1 PRODIGE regular 3.5 PRODIGE trendy PRODIGE monthly 3.6 MASH SL 3.4 3.2 3.9 MASH Marinova 3.7 MASH Kolokythas 1.3 0.8 1.1 MASH Basic 6.6 MASH Light 6.9 MASH Strict 5.9 MASH No meta 7.3 iCraddock Vertacnik 5.7 6.7 5.4 iCraddock Klancar 4.6 Craddock Brunetti 8.9 AnClim Stepanek 4.4 2.7 4.8 4.5 AnClim Andresen AnClim Likso 21.4 RhTestV2 abs RhTestV2 rel C3SNHT 5 SNHT DWD 1 Climatol 1.5 Domonkos 4.9 4.7 Mean no. breaks per station Mash contributions divided by 12
21
Analysing the results What measures define a well homogenised dataset?
Real data vs. data with known truth Ensemble mean for real data? Data itself Root mean square error (RMSE) RMSE (without outliers or Mean Absolute Error) RMSE (bias corrected; std. dev. of difference) Uncertainty in the network mean trend Breaks Position, hit rate Size distribution Detection probability as function of size
22
RMSE – surrogate temperature
The following plots were made in a hurry, just before the workshop. They likely contain errors. The last contribution on the x-axis without a name of PRDIGE monthly.
23
SD Difference - Temperature surrogate
The last contribution on the x-axis without a name of PRDIGE monthly.
24
SD Difference - Precipitation surrogate
The last contribution on the x-axis without a name of PRDIGE monthly.
25
RMSE anomaly – Surrogate & Temperature
boxplot(X) produces a box and whisker plot for each column of the matrix X. The box has lines at the lower quartile, median, and upper quartile values. Whiskers extend from each end of the box to the adjacent values in the data; by default, the most extreme values within 1.5 times the interquartile range from the ends of the box. Outliers are data with values beyond the ends of the whiskers. Outliers are displayed with a red + sign.
26
RMSE anomaly – Surrogate & Precipitation
boxplot(X) produces a box and whisker plot for each column of the matrix X. The box has lines at the lower quartile, median, and upper quartile values. Whiskers extend from each end of the box to the adjacent values in the data; by default, the most extreme values within 1.5 times the interquartile range from the ends of the box. Outliers are data with values beyond the ends of the whiskers. Outliers are displayed with a red + sign.
27
SD Diff anomaly – Surrogate & Temperature
28
SD Diff anomaly – Surrogate & Precipitation
29
Conclusions from results
Absolute statistical homogenisation is a bad idea Some indications of “programming” errors The coder is the best operator Many manual methods are worse than the best automatic ones Automatic algorithms can be quite good Benchmark with multiple networks is needed
30
Analysing the results How to study which components are best?
Correct with known breaks and outliers dates Who would be interested? Situation dependent analysis No. stations; outliers; type break; … Algorithm type Need descriptions! Reference/pair-wise; manual/automatic; … Everyone should try to understand performance of his algorithm Suggest analysis methods
31
Articles Overview COST Action & benchmark with very basic analysis results Do we have references to all algorithms? Deadline for description of the algorithm Others articles? Analysing results, which components are best Who will organise, coordinate it? Not everyone should do the same analysis How to subdivide the work?
32
Future A new benchmark in x years? Ideas for a better benchmark?
For example, for other inhomogeneities, constants Types of inhomogeneities for daily data (WG4) Automatic homogenisation algorithms Larger benchmark
Similar presentations
