Compliance for statistics Transparent and objective approach to evaluate data delivery and data compliance; outcome of task force discussions So far, when carrying out the compliance assessment, the domain managers always tried to adopt a pragmatic approach. But often – also by the Domain Managers themselves - the current approach was considered as insufficiently objective and associated with lacking transparency. So to say, the guidelines for the evaluation process are rather vague and leave space for subjective decisions. Therefore - in order to review the monitoring method and to improve the efficiency – internal Task force was established - composed of 2 members of each team – work is still ongoing The task force has started its work in June, investigating on the differences in the evaluation methods applied by the single domains. Work is still on-going and further discussions will be needed. But for the next evaluation exercise we will hopefully be in a position to switch to a new, more transparent evaluation process.

Task force on compliance assessment The evaluation comprises 4 criteria: Availability of the data files: percentage of received datasets compared to the required datasets for a given deadline Completeness of the received data: percentage of received records compared to the required records Timeliness: average of delay in days for all datasets required during the assessment period (12 months); Data Quality: based on quality reports and outlier detection when importing the data in the database Evaluation process – rather complex: The task force has identified 4 quality criteria which should be comprised in the evaluation: - Availability of the data files: refers to the percentage of received datasets compared to the required datasets for a given deadline (information could be drawn from eDamis report). Missing files whose deadline is close to the evaluation deadline may be considered as late instead of missing. - Completeness of the received data: percentage of received records compared to the required records (of course measuring completeness does not make sense if no data file is available. In this case, completeness is 0%); - Timeliness: average of delay in days for all datasets required during the assessment period (12 months); Data Quality: still under discussion what kind of measures could be applied. The task-force is analysing different options (based on quality reports – linked to the validation steps when importing the data in the database – suspicious data, outliers) Further,it was decided that the evaluation covers always a period of one year. This means that for domains where monthly data are to be delivered, it covers several data transmissions

Task force on compliance assessment - criteria for a domain may not fit with the needs of another domain - The priorities of the task force group are the following: 1) transparency 2) objectivity 3) harmonisation 4) simplicity Discussions – reached to the conclusion – often, the weight given to the single assessment criteria (availability of data files, completeness, timeliness and data quality) differs from one domain to another. E.g. for some data collections, timeliness is considered more relevant than for others, in particular when the absence of data for one or more Member States could put at stake one of the key publications of Eurostat. In agro-monetary statistics for instance, a delay of more than 2 days is already considered a serious incompliance issue. Other domains with different needs are less strict regarding timeliness and rate even a delay of 5 days as a minor incompliance issue. It has also to be said that the current statistical legislation is not homogenous concerning the requirements established in the field of quality and methodology. Prioities of the task force group: first of all transparency, objectivity, only at the 3th instance harmonisation and simplicity

Proposal for strengthened evaluation… One important requirement for achieving transparency, was considered the disclosure of the weights given to the different assessment criteria by the single domains. To do so we have elaborated an evaluation scheme which comprises the 4 criteria criteria, mentioned before: - Availability of the data files: refers to the percentage of received datasets compared to the required datasets for a given deadline (information could be drawn from eDamis report). Missing files whose deadline is close to the evaluation deadline may be considered as late instead of missing. - Completeness of the received data: percentage of received records compared to the required records (of course measuring completeness does not make sense if no data file is available. In this case, completeness is 0%); - Timeliness or punctuality: average of delays in days for all datasets required during the assessment period (12 months); - Data Quality (based on quality reports – linked to the validation steps when importing the data in the database – suspicious data, outliers) still under discussion what kind of measures could be applied. Underlying each criteria we will have 4 evaluation classes (full compliance, minor incompliance, major incompliance and persistent major incompliance). and for each data collection and for each of the classes we have tried to define the range of acceptance. This is considered a first step towards more transparency. ANI reg. 1165/2008 acceptable ranges for availability Another issue is the summary table: Serious doubts were expressed regarding a unique compliance summary table for data under legislation and under gentlemen's agreement. In order to allow a linkage between compliance marks and eventual actions to be taken (say infringement procedure), the evaluation will probably have to be split in two parts. one for data under regulation and another one for data under gentlemen's agreement. On the other hand, the task force has decided to evaluate candidate countries in the same way and on the same table as MS, with the only exception, that the should never be marked with "persistent major incompliance". Candidate countries might have an interest in comparing their situation with those of EU Member States before joining the EU.

Proposal for strengthened evaluation… The aim of the evaluation scheme is to acheive a transparent way of evaluation. The final output, the document to be produced for the CPSA will not differ much from what we produce now. The difference is that the evaluation behind and the calculation of the final marks will be much more objecive and explainable. Obviously the compliance document is supposed to provide a summary of the situation of the previous 12 months (including all deadlines) and is supposed to cover all 4 of the previously mentioned quality criteria. Therefore we will have to agree on an way to calculate the final marks: On proposal was the following. According to this module we would assign a green check when all quality criteria (avail., completeness, timeliness and quality) are marked with "full compliance" for all datasets and all deadlines in the reference period. Green check In case any criteria is marked with "persistent major incompliance" we would assign a ret dot for ""persistent major incompliance" At the third instance - If the availability class is marked with something less then (full compliance) then we would assign the Min(marks) for availability, which puts a higher emphasis on the availability. - Finally - Availability is set at "full compliance" (to say if all files for a given data collection are available) --> Average (marks) Another idea was to apply a formula "availability + availability + timliness + completeness / 4" Regarding the calculation method for the final marks on the summary table, the task-force is analysing different options. The task force will meet again at end of November in order to come to an agreement.

Many thanks for your attention. Your proposals on the way compliance issues should be addressed in the future and ideas regarding a more objective evaluation method are welcome. Many thanks for your attention. Eurostat thanks those MS which respect the deadlines and take care about completeness and quality of the data and delivering reliable data. but I would welcome very much concrete actions to improve the mentioned shortcomings Stress once again that missing and/or incomplete data lead to shortcomings also on our side, i.e. it is not possible to calculate EU aggregates and to publish data in planned time schedule. This document does not aim to be an official compliance monitoring exercise