Compliance for statistics

Slides:



Advertisements
Similar presentations
1 European Conference on Quality in Official Statistics Rome, 8-11 July 2008 Improving the quality and the quality assessment of the Labour Force Survey.
Advertisements

Exploring the use of QSR Software for understanding quality - from a research funder’s perspective Janice Fong Research Officer Strategies in Qualitative.
Joint meeting Working Groups on Environmental Accounts & Environmental Expenditure Statistics Luxembourg, 10 March 2015 Forest Accounts (point 8 of the.
Defining and applying mitigating and aggravating circumstances. Relevant changes to the amount of fine. Defining and applying mitigating and aggravating.
Professional Certificate in Electoral Processes Understanding and Demonstrating Assessment Criteria Facilitator: Tony Cash.
1 Use of aggregated SPPIs as a macro economic indicator of inflationary pressure: country views and possible future STESWP work Richard McKenzie & Seppo.
Estimation of emigration flows by using immigration figures in receiving countries Michel POULAIN GéDAP UCL Belgium.
Fundamentals of Governance: Parliament and Government Understanding and Demonstrating Assessment Criteria Facilitator: Tony Cash.
E-PRTR incompleteness check Irene Olivares Industrial Pollution Group Air and Climate Change Programme Eionet NRC workshop on Industrial Pollution Copenhagen.
7-8-March 2011 Task Force "Organic farming statistics"-Luxembourg, 7-8 March Item 4 Harmonised questionnaire for data collection: State of the art.
REPUBLIC OF ALBANIA PUBLIC PROCUREMENT AGENCY Eighth Regional Public Procurement Forum May, 22-25, 2012 Tirana
Working group “Maritime Transport Statistics” Luxembourg, April 2008 Data related issues Item 8 of the agenda.
Development of Assessments Laura Mason Consultant.
Meeting of the „Resolution 6” CC Task Force
PRESENTATION OF MONTENEGRO
UNECE-CES Work session on Statistical Data Editing
Governance, Fraud, Ethics and Corporate Social Responsibility
Discussion: Timely estimates of economic indicators – Session C3 –
Mexico 8th Meeting of the Steering Committee of INTOSAI Committee on
COORDINATING GROUP FOR STATISTICS ON TRANSPORT
Seminar on ESA 2010 Metadata
INTRODUCTION TO Compliance audit METHODOLGY and CAM
Diffuse Sources of Water Pollution
Economy-wide Material Flow Accounts (EW-MFA) (point 4 of the agenda)
Education and Training Statistics Working Group – June 2014
Rolling Review of Education Statistics
The new metadata structure & Country Specific Notes
Fitness Check of environmental reporting
Legal framework of territorial classifications and typologies for European statistics – state of play NUAC meeting, Brussels June 2015 Gorja Bartsch.
Compliance Item 4.2 Compliance Doc. ASA/TE/743-rev1
Data Validation in the ESS Context
Agenda Item 2.1 SES 2014: follow-up
“Land Cover/Use Statistics”
ETS WG meeting 6-7 September 2006
Passenger Mobility Statistics 21 May 2015
Prodcom ESTP course October 2010
Quality Criteria Initial Ideas.
Item 7.5 (2012-ETS-16) – Statistics on Special Needs Education
LAMAS Working Group October 2014
Improving information exchange:
ETS WG 6-7 September, 2006 Point 6: UOE quality report History
Task Force on Environmental transfers of the Working Group on
ESS VIP ICT Project Task Force Meeting 5-6 March 2013.
LAMAS Working Group December 2013
Item 7 - Roadmap and mandate for the Task Force on UOE Education Expenditure Data Eurostat Education and Training Statistics Working Group - Luxembourg,
Revision of Error Margins Agenda point 9 (point for decision)
ESTP course on International Trade in Goods Statistics
XBRL PILOT TASK FORCE MEETING
Point 6. Eurostat plans for Time Use Survey data processing and dissemination Working Group on Time Use Surveys 10 April 2013.
Diffuse Sources of Water Pollution
Mexico 8th Meeting of the Steering Committee of INTOSAI Committee on
Multinational enterprise groups in the EU Dissemination from the EGR
Policy needs for rural development statistics and data analysis
Task Force 3, Cultural Industries Kutt Kommel
Marleen De Smedt Geoffrey Thomas Cynthia Tavares
Rail transport developments Agenda point 7.2
Quality reporting under Regulation (EC) No 1165/2008
Streamlining of monitoring and reporting under WFD, Nitrates Directive and EEA's SoE –concept paper DG Environment.
Education and Training Statistics Working Group, May 2011
Monthly other slaughtering
BPR AS Review Programme
European Statistical System Network on Culture (ESSnet Culture)
Item 1 – WFD Implementation Report 2007
Concept paper on the assessment of WFD River Basin Management Plans
2.7 Annex 3 – Quality reports
Passenger Mobility Statistics 21 May 2015
Draft implementing act on Monthly Unemployment Rate (MUR) Item 3
Item 5 Modernisation of the EU-SILC Production
Future of EDAMIS Webforms
Task Force on GNB 14 – 15 November
Presentation transcript:

Compliance for statistics Transparent and objective approach to evaluate data delivery and data compliance; outcome of task force discussions So far, when carrying out the compliance assessment, the domain managers always tried to adopt a pragmatic approach. But often – also by the Domain Managers themselves - the current approach was considered as insufficiently objective and associated with lacking transparency. So to say, the guidelines for the evaluation process are rather vague and leave space for subjective decisions. Therefore - in order to review the monitoring method and to improve the efficiency – internal Task force was established - composed of 2 members of each team – work is still ongoing The task force has started its work in June, investigating on the differences in the evaluation methods applied by the single domains. Work is still on-going and further discussions will be needed. But for the next evaluation exercise we will hopefully be in a position to switch to a new, more transparent evaluation process.

Task force on compliance assessment The evaluation comprises 4 criteria: Availability of the data files: percentage of received datasets compared to the required datasets for a given deadline Completeness of the received data: percentage of received records compared to the required records Timeliness: average of delay in days for all datasets required during the assessment period (12 months); Data Quality: based on quality reports and outlier detection when importing the data in the database Evaluation process – rather complex: The task force has identified 4 quality criteria which should be comprised in the evaluation: - Availability of the data files: refers to the percentage of received datasets compared to the required datasets for a given deadline (information could be drawn from eDamis report). Missing files whose deadline is close to the evaluation deadline may be considered as late instead of missing. - Completeness of the received data: percentage of received records compared to the required records (of course measuring completeness does not make sense if no data file is available. In this case, completeness is 0%); - Timeliness: average of delay in days for all datasets required during the assessment period (12 months); Data Quality: still under discussion what kind of measures could be applied. The task-force is analysing different options (based on quality reports – linked to the validation steps when importing the data in the database – suspicious data, outliers) Further,it was decided that the evaluation covers always a period of one year. This means that for domains where monthly data are to be delivered, it covers several data transmissions

Task force on compliance assessment - criteria for a domain may not fit with the needs of another domain - The priorities of the task force group are the following: 1) transparency 2) objectivity 3) harmonisation 4) simplicity Discussions – reached to the conclusion – often, the weight given to the single assessment criteria (availability of data files, completeness, timeliness and data quality) differs from one domain to another. E.g. for some data collections, timeliness is considered more relevant than for others, in particular when the absence of data for one or more Member States could put at stake one of the key publications of Eurostat. In agro-monetary statistics for instance, a delay of more than 2 days is already considered a serious incompliance issue. Other domains with different needs are less strict regarding timeliness and rate even a delay of 5 days as a minor incompliance issue. It has also to be said that the current statistical legislation is not homogenous concerning the requirements established in the field of quality and methodology. Prioities of the task force group: first of all transparency, objectivity, only at the 3th instance harmonisation and simplicity

Proposal for strengthened evaluation… One important requirement for achieving transparency, was considered the disclosure of the weights given to the different assessment criteria by the single domains. To do so we have elaborated an evaluation scheme which comprises the 4 criteria criteria, mentioned before: - Availability of the data files: refers to the percentage of received datasets compared to the required datasets for a given deadline (information could be drawn from eDamis report). Missing files whose deadline is close to the evaluation deadline may be considered as late instead of missing. - Completeness of the received data: percentage of received records compared to the required records (of course measuring completeness does not make sense if no data file is available. In this case, completeness is 0%); - Timeliness or punctuality: average of delays in days for all datasets required during the assessment period (12 months); - Data Quality (based on quality reports – linked to the validation steps when importing the data in the database – suspicious data, outliers) still under discussion what kind of measures could be applied. Underlying each criteria we will have 4 evaluation classes (full compliance, minor incompliance, major incompliance and persistent major incompliance). and for each data collection and for each of the classes we have tried to define the range of acceptance. This is considered a first step towards more transparency. ANI reg. 1165/2008 acceptable ranges for availability Another issue is the summary table: Serious doubts were expressed regarding a unique compliance summary table for data under legislation and under gentlemen's agreement. In order to allow a linkage between compliance marks and eventual actions to be taken (say infringement procedure), the evaluation will probably have to be split in two parts. one for data under regulation and another one for data under gentlemen's agreement. On the other hand, the task force has decided to evaluate candidate countries in the same way and on the same table as MS, with the only exception, that the should never be marked with "persistent major incompliance". Candidate countries might have an interest in comparing their situation with those of EU Member States before joining the EU.

Proposal for strengthened evaluation… The aim of the evaluation scheme is to acheive a transparent way of evaluation. The final output, the document to be produced for the CPSA will not differ much from what we produce now. The difference is that the evaluation behind and the calculation of the final marks will be much more objecive and explainable. Obviously the compliance document is supposed to provide a summary of the situation of the previous 12 months (including all deadlines) and is supposed to cover all 4 of the previously mentioned quality criteria. Therefore we will have to agree on an way to calculate the final marks: On proposal was the following. According to this module we would assign a green check when all quality criteria (avail., completeness, timeliness and quality) are marked with "full compliance" for all datasets and all deadlines in the reference period. Green check In case any criteria is marked with "persistent major incompliance" we would assign a ret dot for ""persistent major incompliance" At the third instance - If the availability class is marked with something less then (full compliance) then we would assign the Min(marks) for availability, which puts a higher emphasis on the availability. - Finally - Availability is set at "full compliance" (to say if all files for a given data collection are available) --> Average (marks) Another idea was to apply a formula "availability + availability + timliness + completeness / 4" Regarding the calculation method for the final marks on the summary table, the task-force is analysing different options. The task force will meet again at end of November in order to come to an agreement.

Many thanks for your attention. Your proposals on the way compliance issues should be addressed in the future and ideas regarding a more objective evaluation method are welcome. Many thanks for your attention. Eurostat thanks those MS which respect the deadlines and take care about completeness and quality of the data and delivering reliable data. but I would welcome very much concrete actions to improve the mentioned shortcomings Stress once again that missing and/or incomplete data lead to shortcomings also on our side, i.e. it is not possible to calculate EU aggregates and to publish data in planned time schedule. This document does not aim to be an official compliance monitoring exercise