1. Prodcom Working Group Item 08.2 - Quality reporting and indicators
18th -19th September 2014

2. CURRENT SITUATION: NO REGULAR quality reporting from MS
2. Very general provision in PRODCOM, under Art.6 of Regulation 912/2004:
”[…] Member States are also to provide Eurostat with the necessary information about their survey methods, samples and coverage for the purpose of documentation of compliance with the principles of the PRODCOM methodology”
Item 14_08
Prodcom Working Group
18th -19th September 2014

3. IN THE PAST:
2003 & Quality reports have been provided on voluntary basis
Eurostat presented the National Reference Metadata Editor (NRME) in the Working Group. 4 countries (DK, IE, PT, UK) agreed to test the reporting tool in 2013
2013 – Denmark presented their experience with NRME to the WG held in October.
CONCLUSION: the application was EASY TO USE
Item 14_08
Prodcom Working Group
18th -19th September 2014

4. Current context of FRIBS
6 preliminary provisions, out of which:
Member States shall take all measures necessary to ensure the quality of the statistics transmitted and of the business registers according to common quality criteria.
For the purpose of this Regulation, the quality criteria defined in Article 12(1) of Regulation (EC) 223/2009 shall apply.
The modalities, structure and periodicity of the quality reports shall be adopted by the Commission (Eurostat) in the form of implementing acts. Those implementing acts shall be adopted in accordance with the examination procedure referred to Article 13(2).
Mention : The 6 provisions appear in preliminary version of FRIBS legislation
2 – Code of Practice (quality criteria)
3 – In principiu exista flexibilitate si implementarea va depinde de la domeniu la domeniu, dar va trebui sa introducem un standard de raportare in Regular Reporting
Item 14_08
Prodcom Working Group
18th -19th September 2014

5. The quality criteria & indicators (1)
(defined in Article 12(1) of Regulation (EC) 223/2009)
RELEVANCE
R1. User satisfaction index
R2. Rate of available statistics
ACCURACY
A1. Coefficient of variation
A2. Unit response rate (un-weighted/weighted)
A3. Item response rate (un-weighted/weighted)
A4. Imputation rate and ratio
A5. Over-coverage and misclassification rates
A6. Geographical under-coverage ratio
A7. Average size of revisions
Item 14_08
Prodcom Working Group
18th -19th September 2014

6. The quality criteria & indicators (2)
(defined in Article 12(1) of Regulation (EC) 223/2009)
TIMELINESS AND PUNCTUALITY
T1. Punctuality of time schedule of effective publication
T2. Time lag between the end of reference period and the date of first results
T3. Time lag between the end of reference period and the date of the final results
ACCESSIBILITY AND CLARITY
AC1. Number of publications disseminated and/ or sold
AC2. Number of accesses to databases
AC3. Rate of completeness of metadata information for released statistics.
Item 14_08
Prodcom Working Group
18th -19th September 2014

7. The quality criteria & indicators (3)
(defined in Article 12(1) of Regulation (EC) 223/2009)
COHERENCE AND COMPARABILITY
C1. Length of comparable time-series
C2. Number of comparable time-series
C3. Rate of differences in concepts and measurement from European norms
C4. Asymmetries for statistics mirror flows
CH1. Rate of statistics that satisfies the requirements for the main secondary use
Item 14_08
Prodcom Working Group
18th -19th September 2014

8. REPORTING STANDARDS
(introduced in Nov 2013 by SIMS)
ESMS: the Euro SDMX Metadata Structure - is the basic ESS structure for reference metadata.
ESQRS: the ESS Standard for Quality Reports Structure - is the harmonized report structure for ESS quality reports
Item 14_08
Prodcom Working Group
18th -19th September 2014

9. ESMS structure Item XX Prodcom Working Group 18th -19th September 2014
The ESMS structure is not corresponding exactly with quality criteria, but with a slight adjustment it covers all of them
The proposed indicators ( which can be found in handbook / task force – as source) constitutes a good starting point in building of Prodcom indicators, depending on specificities of our activity
Need for quality reporting in PRODCOM.
The guiding lines are given by:
- The ESMS structure is not corresponding exactly with quality criteria, but with a slight adjustment it covers all of them
the quality dimensions in the code of practice & mentioned indicators
Additional work done for the development of indicators corresponding to the different quality dimensions (handbook…task force); this can be a starting point to complement or substitute the current compliance score which is a composite indicator that is however difficult to interpret. Other criteria important for PRODCOM should be considered: e.g. the 90% representativeness threshold or other criteria in the regulation. In case that indicators cannot be computed, some qualitative information on the methodology is necessary for the quality and compliance assessment.
Item XX
Prodcom Working Group
18th -19th September 2014

10. RECOMMENDATION OF REPORTING
(technical manual annexed to the ESSC document of November 2013)
“It is recommended that all statistical processes in the ESS should at least have a basic quality report in the form of short or user quality report (ESMS)”.
Prodcom Working Group
18th -19th September 2014

11. RECOMMENDATION OF REPORTING
(DIME-IT Directors Joint meeting, March 2014)
“All statistical processes producing, exchanging and disseminating European statistics should be accompanied by general reference metadata – including basic quality information according to the ESS quality criteria – in using the ESMS structure.”
Item 14_08
Prodcom Working Group
18th -19th September 2014

12. EUROSTAT PROPOSALS:
To use ESMS as quality reporting standard (filled template attached in annex 2 of document)
MS to send their annual quality report recommendable till 15th of November of every year (this deadline could be later reviewed in the terms of FRIBS)
A limited number of quality indicators should be computed – or at least qualitative information where indicators can not be computed
Item XX
Prodcom Working Group
18th -19th September 2014

13. EUROSTAT PROPOSAL: Member States voluntary for tests for 2103 data
Using the given structure and the guidelines provided as Annex (the guidelines could be refined during the testing period)
Eurostat will provide its support and feedback
When FRIBS will be final, this reporting will become compulsory
Written consultation for quality indicators and compliance assessment, if necessary
Item XX
Prodcom Working Group
18th -19th September 2014

14. THANK YOU
Prodcom Working Group
18th -19th September 2014