Quality of MIP indicators: Assessment of data and metadata ESTP Course Luxembourg 9-11 December 2014 Ivana Jablonska & Julien Bollati, MIP TF

Outline Quality framework of MIP Indicators Work done by now Future steps / Discussion

Quality framework of MIP Indicators

(Draft) MIP Regulation COM(2013) 342 Final Article 5 The Commission (Eurostat) shall regularly assess the quality of the MIP relevant data (…). The quality assessments shall, as appropriate, make full use of the work carried out, and the results obtained, in the context of existing quality frameworks for MIP relevant data.

Quality principles Eurostat mission: to be the leading provider of high quality statistics on Europe European statistics Code of Practice (28th September 2011)

Quality principles Public commitment on European Statistics by the ESCB

What for? The big question: Our preliminary answer: What about the quality of the MIP indicator? Our preliminary answer: In order to answer to this question we need to have a look to the inventories/quality report and run a risk assessment in the framework of a stocktaking exercise. Per country and per indicator

What for? The small question: Our preliminary answer: How safe are the MIP indicator? Our preliminary answer: Let's run an expert opinion poll among our team in order to make a ranking. Per country and per indicator

Our safeness definitions Information on sources and methods is clear in inventories/quality reports Sources cover the necessary basic information The compilation practices are in line with legal requirements and good/best practices

Our safeness definitions Risk under control: Information on sources and methods is generally available and mostly clear in inventories/quality reports Sources cover most of the necessary basic information, estimation methods are only used to compensate for a small part of the basic information Compilation practices are in line with legal requirements, but most other Member States use different practices

Our safeness definitions Potential risk: Information on sources and methods is partially available in inventories/quality reports, or fully available but suggesting an incomplete implementation of the methodology The sources of basic information are incomplete Compilation practices are not adequate compared to other Member States or not in line with legal requirements

Our safeness definitions Not known: Information on sources and methods is generally poor It is not possible to quantify whether there is any risk of significant revisions to the data

Guiding principles Factual assessment of data and metadata (vs perception) Standardised approach towards different domains and countries

The outcomes Based on the reading of data, metadata and quality reports Footnotes and proposed text for the Statistical Annex to the AMR Check for quality improvements with respect to the previous year Clustering by MIP headlines according to risk profile

Work done by now

The questionnaire We have developed a structured template with over 30 questions

Topics covered Institutional environment Resources CoP / PC Principle 1 Professional independence Institutional environment Authority responsible Legal and institutional environment Sharing of responsibilities Resources Adequacy of resources Cost and burden Principle 2 Mandate for data collection Principle 5 Statistical confidentiality Principle 6 Impartiality and objectivity CoP / PC Principle 3 Adequacy of resources Principle 10 Cost effectiveness

Appropriate statistical procedures Topics covered Quality Management Completeness and timeliness of information provided in the existing inventories and quality reports Quality control procedures Clarity Communication with Eurostat CoP / PC Principle 4 Commitment to quality Principle 8 Appropriate statistical procedures

Topics covered Methodological soundness Reliability of the methodology Implementation of regulations/guidelines/recommendations Expiry of derogations Unexplained breaks in the series CoP / PC Principle 7 Sound methodology

Topics covered Revisions Size of routine revision policies Information on major revisions Data analysis Completeness Timeliness CoP / PC Principle 12 Accuracy and reliability CoP / PC Principle 13 Timeliness and punctuality

Coherence and comparability Topics covered Internal coherence Aggregation checks, outlier tests External coherence Consistency with similar or related data sets Other risks Others CoP / PC Principle 14 Coherence and comparability

Sources used BoP Indicators Eurostat database BoP book 2007 (ECB) SDDS (IMF) Quality reports 2013 (Eurostat) Assessment of the QR 2012 (Eurostat) Quality Report on BoP of MS to European Parliament 2011 (Eurostat) Quality Report on Euroarea data 2013 (ECB)

Sources used Financial Sector Indicators Eurostat database Manual on sources and methods for the compilation of ESA95 financial accounts 2002 (Eurostat) Manual on sources and methods for the compilation of ESA95 financial accounts, 2nd edition – 2011 (Eurostat) Communication from the Commission to the European parliament, the Council and the Eurogroup (Eurostat) Other (websites, other documents)

Sources used General Government Debt Eurostat database EDP Inventories on sources and methods (MS to Eurostat) EDP Mission reports Other (websites, other available documents)

Sources used Share of world export Eurostat database European Union balance of payments/international investment position statistical methods (ECB) SDDS - Balance of Payments (IMF) BOP Quality Reports (Eurostat)

Sources used Nominal Unit Labour Cost Index Databases (Eurostat, OECD, National institutes) SDDS metadata (IMF) Joint OECD/Eurostat questionnaire on NA employment and hours worked Task Force Report on the Quality of the LFS 2009 (Eurostat)

Sources used House Price Index Eurostat Database HPI inventories mapped to ESMS (Eurostat)

Sources used Unemployment rate Eurostat database Quality reports (Eurostat / MS)

Sources used Real Effective Exchange Rate "Self assessment" made by Eurostat

Fitness Index On the basis of the scores assigned to the different questions we have compiled a single Fitness for Purpose index. This was obtained aggregating the quantitative scores and the weights assigned to the questions. The final index is normalised and ranges between 0 (maximum risk) and 100 (totally safe).

Benchmarking Our answers were checked by domain managers in Eurostat. More in detailed they were asked to: Challenge our scores Complement with additional information (if any) Special case for General Government Debt

Assessment by Country The information is already partially available but it has not been fully exploited yet. Example

Evolution of the Safebook excercise 2013 2014 Only internal MIP project Endorsed by Eurostat management MIP team making assessment Benchmarked by domain managers Only IDR Countries (+ Croatia) All Member States Improved questionnaire

Future Self-assessment by domain (short term) Assessment by Member States (long term) Run the exercise for MIP auxiliary indicators

What is your experience? Brainstorming group work How would you imagine doing a similar quality assessment exercise? Is a quality assessment done for your domain? If yes, how is it done? If not, would it be needed?