ESS.VIP ADMIN EssNet on Quality in Multi-source Statistics, progress report 19TH WORKING GROUP ON QUALITY IN STATISTICS, 6 December 2016 Fabrice Gras, Unit B1 – methodology and corporate architecture

ESS VIP.ADMIN Quality of multisource statistics Three specific grant agreements (SGA) foreseen under a framework partnership agreement (DK, IE, IT, LT, HU, NL, AT, NO): SGA1: exploratory work and gap analysis, development of some indicators (signed in 2015) SGA2: proposal of a structure for the guidelines and development of some chapters, including proper methods and indicators (to be signed in autumn 2016) SGA3: completion of the guidelines (to be signed in autumn 2017)

ESSnet on quality of multisource statistics estimated timetable Framework Partnership Agreement 08/2015 -08/2019 1st Specific Grant Agreement 01/2016 – 04/2017 2nd Specific Grant Agreement 05/2017 -05/2018 3rd Specific Grant Agreement 06/2018 – 08/2019

Quality of multisource statistics On-going first SGA of the ESSnet WP1: Quality of input data WP2: Frame Quality WP3: Quality of statistical output WP4: Communication, dissemination and implementation

ESSnet on quality of multisource statistics -goals Work Packages Contents WP1. Checklist for evaluating the quality of input data Review and testing of existing checklists Commented repository Recommendation on further action WP2. Methodology for the assessment of the quality of frames for social statistics Indicators Sufficiency criteria Metadata requirements Recommendation for a monitoring process WP3. Framework for the quality evaluation of statistical output based on multiple sources Critical review of existing indicators and approaches; testing Proposal of new approaches and indicators Implementation guidelines Recommendation for updating the for quality reports WP4. Dissemination and implementation Produce some quality reports Workshop – stock taking, exchange of experience Promotion of results

Quality of multisource statistics Input quality Quality of frames for social statistics Output quality

1st SGA Milestones: January 2016 – first ESSnet meeting April 2016 – workshop on quality aspects of using administrative sources in the production of statistics June 2016: final report for WP1,intermediate report for WP2 and WP3 and summary document on the workshop. September 2016: intermediate ESSnet meeting. November 2016: intermediate report for WP2 and WP3 March 2017: final ESSnet meeting April 2017: Final report for WP2 and WP3

WP1 objective: Creation of checklists for evaluating the quality of input data Subtasks: Critical review and testing of existing methodology Commented repository Consolidated version of checklist Identification of possible gaps

The work so far, WP1: A gross list was created with more than 500 indicators Members of the ESSNET selected indicators for testing Clear indicators with precise definitions Six dimensions are used (similarly to the ESSnet Admin project) At least 2 indicators for each dimension Our list uses 16 quantitative indicators Tests were carried out by 3 countries Report of WP1 has been circulated in the ESS and is under approval

First Specific Grant Agreement –WP1 on input data Gaps: Pre-checklist to use before data becomes available Check-list to compare two sources Deliverable Expected delivery Actual delivery Status Draft report on checklists for evaluating the quality of input data May 2016 June 2016 Circulated to MS Final report June 2016 August 2016 Under revision

WP2 objective: Produce guidelines on how to assess frame quality for social statistics Subtasks: Literature review Comparative analysis Gap analysis Proposal of quality measures Development and test

First Specific Grant Agreement –WP2 on frames for social statistics Gaps: The analysis list the existing approaches and levels of maturity Less mature approaches should be developed Deliverable Expected delivery Actual delivery Status Intermediate report on the quality of frames for social statistics June 2016 July 2016 Under revision Intermediate report Draft action plan November 2016   Will be circulated to MS Final action plan Final report April 2017

WP3 objective: Produce relevant measures for the quality of the output Subtasks: Critical review on eight topics: classification error, linkage error, under coverage error, measurement errors, macro-integration, repeated weighting, benchmarking and time series/small area estimation Suitability tests: 7 basic data configuration to be considered Action plan

The work so far, WP3: Definition of basic data configurations to be tested (Komuso classification) Literature review on: quality assessment of imputations in administrative data, unit non-response, coverage (frame) errors, capture-recapture techniques, and classification errors 11 suitability tests already carried out (use of latent variable, use of simulation, estimation of classification errors). Action plan: Making indicators more suitable for application in practice Existing gap: assessment of the coherence dimension of quality (difference in concepts or in methods)

First Specific Grant Agreement –WP3 on output Gaps: Several data configurations have been analysed Gaps for each configuration are identified Other data configurations could be covered Deliverable Expected delivery Actual delivery Status Intermediate report on the quality of statistical output based on multiple sources June 2016 July 2016 Under revision Draft action plan Intermediate report November 2016   Will be circulated to MS Final action plan Final report April 2017

Example of basic data configuration (1)

Example of basic data configuration (2)

First Specific Grant Agreement –WP4 on dissemination Deliverable Expected delivery Actual delivery Status Communication plan March 2016 June 2016 Accepted Workshop on quality of multisource statistics April/May 2016 April 2016 Workshop documentation May 2016 CROS portal page April 2016 On-going updates Workshop: Validation of the ESSnet approach Request to produce guidelines Communication: VIG, DIME, DSS, WG Methodology, WG Quality, ESS.VIP BIGD, ESS.VIP ESBRs, UNECE Data Integration, UNECE TF on census, Q2016, CESS 2016, NTTS 2017, ISI 2017.

Focus on the workshop: Knowledge of the ESSnet – its members and its workplan Presentation of good practices Discussion of the tasks and objectives of the ESSnet Discussion in general on quality Presentations available on the CROS portal at: https://ec.europa.eu/eurostat/cros/content/workshop-presentations-and-handouts_en

Next steps Second SGA: Third SGA Synergies Structure of guidelines Complete some parts of it Third SGA Finalise guidelines Practical examples/quality reports Synergies Other ADMIN WPs (applications, guidance on estimation methods) Planned implementing regulation on frames for social statistics ESS.VIP BIGD

Thank you for your attention! CONTACT: ESS. VIP ADMIN Sorina Vâju (Eurostat): Sorina-Carmen.Vaju@ec.europa.eu Fabrice Gras – fabrice.gras@ec.europa.eu ESSnet on quality of multisource statistics Niels Ploug (Statistics Denmark): npl@dst.dk