Access to European microdata for scientific purposes DIME, 23 February 2018 Item 13 Fabian Bach, Aleksandra Bujnowska ESTAT.B.1 Eurostat

Outline Introduction Access to European microdata for scientific purposes - recent activities Microdata access: plans for 2018+ Confidentiality on the fly Eurostat

Need for modernisation in microdata access 1. Introduction Need for modernisation in microdata access 2002 access to microdata regulated by EU law 2013 Reg. replaced: procedures changed (new types of research organisations, enlarge the scope of the European microdata sets , allow new access modes) 2013-2018 ~750 eligibility agreements signed all over the world ~1500 applications for access received (growing yearly) +5 new datasets have become available for scientific use Two access modes: Continuous modernisation (handle volume + improvements) Secure use files Scientific use files Eurostat

Alignment to the personal data protection framework 2. Recent activities Alignment to the personal data protection framework European Data Protection Supervisor (EDPS) - assessment of microdata access procedures EDPS recommendations concerned mostly treatment of applications from entities located outside EU/EEA Information to respondents and data users about rights and obligations related with personal data protection framework Eurostat

On-line system for microdata access 2. Recent activities On-line system for microdata access research proposal application webform Eurostat

On-line system for microdata access 2. Recent activities On-line system for microdata access NEW Eurostat Workflow tool Eurostat

Public use files: EU-LFS & EU-SILC 2. Recent activities Public use files: EU-LFS & EU-SILC 2015 Anonymisation methodology developed by Centre of Excellence on statistical disclosure control; Data concerned: EU Labour Force Survey, EU Statistics on Income and Living Conditions 2016 Validation of the method by the WG on Methodology 2017 Application of the methodology on all countries data (by Eurostat) EU-SILC: consultation with countries, 20 agreements EU-LFS: anonymisation on-going, consultations to be launched soon Eurostat

On-line microdata transmission (implementation on-going) 2. Recent activities On-line microdata transmission (implementation on-going) Currently 12 microdata sets … LFS SILC HBS Eurostat

On-line microdata transmission (implementation on-going) 2. Recent activities On-line microdata transmission (implementation on-going) Currently 12 microdata sets … LFS SILC HBS … Research Project 1 Research Project X Eurostat

Microdata access: plans for 2018+ New datasets New access modes Confidentiality on the fly Eurostat

Microdata access: plans for 2018+ New datasets New access modes Confidentiality on the fly (next slides) Eurostat

Cell suppression and 'c' flags 4. Confidentiality on the fly Cell suppression and 'c' flags Secondary suppression Consistency issues Generic drawbacks Traditional method to address disclosure risks: Cell suppression ('c' flags) Eurostat

Cell suppression and 'c' flags 4. Confidentiality on the fly Cell suppression and 'c' flags Secondary suppression Consistency issues Generic drawbacks Traditional method to address disclosure risks: Cell suppression ('c' flags) 'c' flags sometimes misused / misunderstood EU totals sometimes not published Eurostat issues Human intervention Eurostat

Eurostat issues with 'c' flags / suppression 4. Confidentiality on the fly Eurostat plans Eurostat issues with 'c' flags / suppression Raise awareness on 'c' flags Intended function Correct application Eurostat

Eurostat issues with 'c' flags / suppression 4. Confidentiality on the fly Eurostat plans Eurostat issues with 'c' flags / suppression Raise awareness on 'c' flags Intended function Correct application Investigate modernization Automatized methods Recent developments using random noise (ABS etc.) Eurostat

Eurostat issues with 'c' flags / suppression 4. Confidentiality on the fly Eurostat plans Eurostat issues with 'c' flags / suppression Raise awareness on 'c' flags Intended function Correct application Investigate modernization Automatized methods Recent developments using random noise (ABS etc.) Eurostat

Random noise & cell key method 4. Confidentiality on the fly Random noise & cell key method Microdata Safe tables Eurostat

Random noise & cell key method 4. Confidentiality on the fly Random noise & cell key method Microdata Cell key module  ensures consistency  same for all statistics Noise module  methodological part  adapts to statistics assign fixed record keys query records in cell calculate cell key from record keys val_out = noise_function (val_in, cell key) Table builder + cell key method Safe tables Eurostat

Random noise & cell key method 4. Confidentiality on the fly Random noise & cell key method Microdata Cell key module  ensures consistency  same for all statistics Noise module  methodological part  adapts to statistics assign fixed record keys query records in cell calculate cell key from record keys val_out = noise_function (val_in, cell key) Table builder + cell key method (additivity module) Safe tables Eurostat

4. Confidentiality on the fly Data pilots: 2021 EU Census ESSnet census protection project recommends Cell Key Method for 2021 census (agenda item 3.2): additive noise for unweighted (census) frequency tables SAS implementation provided Eurostat

4. Confidentiality on the fly Data pilots: 2021 EU Census ESSnet census protection project recommends Cell Key Method for 2021 census (agenda item 3.2): additive noise for unweighted (census) frequency tables SAS implementation provided Possible added value for "confidentiality on the fly": planned code translation to open source (e.g. R) existing methodology and straightforward to extend to other statistics (weighted samples, magnitudes) Eurostat

Data pilots: LFS ad-hoc tables 4. Confidentiality on the fly Data pilots: LFS ad-hoc tables Eurostat service: extract ad-hoc tables from LFS data Old approach: Combination of 'c' + 'a' flags, suppression depends on user No longer supported by ~10 Member States  service currently suspended Eurostat

Data pilots: LFS ad-hoc tables 4. Confidentiality on the fly Data pilots: LFS ad-hoc tables Eurostat service: extract ad-hoc tables from LFS data Old approach: Combination of 'c' + 'a' flags, suppression depends on user No longer supported by ~10 Member States  service currently suspended Mitigation proposal: no 'c' flag, if needed random noise (RN) 10 MS Other MS EU28, EA19 Approved users: All cells + RN All cells Other users: + suppress 'a' Eurostat

Goal: table builder tool 4. Confidentiality on the fly Goal: table builder tool "Confidentiality on the fly" in short: Develop tool for BLACK BOX Safe tables Microdata Eurostat

Goal: table builder tool 4. Confidentiality on the fly Goal: table builder tool "Confidentiality on the fly" in short: Develop tool for BLACK BOX Safe tables Microdata flexible microdata back-end interface portability into other production env's Eurostat

Goal: table builder tool 4. Confidentiality on the fly Goal: table builder tool "Confidentiality on the fly" in short: Develop tool for BLACK BOX Live example: ABS TableBuilder user-friendly + powerful front-end Safe tables Microdata flexible microdata back-end interface graded user access: basic  research  internal portability into other production env's Eurostat