1. Access to European microdata for scientific purposes
DIME, 23 February 2018
Item 13
Fabian Bach, Aleksandra Bujnowska
ESTAT.B.1
Eurostat

2. Outline
Introduction
Access to European microdata for scientific purposes - recent activities
Microdata access: plans for 2018+
Confidentiality on the fly
Eurostat

3. Need for modernisation in microdata access
1. Introduction
Need for modernisation in microdata access
2002 access to microdata regulated by EU law
2013 Reg. replaced: procedures changed (new types of research organisations, enlarge the scope of the European microdata sets , allow new access modes)
~750 eligibility agreements signed all over the world ~1500 applications for access received (growing yearly) +5 new datasets have become available for scientific use
Two access modes:
Continuous modernisation (handle volume + improvements)
Secure use files
Scientific use files
Eurostat

4. Alignment to the personal data protection framework
2. Recent activities
Alignment to the personal data protection framework
European Data Protection Supervisor (EDPS) - assessment of microdata access procedures
EDPS recommendations concerned mostly treatment of applications from entities located outside EU/EEA
Information to respondents and data users about rights and obligations related with personal data protection framework
Eurostat

5. On-line system for microdata access
2. Recent activities
On-line system for microdata access
research proposal application webform
Eurostat

6. On-line system for microdata access
2. Recent activities
On-line system for microdata access
NEW Eurostat Workflow tool
Eurostat

7. Public use files: EU-LFS & EU-SILC
2. Recent activities
Public use files: EU-LFS & EU-SILC
2015 Anonymisation methodology developed by Centre of Excellence on statistical disclosure control;
Data concerned: EU Labour Force Survey, EU Statistics on Income and Living Conditions
2016 Validation of the method by the WG on Methodology
2017 Application of the methodology on all countries data
(by Eurostat)
EU-SILC: consultation with countries, 20 agreements
EU-LFS: anonymisation on-going, consultations to be launched soon
Eurostat

8. On-line microdata transmission (implementation on-going)
2. Recent activities
On-line microdata transmission (implementation on-going)
Currently 12 microdata sets …
LFS
SILC
HBS
Eurostat

9. On-line microdata transmission (implementation on-going)
2. Recent activities
On-line microdata transmission (implementation on-going)
Currently 12 microdata sets …
LFS
SILC
HBS …
Research Project 1
Research Project X
Eurostat

10. Microdata access: plans for 2018+
New datasets
New access modes
Confidentiality on the fly
Eurostat

11. Microdata access: plans for 2018+
New datasets
New access modes
Confidentiality on the fly (next slides)
Eurostat

12. Cell suppression and 'c' flags
4. Confidentiality on the fly
Cell suppression and 'c' flags
Secondary suppression
Consistency issues
Generic drawbacks
Traditional method to address disclosure risks:
Cell suppression ('c' flags)
Eurostat

13. Cell suppression and 'c' flags
4. Confidentiality on the fly
Cell suppression and 'c' flags
Secondary suppression
Consistency issues
Generic drawbacks
Traditional method to address disclosure risks:
Cell suppression ('c' flags)
'c' flags sometimes misused / misunderstood
EU totals sometimes not published
Eurostat issues
Human intervention
Eurostat

14. Eurostat issues with 'c' flags / suppression
4. Confidentiality on the fly
Eurostat plans
Eurostat issues with 'c' flags / suppression
Raise awareness on 'c' flags
Intended function
Correct application
Eurostat

15. Eurostat issues with 'c' flags / suppression
4. Confidentiality on the fly
Eurostat plans
Eurostat issues with 'c' flags / suppression
Raise awareness on 'c' flags
Intended function
Correct application
Investigate modernization
Automatized methods
Recent developments using random noise (ABS etc.)
Eurostat

16. Eurostat issues with 'c' flags / suppression
4. Confidentiality on the fly
Eurostat plans
Eurostat issues with 'c' flags / suppression
Raise awareness on 'c' flags
Intended function
Correct application
Investigate modernization
Automatized methods
Recent developments using random noise (ABS etc.)
Eurostat

17. Random noise & cell key method
4. Confidentiality on the fly
Random noise & cell key method
Microdata
Safe tables
Eurostat

18. Random noise & cell key method
4. Confidentiality on the fly
Random noise & cell key method
Microdata
Cell key module
 ensures consistency
 same for all statistics
Noise module
 methodological part
 adapts to statistics
assign fixed record keys
query records in cell
calculate cell key from record keys
val_out = noise_function (val_in, cell key)
Table builder + cell key method
Safe tables
Eurostat

19. Random noise & cell key method
4. Confidentiality on the fly
Random noise & cell key method
Microdata
Cell key module
 ensures consistency
 same for all statistics
Noise module
 methodological part
 adapts to statistics
assign fixed record keys
query records in cell
calculate cell key from record keys
val_out = noise_function (val_in, cell key)
Table builder + cell key method
(additivity module)
Safe tables
Eurostat

20. 4. Confidentiality on the fly
Data pilots: 2021 EU Census
ESSnet census protection project recommends Cell Key Method for 2021 census (agenda item 3.2):
additive noise for unweighted (census) frequency tables
SAS implementation provided
Eurostat

21. 4. Confidentiality on the fly
Data pilots: 2021 EU Census
ESSnet census protection project recommends Cell Key Method for 2021 census (agenda item 3.2):
additive noise for unweighted (census) frequency tables
SAS implementation provided
Possible added value for "confidentiality on the fly":
planned code translation to open source (e.g. R)
existing methodology and straightforward to extend to other statistics (weighted samples, magnitudes)
Eurostat

22. Data pilots: LFS ad-hoc tables
4. Confidentiality on the fly
Data pilots: LFS ad-hoc tables
Eurostat service: extract ad-hoc tables from LFS data
Old approach:
Combination of 'c' + 'a' flags, suppression depends on user
No longer supported by ~10 Member States
 service currently suspended
Eurostat

23. Data pilots: LFS ad-hoc tables
4. Confidentiality on the fly
Data pilots: LFS ad-hoc tables
Eurostat service: extract ad-hoc tables from LFS data
Old approach:
Combination of 'c' + 'a' flags, suppression depends on user
No longer supported by ~10 Member States
 service currently suspended
Mitigation proposal: no 'c' flag, if needed random noise (RN)
10 MS
Other MS
EU28, EA19
Approved users:
All cells + RN
All cells
Other users:
+ suppress 'a'
Eurostat

24. Goal: table builder tool
4. Confidentiality on the fly
Goal: table builder tool
"Confidentiality on the fly" in short:
Develop tool for BLACK BOX
Safe tables
Microdata
Eurostat

25. Goal: table builder tool
4. Confidentiality on the fly
Goal: table builder tool
"Confidentiality on the fly" in short:
Develop tool for BLACK BOX
Safe tables
Microdata
flexible microdata back-end interface
portability into other production env's
Eurostat

26. Goal: table builder tool
4. Confidentiality on the fly
Goal: table builder tool
"Confidentiality on the fly" in short:
Develop tool for BLACK BOX
Live example: ABS TableBuilder
user-friendly + powerful front-end
Safe tables
Microdata
flexible microdata back-end interface
graded user access: basic  research  internal
portability into other production env's
Eurostat