1. Guidelines on the use of estimation methods for the integration of administrative sources
DIME/ITDG meeting 2018/02/22

2. Contents Purpose of the presentation Objectives of the guidelines
Pre-requisites for integration
Usage scenarios
Typology of methods
Conditions for the use of methods
Structure of the guidelines
Purpose of the presentation
To provide information to the DIME/ITDG about the status of the project and to receive feedback and comments

3. Objective of the guidelines – 1
Nowadays statistical products regularly rely on different data sources
Such products are often called multi-source statistics
This project focused on estimation methods for the integration of different sources
Distinction between two kinds of data sources
Statistical data
Administrative data

4. Objective of the guidelines – 2
The guidelines consider the integration of these two kinds of sources
For the integration of statistical data and administrative data it is necessary:
To create an infrastructure which supports the integration
To distinguish different usage scenarios for the integration
To understand the conditions necessary for the application of the different methods

5. Objective of the guidelines – 3
For a number of usage scenarios the guidelines present:
A short description of methods which are applicable in the usage scenario
Decision rules for usage of the methods under more detailed specifications of the usage scenario
A recommendation of preferred methods if the usage scenario allows the application of different methods

6. Prerequisites for integration of administrative data – 1
For the integration of administrative data in the statistical production process an appropriate infrastructure must be available:
Appropriate IT infrastructure
Database architecture
Software for data access, data manipulation, and technical data integration
(statistical) software which supports the use of methods

7. Prerequisites for integration of administrative data – 2
A model for assessing quality of multi-source statistics
Administrative and organisational infrastructure
Arrangements with administrative units for data exchange including legal aspects (privacy, security)
Competent staff for the production of multi-source statistics

8. Prerequisites for integration of administrative data – 3
Useful sources for details about prerequisites:
ESSnet on quality of multisource statistics – KOMUSO
ESSnet on the use of administrative data and accounts data in business statistics

9. Usage scenarios for administrative data – 1
A bird’s eye view on usage scenarios:
Administrative data used either exclusively or in combination with survey data as source for statistical products (Direct usage)
Administrative data used as source for building and maintaining statistical registers (Indirect usage)
Administrative data used as support in the different sub-processes of the GSBPM (Indirect usage)
Integrate data, Edit & impute, Weighting and estimation, Calculate aggregates, Validation, ….

10. Usage scenarios for administrative data – 2
All usage scenarios use methods for data linkage (GSBPM sub-process 5.1: Data integration in a narrow sense) and require:
The identification of units
The identification of duplicates
Challenges are:
The harmonisation of units and measurements in different sources (alignment)
The presentation of the same figures for the same phenomenon in different sources (univalency)

11. A typology of methods – 1
Type A: Methods for different sub-processes of GSBPM:
Integrate data (5.1)
Edit & Impute (5.4)
Weighting and estimation (5.6 et. al. )
Alignment of statistical units and measurements (hidden in different sub-processes)

12. A typology of methods – 2
Type B: Methods which define a workflow for statistical production
Workflow for producing statistical products from administrative data (register based census)
Workflow for producing statistical products from administrative data in combination with survey data
Workflow for creation a statistical register
Workflow for updating statistical registers

13. A typology of methods – 3
Useful sources for methods (mainly type A methods):
ESS.VIP Admin WP 2: Statistical methods
ESSnet MEMOBUST
ESSnet Data integration
A.&B. Wallgren: Register-based Statistics

14. Conditions for the use of methods – 1
Besides the usage scenario the application of methods depend on some conditions for the data
Structural conditions for the data
Possible structures: micro-data, macro-data
Temporal structure: time series (longitudinal data)
Due to the fact that physical integration is done step by step it is sufficient to consider the structure only for the use of two datasets

15. Conditions for the use of methods – 2
Possible combination of the structural conditions:
Both datasets are microdata
Both datasets are macro-data
A combination between micro-data and macro-data
Systematic treatment of different temporal structures is at the moment not considered

16. Conditions for the use of methods – 3
Representation of the envisaged population:
Complete vs. incomplete
Disjoint vs. overlap
Representation of the variables of interest:
Unique representation (variable only in dataset) vs. multiple representation
Completeness with respect to statistical units vs. incompleteness (missing values)

17. Conditions for the use of methods – 4
Conditions specific to Type A methods (GSBPM sub-processes):
Characterisation of the variables (numeric, categorical,….)
Information about possible errors in the data
Conditions specific to Type B methods:
The basic methods are a detailed specification of the workflow and conceptual data modelling (for statistical units
Conditions for detailed specification of some of the sub-processes (e.g. mass imputation, or alignment of measurements)

18. Structure of the guidelines – 1
The guidelines are organised as follows
Introduction
The use of administrative data in statistical production (Basically this presentation)
Estimation methods for specific GSBPM and sub-processes in the case of micro-data and time series (editing, imputation, weighting and estimation
Alignment of units and measurements
The Integration of statistical and administrative micro-data
Macro-integration
Using administrative data for creation and maintenance of registers
Direct usage of administrative data for statistical products

19. Structure of the guidelines 2
The chapters 3 – 8 are self-contained and can be read independently
The structure of chapters 3 – 8 is as follows:
Description of the problem (usage scenario)
Conditions on the data for the possible methods
Short description of the possible methods
Evaluation of the methods:
A decision tree branching according to the conditions
If there are more than on methods in a terminal node the methods will be ordered from recommended up to not recommended