1. Centralised System for Administrative Data Collection at Statistics Finland
Kirsi-Marja Aalto, Perttu Muurimäki
, Brussels
New Techniques and Technologies for Statistics 2017

2. Centralized System for Administrative Data Collection at Statistics Finland – AVa
Most of the data used in statistical production are administrative data or registers – some 150 individual admin data from 50 different data providers
In the Statistics Act it is defined that official statistical organisations and some other organisations have the obligation to give Statistics Finland (statistical) data for producing statistics needed both by national and European Union authorities
More than two thirds of the admin data (in numbers) come via administrative data collection system, AVa
23 November 2018
Kirsi-Marja Aalto

3. Cooperation with data providers
Data providers are mostly government agencies and official organisations
With the biggest data providers Statistics Finland has skeleton agreements – the general terms of cooperation between Statistics Finland and the organisation
More specific matters are agreed in the data acquisition agreement, which is included in the skeleton agreement as enclosures
when and how often the data are sent
how the data will be transferred
With smaller data providers Statistics Finland has only data acquisition agreements
23 November 2018
Kirsi-Marja Aalto

4. Phases of administrative data collection system AVa
The AVa system consists of three phases: automatic phase, centralised phase and statistical production process.
23 November 2018
Kirsi-Marja Aalto

5. Ava process needs standardising
23 November 2018
Kirsi-Marja Aalto

6. Data in transit 1/2
For file transfer AVa uses generic file transfer service (FiTS) developed in Statistics Finland.
FiTS provides several key features:
Bi-directional file transfers from and to external data providers (file exchange)
Passive/active file transfers i.e. FiTS either waits for data providers to fetch/push files or FiTS fetches/pushes the files itself.
Pluggable file transfer methods for external data providers i.e. sftp, https and ftp protocols with various authentication methods (password, public key, certificate)
Pluggable file transfer methods to internal data receivers (sftp for unix/linux servers, smb for Windows shares and servers)
23 November 2018
Perttu Muurimäki

7. Data in transit 2/2
Method for grouping files into datasets by filename or filename wildcard and provider id
Method for routing datasets to one or more recipients
Configurable fetch poll scheduling (hourly, "sensible", daily, weekly)
Possibility to use hook-scripts in various phases of file transfer to e.g. unzip files, rename files or split files into several smaller files. Other uses are also possible.
Establishing FiTS connection for AVa requires that the connection details are first sorted out (provider id, protocols, usernames, servers, directories). Then AVa team uses provider id to route data to one or more recipients target directories (on target servers). After that data "just flows".
23 November 2018
Perttu Muurimäki

8. Process flow for getting new data
23 November 2018
Perttu Muurimäki

9. Future challenges New forms of data and the use of big data
The amount and the number of incoming data increases → more automatisation i.e. automatic technical validation, data transformation into a SAS dataset and routing data to their predetermined target folder.
To develop an interface that could be used to control and manage the whole process of centralised administrative data collection
23 November 2018
Kirsi-Marja Aalto

10. Benefits of centralized administrative data collection
More efficient, safe and systematic administrative data receiving processes
Better transparency of the utilized administrative data
Rationalised data requests and advanced data usage
Before
Statistics Finland
After
Statistics Finland
Statistical unit
Statistical unit
Data provider
Data provider
AVa
Statistical unit
Stat unit
Statistical unit
Statistical unit
. Stat unit
23 November 2018
Kirsi-Marja Aalto

11. Kirsi-Marja Aalto kirsi-marja.aalto@stat.fi
Perttu Muurimäki
NTTS – 2017