1. Reengineering of Administrative Data Acquisition
Maura Giacummo, Eleonora Sibilio, Guido Drovandi, Paolo Giacomi
(ISTAT - Italian National Istitute of Statistics)
NTTS – Brussels
15 March 2017

2. ISTAT modernization process
ISTAT designed and implemented a new statistical production and organizational model, reviewing the previous model deeply and pervasively.
This innovation will result in the improvement and full exploitation of the direct statistical surveys results, by integrating them with administrative data
Administrative data :
must have a security policy of acquisition
must be immediately available to subsequent processes of linkage and dissemination

3. Administrative Data acquisition
Administrative Data acquisition for statistical purposes has increased
from 90 data sets in the 2009 to about 300 in the 2016.
It involves about
100 source subsets
40 data providers
for more than a terabyte of data, mainly containing personal data.
design engineered processes are needed in order to automate and to manage the acquisition and distribution of administrative data.

4. A.D. acquisition and monitoring system
ARCAM
Engineered process for Administrative Data acquisition
Strengths:
security of data transmission - guaranteed through ISTAT standard technologies.
different ways of data transmission - meet the suppliers’ needs
centralized repository - ensure compliance with the legislation on the data treatment
metadata allow the monitoring of the acquisition process

5. Administrative Data Acquisition
INTERNET
Provider
Administrative Data are saved into a temporary storage area to allow the provider to modify them.
Temporary Area
Temporary Area
Temporary Area
INTRANET
Permanent Repository
Once a data set is moved into the permanent repository, changes are not allowed

6. Web Acquisition System transferring files procedure
Trasmit AD using a simple web application
INTERNET
Web Acquisition System
Provider send files regardless of the size in multiple working sessions.
File Data
Files are divided into two different typologies: data and documentation. files.
Provider
Chunks of fixed dimension (1 MB) sent to the server sequentially.
File Data
Metadata
If exception occurs or user closes the session it is possible to continue the transmission from the last stored chunk.
Temporary Area
ARCAM database
Metadata
transferring files procedure
INTRANET
Permanent Repository

7. Transmit AD using a simple web application
XMLHttp Request browser feature - in order to manage huge files
SHA-1 - used to compute the hash, to verify data transmission
Data confidentiality - only users of same supplier operate on their data sets
Different users can collaborate to upload same data
but are not allowed to download them

8. Real-time monitoring and management data transmissions
INTERNET
Checks data supplies status reports
Sends reminders to users not complying with the assigned deadlines
Assigns transmission channel (HTTPS, FTP or WS) to data supplies
Admin
user
Enables/disables data supplies for providers
Enables/disables requesting users (end points of transmission)
Rejects data supplies due to errors or inconsistencies
ARCAM
Database
Setup the new annual data supplies timetable using previous timetables as templates.
INTRANET
Metadata
Delivery system
Data download
Data
Permanent Repository
Requesting
user

9. Database Outline Scheme
The database managing both descriptive information of dataset and information relating to data acquisition
Version
File data
Data Supply
Function
Source subsets
Provider
User
Profile

10. Transferring files procedure
Acquisition, monitoring, dissemination
INTERNET
Acquisition
Metadata
File data
ARCAM
Admin
Monitoring
ARCAM
Database
Provider
Temporary Area
Metadata
Transferring files procedure
INTRANET
Delivery system
A.D. download
Requesting
user
Permanent Repository

11. ARCAM was successfully used from 2016
Conclusions
ARCAM was successfully used from 2016
About a thousand files have been acquired through ARCAM
98 % via HTTPS channel
2% via SFTP channel
from 150 users
belonging to 40 providers
It is planned to extend the ARCAM domain to other type of data like:
Surveys data
Scanner Data