Reengineering of Administrative Data Acquisition Maura Giacummo, Eleonora Sibilio, Guido Drovandi, Paolo Giacomi (ISTAT - Italian National Istitute of Statistics) NTTS – Brussels 15 March 2017

ISTAT modernization process ISTAT designed and implemented a new statistical production and organizational model, reviewing the previous model deeply and pervasively. This innovation will result in the improvement and full exploitation of the direct statistical surveys results, by integrating them with administrative data Administrative data : must have a security policy of acquisition must be immediately available to subsequent processes of linkage and dissemination

Administrative Data acquisition Administrative Data acquisition for statistical purposes has increased from 90 data sets in the 2009 to about 300 in the 2016. It involves about 100 source subsets 40 data providers for more than a terabyte of data, mainly containing personal data. design engineered processes are needed in order to automate and to manage the acquisition and distribution of administrative data.

A.D. acquisition and monitoring system ARCAM Engineered process for Administrative Data acquisition Strengths: security of data transmission - guaranteed through ISTAT standard technologies. different ways of data transmission - meet the suppliers’ needs centralized repository - ensure compliance with the legislation on the data treatment metadata allow the monitoring of the acquisition process

Administrative Data Acquisition INTERNET Provider Administrative Data are saved into a temporary storage area to allow the provider to modify them. Temporary Area Temporary Area Temporary Area INTRANET Permanent Repository Once a data set is moved into the permanent repository, changes are not allowed

Web Acquisition System transferring files procedure Trasmit AD using a simple web application INTERNET Web Acquisition System Provider send files regardless of the size in multiple working sessions. File Data Files are divided into two different typologies: data and documentation. files. Provider Chunks of fixed dimension (1 MB) sent to the server sequentially. File Data Metadata If exception occurs or user closes the session it is possible to continue the transmission from the last stored chunk. Temporary Area ARCAM database Metadata transferring files procedure INTRANET Permanent Repository

Transmit AD using a simple web application XMLHttp Request browser feature - in order to manage huge files SHA-1 - used to compute the hash, to verify data transmission Data confidentiality - only users of same supplier operate on their data sets Different users can collaborate to upload same data but are not allowed to download them

Real-time monitoring and management data transmissions INTERNET Checks data supplies status reports Sends reminders to users not complying with the assigned deadlines Assigns transmission channel (HTTPS, FTP or WS) to data supplies Admin user Enables/disables data supplies for providers Enables/disables requesting users (end points of transmission) Rejects data supplies due to errors or inconsistencies ARCAM Database Setup the new annual data supplies timetable using previous timetables as templates. INTRANET Metadata Delivery system Data download Data Permanent Repository Requesting user

Database Outline Scheme The database managing both descriptive information of dataset and information relating to data acquisition Version File data Data Supply Function Source subsets Provider User Profile

Transferring files procedure Acquisition, monitoring, dissemination INTERNET Acquisition Metadata File data ARCAM Admin Monitoring ARCAM Database Provider Temporary Area Metadata Transferring files procedure INTRANET Delivery system A.D. download Requesting user Permanent Repository

ARCAM was successfully used from 2016 Conclusions ARCAM was successfully used from 2016 About a thousand files have been acquired through ARCAM 98 % via HTTPS channel 2% via SFTP channel from 150 users belonging to 40 providers It is planned to extend the ARCAM domain to other type of data like: Surveys data Scanner Data