Transmission and preparation of data from secondary data sources at HCSO

Topics Background of the development  KARÁT - data transmission system ADAMES - data preparation system  In my presentation, I’ll focus on 3 major issues: Background of the development The Data Transmisiion system, which is called KARAT system And the data preparation system, which name is ADAMES So let’s start by the background of the development.

Secondary data sources Data sources at HCSO, 2016 Secondary data sources Primary data sources

Standard process model in HCSO: GSBPM v.4 adaptation - ESTFM

Data collection system of the HCSO Institutional data collections GÉSA, ELEKTRA Primary sources Population surveys LAKOS Data Collection Before 2015: no standard solution Secondary sources From 2015: KARAT

Data preparation at HCSO Institutional data collections ADÉL Primary sources Population surveys SAS Data Preparation Before 2015: no standard solution Secondary sources Forthcoming: ADAMES

Topics Background of the development  KARÁT - data transmission system ADAMES - data preparation system  

Main objectives of the development of the KARÁT system Documentation of secondary sources Documentation of data transmissions Secure channels for the transmission of the data sets Formal control and check of the transmitted data Loading data into data base Archiving data Supporting data providers with proactive functions Monitoring the data transmissions

Metadata for the KARAT system Data source register of the HCSO and the NSS Identification of data sources Description of the main characteristics of data sources Metadata for the secondary sources - KARAT metadata Reference periods and deadlines Type and structure of the datasets Other files to be transmitted Special processing tasks related to the datasets Special attachments to inform the data providers Message types to be sent to data providers and statisticians

Operating environment of KARÁT System-system type interfaces System-system type interfaces External appl. FOR SENDING TOWARDS HCSO External appl. FOR DATA REQUESTS FROM HCSO External data base ADAMES Internal data base META Internal KARÁT

The procedure of data transfer Design ORGANISATION RECEPTION DEADLINE CONTROL, REMINDERS VALIDATION OF DATASETS, Loading INto DATA BASE Download of data sets Monitoring of data transfers Passing to the next phase, Messages

Indicators computed by KARAT Quality indicators on secondary data sources Indicators on the documentation of secondary data sources Indicators on the complexity of data transmissions Indicators on the quality of data transmission processes Indicators on the accuracy of the data transmission processes Indicators on the timeliness of data transmissions Indicators on the respondent burden Number of datasets and documentation to be transmitted Channel of data transmission

Involvement of data sources in KARÁT Number of secondary data sources 266 Number of secondary sources involved in KARÁT 213 Number of secondary sources loaded directly into data base 20 Number of data transmission tasks 536 Number of realised data transmissions 402 Rate of realised data transmissions 75% 11 October, 2016

Topics Background of the development  KARÁT - data transmission system ADAMES - data preparation system   

Objectives of the development Standardised solution for the preparation of data sets transmitted to HCSO via KARÁT (as data base tables) General tool for statisticians (without the need for a regular IT support) Standardised communication with other systems of the HCSO

Main functions of ADAMES Data transfer from KARÁT to ADAMES Validation Other data preparation tasks Editing – manual or automatic Monitoring – quality indicators Transfer of the output to other IT systems of HCSO Other data preparation tasks: transformation, coding, definition of new variables… A rendszerbe külső tábla is beolvasható lehetővé téve az ellenőrzött állomány adatainak összehasonlítását máshonnan származó értékekkel. Az ADAMES-en belül egyszerűbb adat-összekapcsolások is megvalósíthatók

Introduction of the new system January 2015 – September 2016 : test period September 2016 – September 2017: pilot period From October 2017 : permanent operation

Conclusions KARÁT SYSTEM ADAMES SYSTEM GENERAL: for all secondary sources / all data release METADRIVEN: attributes described in meta database general procedures are parameterized by metadata process driven: the queue of actions and procedures is regulated Logged history management, logging, version management Proactive support the partners in data supply Standardised documentation system Secure channel Standardised processes Monitoring functions Standardised data preparation

Thank you for your attention! Ildikó Szűcs Hungarian Central Statistical Office Methodology Department Ildiko.Szucs@ksh.hu