Data validation in Statistical Office of the Republic of Serbia

Statistical Office of the Republic of Serbia As the backbone of the national statistical agency, SORS is the main producer of official statistics in the service of citizens and policy-makers. SORS has over 400 employees and 14 regional offices throughout Serbia; central office is located on three different locations in Belgrade. We annually conduct over 300 studies and various one-off projects. Many of the surveys are based on a monthly dynamic (that may span many years and decades), so at any moment the SORS has over 250 active applications (IT solutions). More than 90% of statistical production in SORS is done in IST IST is a software tool, metadata driven system, completely designed and developed in the Statistical Office of Serbia

Metadata driven codeless development environment IST concept: simple metadata database - IST metadata program that interprets metadata - IST program

Metadata driven codeless development environment IST has been fully developed on Microsoft development tools, on the .net technology and the IST metadata is on the MS SQL server IST is a .net application and a simple metadata database (only six tables) These two equally important components deal with a large number of different databases containing individual and aggregate data IST.net application is an interpreter. It, literally, reads data from the IST metadata that represents a description of each application, and based on the read data it generates and in real time executes each stage of statistical research or any other project, from data entry, data editing and validation to the dissemination of the data. The most important components of the IST.net application are: updating of the IST metadata, generated input data ('data entry interpreter'), generator for 'web' applications, data editing and data correction, automatic correction, advanced search of individual and aggregated data from the database, generating reports and the ability to manage procedures in database containing individual and aggregate data.

Rules of logical control in metadata database

Rules of logical control in metadata database All data validation rules for all statistical surveys conducted in our office are stored in ONE table of IST metadata database Simple rules for validation: built-in functions and the SQL statements Every physical table has aditional columns (boolean data type) for keeping information about errors in records for each validation rule described in IST metadata database and also one for whole record as a flag that record is not valid All Validation rules are described with SQL statements (just „WHERE“ clause of the UPDATE statement) After running batch logical control IST generates validation report Data validation is followed by corrective actions, such as data editing or data imputation. We are accessing records with errors trough same application

Logical control

Logical control - report

Data editing

Sending data to Eurostat First part of preparation of data for EUROSTAT and validation is done on IST platform All reporting procedures for all surveys are stored in one table in IST metadata database. Interpreter get data from database and generate report in .xls, .xml or .json format Although all statistical production has been carried out in the SORS on IST platform, there are no standard procedures for the way that data that is being sent to Eurostat at the institutional level. Every department and responsible statistician validate and send data using different tools and platforms provided by EUROSTAT. If data sent to Eurostat is not valid, in cooperation with subject meter experts, developers are adjusting validation rules in IST and after corrections they prepare new reports for Eurostat again Future plans: Cooperation with Eurostat regarding services for data and metadata validation and to explore is there a way to integrate developed services in our system, in order to improve statistical system and improve validation, data and metadata exchange

Thank you for your attention!