1. Open GSBPM compliant data processing system in Statistics Estonia (VAIS) 2011 MSIS Conference Maia Ennok Head of Data Warehouse Service Data Processing Systems Department Statistics Estonia 23th. of May 2011

2. Strategy of Statistics Estonia 2008–2011 “From data collector to information service provider” Objective: High-quality information service Standardise the process of data processing: Indicator: Introduction of the unified data processing software Working out and introduction of the universal data processing information system Open GSBPM compliant data processing system in Statistics Estonia (VAIS) 22.12.2015

3. Architecture of the information system Dissemination Statistical registers Metadata system Data collection Processing Statistical analysis Persons Administrative registers Users iMETA VVIS ADAM eGeostat SRS VAIS Economic entities eSTAT PX-Web Census-HUB KUNDE Data Warehouse 22.12.2015 Open GSBPM compliant data processing system in Statistics Estonia (VAIS)

4. Data processing system (VAIS) VAIS is a collection of tools and technologies aimed at automating data processing (Phase 5 in GSBPM). In essence, the task of check, clean, and transforming statistical activity data can be identified as taking the raw data from one or more sources and transforming it to analytical system source data input data base structures (observation registry). 22.12.2015 Open GSBPM compliant data processing system in Statistics Estonia (VAIS)

5. Framework for … Integrate data Classify & code Review, validate and edit Impute Derive new variables & statistical units Calculate weights Calculate aggregates Finalize data files 22.12.2015 Open GSBPM compliant data processing system in Statistics Estonia (VAIS)

6. Metadata driven template based tool Template driven approach provides an universal solution for three main goals of the VAIS project: Create an easy to use statistical data processing tool requiring minimal programming skills for transformation package creation. Create a metadata driven process-oriented and automated statistical data processing tool. Create an extendable data transformation tool. 22.12.2015 Open GSBPM compliant data processing system in Statistics Estonia (VAIS)

7. Design Phase 22.12.2015 Common Metadata Repository Data Sources for Statistical Activity N Validation Rules for Statistical Activity N Imputation Method for Statistical Activity N Aggregation Def for Statistical Activity N Data Sources for Statistical Activity N Target Dataset for Statistical Activity N Data processingng package (XDTL) for Statistical Activity N INTEGRATE DATA VALIADTE IMPUTE AGGREGATE INTEGRATE DATA LOAD DATA Common XDTL Package s INTEGRATE DATA VALIADTE IMPUTE AGGREGATE INTEGRATE DATA LOAD DATA Common XDTL Packages Open GSBPM compliant data processing system in Statistics Estonia (VAIS)

8. Data processing with VAIS 22.12.2015 Automating and speeding up data transformation Raw data, transformation metadata and source data audit trails Metadata driven template based tool Balancing automation and manual intervention Open GSBPM compliant data processing system in Statistics Estonia (VAIS)

9. VAIS architecture

10. Balancing automation and manual intervention 22.12.2015 RAW data Metadata (validation and transformation rules) Automated data processing Manual data processing OK? Data Warehou se Open GSBPM compliant data processing system in Statistics Estonia (VAIS)

11. VAIS applications and roles Open GSBPM compliant data processing system in Statistics Estonia (VAIS) 22.12.2015 RollVAIS Designer VAIS Operator VAIS Administrator URMA Designerx Data Warehouse programmer x Chief operatorx Operatorx Administratorxx

12. URMA User rights management application Allows using existing user for authorization Allows create roles and link users with roles Allows set rights according to domain statistical work Open GSBPM compliant data processing system in Statistics Estonia (VAIS) 22.12.2015

13. VAIS Designer Application for data processing design User interfaces for designing each processing procedures Procedures group to packages Packages setup fallows policy of ETL Packages are designed for each statistical work version Open GSBPM compliant data processing system in Statistics Estonia (VAIS) 22.12.2015

14. VAIS Operator Allows user to manually intervene to data processing. Allows to solve tasks created from data validation. Report of data processing gives overview of data in process. Gives users information for decision, that is necessary to solve tasks. Open GSBPM compliant data processing system in Statistics Estonia (VAIS) 22.12.2015

15. Technical platform VAIS is built on open-sourced freely available technological components. XDTL (eXtensible Data Transformation Language – an XML based descriptional language designed for specifying data transformations, see http://xdtl.org) run-time engine (XDTL RT).http://xdtl.org MMX Metadata Repository, part of Metadata Framework (a MOF compliant metadata management environment designed with a wide variety of metadata-driven applications in mind, see http://mmframework.org). http://mmframework.org Apache Foundation's Velocity template engine (http://velocity.apache.org) is used as the template engine combining excellent template rendering functionality with very easy to use template language.http://velocity.apache.org The user applications are programmed in Java, based on Wicket MVC framework (http://wicket.apache.org)http://wicket.apache.org Quartz scheduling framework (http://www.quartz-scheduler.org) is used for execution scheduling.http://www.quartz-scheduler.org 22.12.2015 Open GSBPM compliant data processing system in Statistics Estonia (VAIS)

16. Implementation VAIS development 05.2010- 10.2011 Data processing of Population and Housing Census 2011 (31.12.2011) Reuse administrative data (2012) Data collecting system for administrative data (ADAM) and eSTAT development for prefilling questionnaires in eSTAT with administrative data (annual bookkeeping report). (31.08.2011). VAIS is used for converting administrative data into the statistical data format. (for the year 2012 i.e for the reference year 2011 data collection) Data processing of other statistical activities (first pilots 2013) Data processing of next registry based Population and Housing Census (pilot 2014) 22.12.2015 Open GSBPM compliant data processing system in Statistics Estonia (VAIS)

17. Questions? Thank you!