Theme (iv): Standards and international collaboration UNITED NATIONS ECONOMIC COMMISSION FOR EUROPE CONFERENCE OF EUROPEAN STATISTICIANS Work Session on Statistical Data Editing 24-26 April 2017 The Hague, Netherlands Theme (iv): Standards and international collaboration Discussants: Agnes Andics, Simona Rosati, Sander Scholtus

Introduction to theme (iv) Full title: Standards and international collaboration – including implementation of the new and emerging standards: VTL, GSDEMs, CSPA Development of methods, processes and software tools for data editing is often complex, time-consuming and costly Statistical organisations face similar challenges It seems natural to collaborate and share methods and tools However, sharing requires agreement on standards

Introduction to theme (iv) New and emerging standards: Generic Statistical Business Process Model (GSBPM) Generic Statistical Data Editing Models (GSDEMs) Generic Statistical Information Model (GSIM) Common Statistical Production Architecture (CSPA) Validation and Transformation Language (VTL) In this theme: presentations on developing data editing infrastructure based on these standards initiatives by individual statistical institutes and international projects

Agenda for theme (iv) Monday afternoon: Tuesday morning: Eurostat: The modernisation of validation in the ESS – a multidimensional approach Norway: Methods library as part of the modernization of the statistical production in Norway Poland: Technical aspects of VTL to SQL translation Tuesday morning: Norway: A new GSDEM of multisource data for multiple statistics Germany: Validation, Shared Services and Enterprise Architecture: How it fits Germany: The ESSnet ValiDat Integration General discussion

Summary: presentations in theme (iv) WP.13 The modernisation of validation in the ESS – a multidimensional approach (Eurostat) The ESS and the ambitious programme to improve the way data sent to Eurostat are validated. A multi-pronged approach chosen by the ESS to achieve its medium-term goals for validation: Ensure the transparency of the validation procedures through a common validation policy. Improve the interoperability between Eurostat and Member States. A concrete example of implementation of this approach in the National Account domain. ESS Validation principles

Summary: presentations in theme (iv) WP.14 Methods library as part of the modernization of the statistical production in Norway (Norway) A method library to offer common methods throughout the production process: CSPA-compatible R language R packages Wiki page for user documentation Separate graphics module Methods included: Macro-editing Imputation Confidentiality

Summary: presentations in theme (iv) WP.15 Technical aspects of VTL to SQL translation (Poland) VTL as a new international standard language for defining validation and transformation rules on statistical data: Different version of VTL. Translation from VTL to typical data processing languages. A source-to-source compiler – VtlProcessingLib – was developed to validate data and aggregate: A programming library that allows for translating source code of VTL to target code of different programming languages. The efforts are concentrated on developing overall software architecture independent on the language specification’s details.

Summary: presentations in theme (iv) WP.16 A new GSDEM of multisource data for multiple statistics (Norway) Editing multisource data for producing multiple statistics “Parallel editing” versus “coordinated editing” Advantages of coordinated editing: Avoids duplication of data editing functions Ensures harmonisation of related statistical outputs Possibility for generating new statistics Application at Statistics Norway: employment and wage statistics

Summary: presentations in theme (iv) WP.17 Validation, Shared Services and Enterprise Architecture: How it fits (Germany) Different types of architecture relevant to the production of statistics: Business Architecture (GSBPM, GSDEMs) Information Architecture (GSIM, LIM) Application Architecture (CSPA) Infrastructure Architecture HLG-MOS and ESS Vision 2020: move towards a service-oriented architecture (CSPA) Application of these ideas to validation (ESSnet ValiDat Integration)

Summary: presentations in theme (iv) WP.18 The ESSnet ValiDat Integration (Germany) ESSnet project of six national statistical institutes Builds on results of previous ESSnet ValiDat Foundation Handbook on methodology for data validation (2016) Goals for this project: Improve handbook Develop better validation metrics (to evaluate rules and data) Investigate costs and benefits of different strategies to implement validation services within the ESS

Questions / Points for discussion New GSDEM: coordinated editing Do other institutes have experiences with parallel editing / coordinated editing of multisource data? How to maintain coordination in practice between the statistic-specific editing functions beyond the intermediate statistical data?

Questions / Points for discussion Several initiatives presented to create new tools for data editing and validation that comply with standards (CSPA, VTL). What about re-use? Are there institutes already (planning to) re-use some of these tools in their own statistical production? Or are many institutes (planning to) develop their own shareable tools, which may lead to many competing services that offer a similar functionality?