SDMX IT Tools Introduction Jean-Francois LEBLANC Christian SEBASTIAN November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange

Table of contents Where are we? Standardization Why do we need a model? GSBPM Generic Statistical Business Process Model Phases Key features Other uses Standards – Relations GSIM Generic Statistical Information Model

Table of contents SDMX & DDI SDMX Summary Why? Benefits Costs Opportunities Impacts From 1.0 to 2.1 The SDMX components SDMX in practice Summary

Where are we? Dramatic changes in the environment of official statistics producers (e.g. data deluge) Modernization of statistical information system seen as a question of survival for the sector of official statistics Standardization viewed as a key enabler for modernization "Standards-based” industrialization of statistical production

2. Standardization Why is it necessary? What does it imply? Harmonization Reusability and interoperability Shared solutions across statistical institutes What does it imply? Common processes Common tools Common methodologies Why is it necessary? Harmonization of statistical data (leading to more comparable data) Reusable, interoperable data for users Economies of scale across statistical institutes internationally - shared solutions Good vendor support for the industry

2. Standardization Industry Standards Other major standards GSBPM - Generic Statistical Business Process Model GSIM - Generic Statistical Information Model SDMX - Statistical Data and Metadata eXchange DDI - Data Documentation Initiative Other major standards RDF - Resource Description Framework LOD - Linked Open Data JSON - JavaScript Object Notation XBRL - eXtensible Business Reporting Language GSBPM GSIM SDMX DDI SDMX Preferred standard for exchange and sharing of data and metadata in the global statistical community (UNSC, 2008) – Widely used in the ESS for aggregated data DDI Standard for the documentation of data, initially focused on archiving micro-data in the area of social sciences – widely used in national data archives – extended to support the full life-cycle of data RDF W3C standard for web-based discovery, dissemination, and linking – an alternative to XML Allows for after-the-fact linking of any types of resources on the Web, e.g. linking press releases and speeches with relevant data from a statistical organization Powerful querying language for distributed searches on the web Very popular with “open data” and “open government” initiatives JSON Web-developer-friendly alternative to XML. JSON version of SDMX XBRL Standard for reporting accounting information and banking supervision data XML-based standard No formal model Communities standardize the taxonomies to support their needs Mapping to other models requires an understanding of the implied model of the community Good tools are required to hide this complexity

3. Why do we need a model? To define and describe statistical processes in a coherent way To standardize process terminology To compare and benchmark processes within and between organisations To identify synergies between processes To inform decisions on systems architectures and organisation of resources

4. GSBPM Generic Statistical Business Process Model Applicable to all activities undertaken by producers of official statistics -> data outputs Used by National and international statistical organisations Independent of data source, can be used for: Surveys / censuses Administrative sources / register-based statistics Mixed sources

4.1 GSBPM - Phases

Not a linear model 4.2 GSBPM – Key features Sub-processes do not have to be followed in a strict order It is a matrix with many possible paths, including iterative loops within and between phases Some iterations of a regular process may skip certain sub-processes

4.3 GSBPM – Other uses Harmonizing statistical computing systems Facilitating sharing of statistical software Framework for process quality management Structure for storage of documents Measuring operational costs

5. Standards - Relations Statistical Information concepts concepts Statistics production GSBPM GSIM Technology Methods Conceptual Practical Statistical how-to Production how-to SDMX, DDI, RDF, ISO-11179, …

6. GSIM Generic Statistical Information Model Other standards DDI SDMX Implementation standards Conceptual model

7. SDMX & DDI DDI offers a very rich model for the documentation of micro-data SDMX offers a very integrated exchange platform for statistical outputs (IT architectures, tools, web services) integration of the complete production process The combined use of both standards could allow a higher level of

8. SDMX Statistical Data and Metadata eXchange UNSD World Bank

SDMX is the global answer to this. 8.1 SDMX – Why? The exchange of statistical data and metadata is complex, resource intensive and expensive In the past, national and international organisations had developed specific approaches and solutions Opportunities and challenges related to new technologies for machine to machine exchange were coming up, e.g. XML, web services. SDMX is the global answer to this.

8.2 SDMX - Benefits Efficiency Reduced burden after low investment Consistent and comparable data and metadata messages produced by different organizations Harmonized statistical processes, offering new ways of data and metadata exchange (such as data hubs) Web-based dissemination formats are provided that are computer “readable” and easier to update.

8.3 SDMX - Costs Development/maintenance of the SDMX standards and guidelines done by the international sponsoring institutions (supported by NSIs) Standards are public and open source IT tools are created by sponsoring or other organizations and made freely available Capacity building by individual sponsoring institutions User community input by means of open process Low investment cost – gradual implementation

8.4 SDMX - Opportunities Streamline data flows Across domains Central management (SDMX Registry) Across domains Across organizations Simplification Standardization Harmonization Software tools Data sharing Data structures Concepts Code lists

8.5 SDMX - Impacts Reduced reporting burden via common formats adopted by international organizations for data and metadata exchange User-friendly access when publishing national data and metadata on the web via global standards for data formats, catalogs/registries and associated services Improved management and analysis of data via global guidelines for metadata vocabularies and repositories in common formats Replicable models and tools for statistical information systems at national levels

SDMX recognised and supported as the preferred standard 8.6 SDMX – From 1.0 to 2.1 Version 2.0 SDMX-EDI SDMX-ML SDMX Registry 2008 SDMX accepted at UN level Version 1.0 GESMES/TS SDMX recognised and supported as the preferred standard Version 1.0 Version 2.0 Version 2.1 September 2004 November 2005 February 2008 April2011

8.7 The SDMX Components Describe statistics in a standard way Objects and their relationships Data Structure Definition (DSD), Concepts, Code List Central management and standard access SDMX Registry, SDMX Web Services Cross Domain Concepts Cross Domain Code Lists Statistical Domains Metadata Common Vocabulary Push Provider generates and sends file to receiver Pull Provider opens web service to data Receiver downloads regularly Hub Special case of pull: receiver downloads on end user request 22 22

9. Summary To enable a modernized statistical production, standards are the key Standards at different levels are being used in an increasingly coherent way GSBPM and GSIM provide conceptual models and facilitate communication SDMX, DDI and other standards provide implementation models which can be used in a coordinated way There are now more technologies than just GESMES and XML: a coherent overall model is critical

Introduction