Practical use cases of SDMX: Census Hub Nadezhda Vlahova Eurostat Unit B3: “IT for statistical production” SDMX Basic course, 2016 1

The European Census Hub: key issues Dissemination of the data from the 2011 population and housing censuses in the European Union Data that are methodologically comparable and structured according to “hypercubes” agreed with Member States (Census Regulation) Providing users with an easy access to detailed census data and metadata (advanced functionalities) Management of massive amounts of data produced and controlled by Member States Maximum flexibility to cross-tabulate data from different sources 2

EU Census: Implementing measures Regulation (EC) 763/2008 on population and housing censuses authorises the European Commission to adopt implementing measures on: technical specifications of the topics and their breakdown (Regulation (EC) 1201/2009) programme of the statistical data and metadata to be transmitted to Eurostat (Regulation (EU) 519/2010) quality reporting and technical format of data transmission (Regulation (EU) 1151/2010)

What we need? Environment for dissemination of massive amounts of harmonised data which is easy to use and reuse Possibility to compare and cross-tabulate topics of interest, on data collected from different sources Countries Census data and metadata should be made available by 31 March 2014 and stored until 2025 Single point to access and retrieve detailed census data, numerical and textual metadata Standardized approach for similar data collections No validation and aggregation on the fly are required, neither supported

The Hub concept The Hub is based on the concept of data sharing, where a group of partners agree on providing access to their data according to standard processes, formats and technologies. The SDMX Hub approach offers several advantages: decoupling of NSIs' systems from the central hub via standard formats and techniques for the exchange; Limited investment, re-usability (with the advantage of using recognized international standards). Software (SDMX-RI) is supplied by Eurostat. 5

Hub approach – PULL data for collection and dissemination register SDMX Registry query NSI P U L Hub Dissemination Received data in SDMX-ML Eurostat Pull Requestor Loader Eurobase Dissemination Verification / Conversion To SDMX eDAMIS P U S H XSL for SDMX-ML Warehouse storage Intermediate storage Data Input

What is the Census Hub ? Interoperability/Initiative (32 countries participating in the project) Business driven project Delivered: Information system to query and display SDMX formatted data (e.g. Census 2011 data) retrieved from different data providers – Census Hub Universal framework for exposure and translation in SDMX-ML format of data stored in a legacy dissemination database – SDMX-Reference Infrastructure (SDMX-RI)

Prerequisite: SDMX compliance* (1/2) Agreed standard/format for data exchange Defined definitions, concepts, codelists, DSDs, transmission model and obligations Requires: NSIs to create and transmit data according to: agreed format mode of data exchange defined time period * Prerequisite for SDMX tools and SDMX exchange

SDMX compliance (2/2) Data preparation Macrodata database Statistical variables (topics) Microdata database Aggregated dataset n Individual records Non-SDMX local dissemination data Aggregated dataset 5 Aggregated dataset 1 Aggregated dataset 2 Aggregated dataset 4 Aggregated dataset 3

Example: DSD for Table 6 (Marital Status) Dimensions Attributes ID CONCEPT CODELIST TIME Time period or range CL_TIME GEO Geographical area CL_GEO SEX Sex CL_SEX FST Family status CL_FST LMS Legal marital status CL_LMS CAS Current activity status CL_CAS POB Country/place of birth CL_POB COC Country of citizenship CL_COC AGE Age CL_AGE FREQ Frequency CL_FREQ ID ATTACHMENT LEVEL CODELIST OBS_STATUS Observation CL_OBS_STATUS OBS_LEVEL CL_OBS_LEVEL OBS_NOTE HC_NOTE Series Measures ID NAME OBS_VALUE Observation value

Census datasets (hypercubes) Hypercube HC06 GEO.L. SEX. FST.H. LMS. CAS.L. POB.M. COC.M. AGE.M. (57) (3) (17) (13) (6) (15) (28) Theoretical number of cells = 57 * 3 * 17 * 13 * 6 * 15 * 13 * 28 = … = 1,238,033,160 cells (NB: for one country and one table…)

Census datasets (hypercubes) Hypercube HC04 GEO.L. SEX. HST.H. LOC. CAS.L. POB.L. COC.L. AGE.M. Using the naming conventions of Eurobase its name would be: Population in NUTS2 regions by sex, five-years age groups, household status, size of the locality and broad groups of current activity status, place of birth and country of citizenship  we need to change the way in which we give access to data

SDMX exchange and supporting tools Process workflow NSI SDMX codes Extract files Transform file SDMX file Dissemination/Transmission SDMX Converter NSI software NSI software SDMX Converter Processing for sending EDAMIS SDMX-RI Non-SDMX local data Processing for sending Mapping Assistant Test client NSI client EDAMIS NSI Web service HUB NSI development EDAMIS NSI software Eurostat tools NSI developed software

SDMX-RI architecture overview SDMX-RI – User Interfaces Data providing organisation Data collecting organisation Web/Test Client Mapping Assistant Internal network SDMX-RI – “Under the hood” Non-SDMX local data SDMX-formatted data 14 14

Census Hub – Standard approach Data Provider = NSI Data Collector SDMX (STANDARD, EXCHANGE, METADATA, REPOSITORY) WEB SERVICES DSD DSD Mapping Assistant Metadata repository Mapping store DSD Web Client SDMX query Test Client SDMX response Census Hub Non-SDMX local database

Data Hub is: No data processing (no editing or aggregation) System of DSDs built in SDMX 2.0 and in use for 32 countries of the ESS Data dissemination portal based on SDMX data model communicating with data providers via SDMX Web Service No data processing (no editing or aggregation) Additional reusable modules for Configuration management Tool for handling SDMX structural metadata Innovative user interface -> extract data starting from a statistical concept MSs status: 32 up and running

A data user can… Browse the Hub to define a dataset of interest, navigating via structural metadata: Search by topic (filters) and select data (level of detail, breakdowns) Select layout (axes) View a table Export a file (CSV, Excel, SDMX-ML) User registration Registered users can: Save, retrieve, modify or delete stored queries Receive an e-mail notification when offline queries are executed 17

How the Hub works National Statistical Institute Eurostat Census Hub 18 18 18

https://ec.europa.eu/CensusHub2/ 19

Reusability Installed within and outside ESS Used for multiple domains Used for dissemination and reporting Supports data sharing, push and pull modes Generic and SDMX based (2.0 and 2.1) Extensible and modular approach Free, open source solution, maintained by Eurostat Support different platforms and DB vendors

For more information ESTAT-CENSUSHUB@ec.europa.eu https://webgate.ec.europa.eu/fpfis/mwikis/sdmx https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/Census_Hub 21

Thank you for your attention 22

DEMO

DEMO