1. 9. Practical use cases of SDMX: Census Hub
Nadezhda Vlahova Eurostat Unit B3: “IT for statistical production”
SDMX Basic course, 2017 1

2. The European Census Hub: key issues
Dissemination of the data from the 2011 & 2021 population and housing censuses in the European Union
Data that are methodologically comparable and structured according to “hypercubes” agreed with Member States (Census Regulation)
Providing users with an easy access to detailed census data and metadata (advanced functionalities)
Management of massive amounts of data produced and controlled by Member States
Maximum flexibility to cross-tabulate data from different sources 2

3. What we need?
Single point to access and retrieve detailed census data, numerical and textual metadata
Environment for dissemination and comparison of massive amounts of harmonised data which is easy to use and reuse
Countries Census data and metadata should be made available until 2035
Do not need
No validation and aggregation on the fly are required, neither supported

4. The Hub concept
The Hub is based on the concept of data sharing, where a group of partners agree on providing access to their data according to standard processes, formats and technologies.
The SDMX Hub approach offers several advantages:
decoupling of NSIs' systems from the central hub via standard formats and techniques for the exchange;
Limited investment, re-usability (with the advantage of using recognized international standards).
Software (SDMX-RI) is supplied by Eurostat. 4

5. Hub approach – PULL data for collection and dissemination
register
SDMX Registry
query
NSI P U L
Hub
Dissemination
Received
data in
SDMX-ML
Eurostat Pull Requestor
Loader
Eurobase
Dissemination
Verification /
Conversion
To SDMX
eDAMIS P U S H
XSL for
SDMX-ML
Warehouse
storage
Intermediate
storage
Data Input

6. What is the Census Hub ?
Interoperability/Initiative (32 countries participating in the project)
Business driven project
Delivered:
Information system to query and display SDMX formatted data (e.g. Census 2011/2021 data) retrieved from different data providers – Census Hub
Universal framework for exposure and translation in SDMX-ML format of data stored in a legacy dissemination database – SDMX-Reference Infrastructure (SDMX-RI)

7. Non-SDMX local dissemination data
Data preparation
Macrodata database
Aggregated dataset n
Non-SDMX local dissemination data
Aggregated dataset 5
Aggregated dataset 1
Aggregated dataset 2
Aggregated dataset 4
Aggregated dataset 3

8. Example: DSD for Table 6 (Marital Status)
Dimensions
Attributes ID
CONCEPT
CODELIST
TIME
Time period or range
CL_TIME
GEO
Geographical area
CL_GEO
SEX
Sex
CL_SEX
FST
Family status
CL_FST
LMS
Legal marital status
CL_LMS
CAS
Current activity status
CL_CAS
POB
Country/place of birth
CL_POB
COC
Country of citizenship
CL_COC
AGE
Age
CL_AGE
FREQ
Frequency
CL_FREQ ID
ATTACHMENT LEVEL
CODELIST
OBS_STATUS
Observation
CL_OBS_STATUS
OBS_LEVEL
CL_OBS_LEVEL
OBS_NOTE
HC_NOTE
Series
Measures ID
NAME
OBS_VALUE
Observation value

9. Census datasets (hypercubes)
Hypercube HC06
GEO.L.
SEX.
FST.H.
LMS.
CAS.L.
POB.M.
COC.M.
AGE.M.
(57)
(3)
(17)
(13)
(6)
(15)
(28)
Theoretical number of cells
= 57 * 3 * 17 * 13 * 6 * 15 * 13 * 28 = …
= 1,238,033,160 cells
(NB: for one country and one table…)

10. Census datasets (hypercubes)
Hypercube HC04
GEO.L.
SEX.
HST.H.
LOC.
CAS.L.
POB.L.
COC.L.
AGE.M.
Using the naming conventions of Eurobase its name would be:
Population in NUTS2 regions by sex, five-years age groups, household status, size of the locality and broad groups of current activity status, place of birth and country of citizenship
 we need to change the way in which we give access to data

11. SDMX-RI architecture overview
SDMX-RI – User Interfaces
Data providing organisation
Data collecting organisation
Web/Test Client
Mapping
Assistant
Internal network
SDMX-RI – “Under the hood”
Non-SDMX local data
SDMX-formatted data 11 11

12. SDMX exchange and supporting tools
Process workflow
NSI
SDMX codes
Extract files
Transform file
SDMX file
Dissemination/Transmission
SDMX Converter
NSI software
NSI software
SDMX Converter
Processing for sending
EDAMIS
SDMX-RI
Non-SDMX local data
Processing for sending
Mapping Assistant
Test client NSI client
EDAMIS
NSI Web service
HUB
NSI development
EDAMIS
NSI software
Eurostat tools
NSI developed software

13. How the Hub works National Statistical Institute
Eurostat Census
Hub 13 13 13

14. https://ec.europa.eu/CensusHub2/14

15. Census 2021 Project Roadmap
Input
Initiation 2017
Define the scope
Collect requirements
Define the outcome
Planning
Elaborate the scope
Create work packages
Design solutions
Execution
Implementation of the project plan
Produce the project deliverables
March 2021
Final version of the software ready for dissemination
Data ready to be disseminated
Census 2021
Legal base
Statistical WG
Social statistics WG
Design solutions which cover the use cases and requirements; which are generic enough and reusable; ensure the solutions are not obsolete
Census IT TF
Monitor & Control
End product:
Better use and presentation of social statistics data;
Modular and reusable;
Compare Census runs data;
Design post 2021?
Principles:
Reuse solutions, experiences and set up
Utilise the investment from 2011 Census
Reduce burden for data providers
Reduce cost

16. Census 2021 IT Task Force Composed by Census 2021 data providers
Focused on technical matters (IT experts)
Support the implementation of the Project Roadmap
Has own programme and deliverables
2017
2019
2020
2021
Design solutions which cover the use cases and requirements; which are generic enough and reusable; ensure the solutions are not obsolete
The task force should establish the references use cases for the community; draft requirements
2024
2018
Initiation
Define the scope and work programme
Elaborate on requirements
Define use cases
Analyse the changes of the regulation
Planning & Preparation
Prepare guidelines
Elaborate requirements
Share experience
Execution
Software installation
Training
Testing
March 2021
National set up prepared
Countries can provide data
2024
Data from all data providers are available

17. Data Hub is: No data processing (no editing or aggregation)
System of DSDs built in SDMX 2.0 and in use for 32 countries of the ESS
Data dissemination portal based on SDMX data model
communicating with data providers via SDMX Web Service
No data processing (no editing or aggregation)
Additional reusable modules for
Configuration management
Tool for handling SDMX structural metadata
Innovative user interface -> extract data starting from a statistical concept
MSs status: 32 up and running

18. A data user can…
Browse the Hub to define a dataset of interest, navigating via structural metadata:
Search by topic (filters) and select data (level of detail, breakdowns)
Select layout (axes)
View a table
Export a file (CSV, Excel, SDMX-ML)
User registration
Registered users can:
Save, retrieve, modify or delete stored queries
Receive an notification when offline queries are executed 18

19. Reusability Installed within and outside ESS Used for multiple domains
Used for dissemination and reporting
Supports data sharing, push and pull modes
Generic and SDMX based (2.0 and 2.1)
Extensible and modular approach
Free, open source solution, maintained by Eurostat
Support different platforms and DB vendors

20. For more information 20

21. Thank you for your attention21

22. DEMO

23. DEMO

29. EU Census: Implementing measures
Regulation (EC) 763/2008 on population and housing censuses authorises the European Commission to adopt implementing measures on:
technical specifications of the topics and their breakdown (Regulation (EC) 1201/2009)
programme of the statistical data and metadata to be transmitted to Eurostat (Regulation (EU) 519/2010)
quality reporting and technical format of data transmission (Regulation (EU) 1151/2010)