1. Practical use cases of SDMX: Census Hub
Nadezhda Vlahova Eurostat Unit B3: “IT for statistical production”
SDMX Basic course, 2016 1

2. The European Census Hub: key issues
Dissemination of the data from the 2011 population and housing censuses in the European Union
Data that are methodologically comparable and structured according to “hypercubes” agreed with Member States (Census Regulation)
Providing users with an easy access to detailed census data and metadata (advanced functionalities)
Management of massive amounts of data produced and controlled by Member States
Maximum flexibility to cross-tabulate data from different sources 2

3. EU Census: Implementing measures
Regulation (EC) 763/2008 on population and housing censuses authorises the European Commission to adopt implementing measures on:
technical specifications of the topics and their breakdown (Regulation (EC) 1201/2009)
programme of the statistical data and metadata to be transmitted to Eurostat (Regulation (EU) 519/2010)
quality reporting and technical format of data transmission (Regulation (EU) 1151/2010)

4. What we need?
Environment for dissemination of massive amounts of harmonised data which is easy to use and reuse
Possibility to compare and cross-tabulate topics of interest, on data collected from different sources
Countries Census data and metadata should be made available by 31 March 2014 and stored until 2025
Single point to access and retrieve detailed census data, numerical and textual metadata
Standardized approach for similar data collections
No validation and aggregation on the fly are required, neither supported

5. The Hub concept
The Hub is based on the concept of data sharing, where a group of partners agree on providing access to their data according to standard processes, formats and technologies.
The SDMX Hub approach offers several advantages:
decoupling of NSIs' systems from the central hub via standard formats and techniques for the exchange;
Limited investment, re-usability (with the advantage of using recognized international standards).
Software (SDMX-RI) is supplied by Eurostat. 5

6. Hub approach – PULL data for collection and dissemination
register
SDMX Registry
query
NSI P U L
Hub
Dissemination
Received
data in
SDMX-ML
Eurostat Pull Requestor
Loader
Eurobase
Dissemination
Verification /
Conversion
To SDMX
eDAMIS P U S H
XSL for
SDMX-ML
Warehouse
storage
Intermediate
storage
Data Input

7. What is the Census Hub ?
Interoperability/Initiative (32 countries participating in the project)
Business driven project
Delivered:
Information system to query and display SDMX formatted data (e.g. Census 2011 data) retrieved from different data providers – Census Hub
Universal framework for exposure and translation in SDMX-ML format of data stored in a legacy dissemination database – SDMX-Reference Infrastructure (SDMX-RI)

8. Prerequisite: SDMX compliance* (1/2)
Agreed standard/format for data exchange
Defined definitions, concepts, codelists, DSDs, transmission model and obligations
Requires:
NSIs to create and transmit data according to:
agreed format
mode of data exchange
defined time period
* Prerequisite for SDMX tools and SDMX exchange

9. SDMX compliance (2/2) Data preparation
Macrodata database
Statistical variables (topics)
Microdata database
Aggregated dataset n
Individual records
Non-SDMX local dissemination data
Aggregated dataset 5
Aggregated dataset 1
Aggregated dataset 2
Aggregated dataset 4
Aggregated dataset 3

10. Example: DSD for Table 6 (Marital Status)
Dimensions
Attributes ID
CONCEPT
CODELIST
TIME
Time period or range
CL_TIME
GEO
Geographical area
CL_GEO
SEX
Sex
CL_SEX
FST
Family status
CL_FST
LMS
Legal marital status
CL_LMS
CAS
Current activity status
CL_CAS
POB
Country/place of birth
CL_POB
COC
Country of citizenship
CL_COC
AGE
Age
CL_AGE
FREQ
Frequency
CL_FREQ ID
ATTACHMENT LEVEL
CODELIST
OBS_STATUS
Observation
CL_OBS_STATUS
OBS_LEVEL
CL_OBS_LEVEL
OBS_NOTE
HC_NOTE
Series
Measures ID
NAME
OBS_VALUE
Observation value

11. Census datasets (hypercubes)
Hypercube HC06
GEO.L.
SEX.
FST.H.
LMS.
CAS.L.
POB.M.
COC.M.
AGE.M.
(57)
(3)
(17)
(13)
(6)
(15)
(28)
Theoretical number of cells
= 57 * 3 * 17 * 13 * 6 * 15 * 13 * 28 = …
= 1,238,033,160 cells
(NB: for one country and one table…)

12. Census datasets (hypercubes)
Hypercube HC04
GEO.L.
SEX.
HST.H.
LOC.
CAS.L.
POB.L.
COC.L.
AGE.M.
Using the naming conventions of Eurobase its name would be:
Population in NUTS2 regions by sex, five-years age groups, household status, size of the locality and broad groups of current activity status, place of birth and country of citizenship
 we need to change the way in which we give access to data

13. SDMX exchange and supporting tools
Process workflow
NSI
SDMX codes
Extract files
Transform file
SDMX file
Dissemination/Transmission
SDMX Converter
NSI software
NSI software
SDMX Converter
Processing for sending
EDAMIS
SDMX-RI
Non-SDMX local data
Processing for sending
Mapping Assistant
Test client NSI client
EDAMIS
NSI Web service
HUB
NSI development
EDAMIS
NSI software
Eurostat tools
NSI developed software

14. SDMX-RI architecture overview
SDMX-RI – User Interfaces
Data providing organisation
Data collecting organisation
Web/Test Client
Mapping
Assistant
Internal network
SDMX-RI – “Under the hood”
Non-SDMX local data
SDMX-formatted data 14 14

15. Census Hub – Standard approach
Data Provider = NSI
Data Collector
SDMX
(STANDARD, EXCHANGE, METADATA, REPOSITORY)
WEB SERVICES
DSD
DSD
Mapping
Assistant
Metadata repository
Mapping store
DSD
Web Client
SDMX query
Test Client
SDMX response
Census Hub
Non-SDMX local database

16. Data Hub is: No data processing (no editing or aggregation)
System of DSDs built in SDMX 2.0 and in use for 32 countries of the ESS
Data dissemination portal based on SDMX data model
communicating with data providers via SDMX Web Service
No data processing (no editing or aggregation)
Additional reusable modules for
Configuration management
Tool for handling SDMX structural metadata
Innovative user interface -> extract data starting from a statistical concept
MSs status: 32 up and running

17. A data user can…
Browse the Hub to define a dataset of interest, navigating via structural metadata:
Search by topic (filters) and select data (level of detail, breakdowns)
Select layout (axes)
View a table
Export a file (CSV, Excel, SDMX-ML)
User registration
Registered users can:
Save, retrieve, modify or delete stored queries
Receive an notification when offline queries are executed 17

18. How the Hub works National Statistical Institute
Eurostat Census
Hub 18 18 18

19. https://ec.europa.eu/CensusHub2/19

20. Reusability Installed within and outside ESS Used for multiple domains
Used for dissemination and reporting
Supports data sharing, push and pull modes
Generic and SDMX based (2.0 and 2.1)
Extensible and modular approach
Free, open source solution, maintained by Eurostat
Support different platforms and DB vendors

21. For more information 21

22. Thank you for your attention22

23. DEMO

24. DEMO