1. 7b. SDMX practical use case: Census Hub
Nadezhda Vlahova Eurostat Unit B3: “IT for statistical production”
SDMX Basics course, October 2015 1

2. The European Census Hub: key issues
Dissemination of the data from the 2011 population and housing censuses in the European Union
Data that are methodologically comparable and structured according to “hypercubes” agreed with Member States (Census Regulation)
Providing users with an easy access to detailed census data (advanced functionalities)
Management of massive amounts of data produced and controlled by Member States
High accessibility to data and metadata
Harmonised concepts and definitions
Maximum flexibility to cross-tabulate data from different sources
Easy to use 2

3. EU Census: Implementing measures
Regulation (EC) 763/2008 on population and housing censuses authorises the European Commission to adopt implementing measures on:
technical specifications of the topics and their breakdown (Regulation (EC) 1201/2009)
programme of the statistical data and metadata to be transmitted to Eurostat (Regulation (EU) 519/2010)
quality reporting and technical format of data transmission (Regulation (EU) 1151/2010)

4. What we need?
Environment for dissemination of massive amounts of harmonised data which is easy to use and reuse
Possibility to compare and cross-tabulate topics of interest, on data collected from different sources
Countries Census data and metadata should be made available by 31 March 2014 and stored until 2025
Single point to access and retrieve detailed census data, numerical and textual metadata
Standardized approach for similar data collections
No validation and aggregation on the fly are required, neither supported

5. The Hub concept
The Hub is based on the concept of data sharing, where a group of partners agree on providing access to their data according to standard processes, formats and technologies.
The SDMX Hub approach offers several advantages:
decoupling of NSIs' systems from the central hub via standard formats and techniques for the exchange;
Limited investment, re-usability (with the advantage of using recognized international standards).
Software (SDMX-RI) is supplied by Eurostat. 5

6. Hub approach – PULL method for data collection and dissemination
register
SDMX Registry
query
NSI P U L
Hub
Dissemination
Received
data in
SDMX-ML
Eurostat Pull Requestor
Loader
Eurobase
Dissemination
Verification /
Conversion
To SDMX
eDAMIS P U S H
XSL for
SDMX-ML
Warehouse
storage
Intermediate
storage
Data Input

7. What is the Census Hub ?
Interoperability/Initiative (32 countries participating in the project)
Business driven project
Delivered:
Information system to query and display SDMX formatted data (e.g. Census 2011 data) retrieved from different data providers – Census Hub
Universal framework for exposure and translation in SDMX-ML format of data stored in a legacy dissemination database – SDMX-Reference Infrastructure (SDMX-RI)

8. Prerequisite: SDMX compliance* (1/2)
Agreed standard/format for data exchange
Defined definitions, concepts, codelists, DSDs, transmission model and obligations
Requires:
NSIs to create and transmit data according to:
agreed format
mode of data exchange
defined time period
* Prerequisite for SDMX tools and SDMX exchange

9. SDMX compliance (2/2) Data preparation
Macrodata database
Statistical variables (topics)
Microdata database
Aggregated dataset n
Individual records
Non-SDMX local dissemination data
Aggregated dataset 5
Aggregated dataset 1
Aggregated dataset 2
Aggregated dataset 4
Aggregated dataset 3

10. Example: DSD for Table 6 (Marital Status)
Dimensions
Attributes ID
CONCEPT
CODELIST
TIME
Time period or range
CL_TIME
GEO
Geographical area
CL_GEO
SEX
Sex
CL_SEX
FST
Family status
CL_FST
LMS
Legal marital status
CL_LMS
CAS
Current activity status
CL_CAS
POB
Country/place of birth
CL_POB
COC
Country of citizenship
CL_COC
AGE
Age
CL_AGE
FREQ
Frequency
CL_FREQ ID
ATTACHMENT LEVEL
CODELIST
OBS_STATUS
Observation
CL_OBS_STATUS
OBS_LEVEL
CL_OBS_LEVEL
OBS_NOTE
HC_NOTE
Series
Measures ID
NAME
OBS_VALUE
Observation value

11. Census datasets (hypercubes)
Hypercube HC06
GEO.L.
SEX.
FST.H.
LMS.
CAS.L.
POB.M.
COC.M.
AGE.M.
(57)
(3)
(17)
(13)
(6)
(15)
(28)
Theoretical number of cells
= 57 * 3 * 17 * 13 * 6 * 15 * 13 * 28 = …
= 1,238,033,160 cells
(NB: for one country and one table…)

12. Census datasets (hypercubes)
Hypercube HC04
GEO.L.
SEX.
HST.H.
LOC.
CAS.L.
POB.L.
COC.L.
AGE.M.
Using the naming conventions of Eurobase its name would be:
Population in NUTS2 regions by sex, five-years age groups, household status, size of the locality and broad groups of current activity status, place of birth and country of citizenship
 we need to change the way in which we give access to data

13. SDMX exchange and supporting tools
Requires: aggregated data and SDMX DSDs
SDMX exchange and supporting tools
Process workflow
NSI
SDMX codes
Extract files
Transform file
SDMX file
Dissemination/Transmission
SDMX Converter
NSI software
NSI software
SDMX Converter
Processing for sending
EDAMIS
SDMX-RI
Non-SDMX local data
Processing for sending
Test client NSI client
EDAMIS
Mapping Assistant
NSI Web service
HUB
NSI development
EDAMIS
NSI software
Eurostat tools
NSI developed software

14. SDMX-RI architecture overview
SDMX-RI – User Interfaces
Data providing organisation
Data collecting organisation
Web/Test Client
Mapping
Assistant
Internal network
SDMX-RI – “Under the hood”
Non-SDMX local data
SDMX-formatted data 14 14

15. Census Hub - Standardisation of the approach
Data Provider = NSI
Data Collector
SDMX
(STANDARD, EXCHANGE, METADATA, REPOSITORY)
WEB SERVICES
DSD
DSD
Mapping
Assistant
Metadata repository
Mapping store
DSD
Web Client
SDMX query
Test Client
SDMX response
Census Hub
Non-SDMX local database

16. Based on Census Regulation - Data Hub is:
System of DSDs built on SDMX 2.0 (standard concepts and codes) and in use for 32 countries of the European Statistical System
Data dissemination portal based on SDMX data model
communicating with data providers via SDMX Web Service
No data processing (no editing or aggregation)
Additional reusable modules for
LAU / NUTS management
Tool for handling SDMX structural metadata
Innovative user interface allowing user to extract data by starting from the statistical concept
MSs status: 32 up and running

17. Through the SDMX Hub, a data user can…
Browse the Hub to define a dataset of interest, navigating via structural metadata: - Search by topic (filters) and select data (level of detail, breakdowns) - Select layout (axes)
View a table
Export a file (CSV, Excel, SDMX-ML)
User can register by:
- Providing a valid address
- The system is generating random password
- User is notified via for his password
Registered users can:
- Save, retrieve, modify or delete stored queries
- Receive an notification when offline queries are executed 17

18. How the Hub works National Statistical Institute
Eurostat Census
Hub 18 18 18

19. https://ec.europa.eu/CensusHub2/19

20. For more information 20

21. Thank you for your attention21

22. DEMO

23. DEMO