1. Census Hub: Progress report
Item 3.3.d of the agenda
Census Hub: Progress report
Marco PELLEGRINO, Nadezhda VLAHOVA
Eurostat, Unit B5
19-20 October 2010
IT Directors’ Group Meeting

2. Starting point Harmonised concepts and definitions
Data that are methodologically comparable and structured according to agreed “hypercubes” (Census Regulation)
Organisation into one common logical model of massive amounts of data produced and controlled by Member States
Dissemination of 2011 population and housing census data in EU
Accessibility to data and metadata to cross-tabulate from different sources
For the first time, the European Union will have a legislation aiming at the availability of harmonised high-quality data from the population and housing censuses conducted in the EU Member States in 2011.
It is a common ambition of the ESS to disseminate the results in a way that provides the users with an easy access to detailed census data that are methodologically comparable between the EU Member States and structured in the same way.
19-20 October 2010
IT Directors’ Group Meeting

3. Where are we? January 2008: start of the pilot project
March-April 2008: specifications, functional and technical analysis, test hypercube and related breakdowns
June - December 2008: pilot modules (Eurostat and NSIs), tests, evaluation reports, more NSIs invited to take part
2009: IT contacts appointed, technical missions
: Enhancement of application, design of all DSDs, IT working group in June 2010 (second meeting to come), technical missions continued
January 2008: start of the pilot project with four countries (Germany, Ireland, Italy and Portugal).
March-April 2008: specifications, functional and technical analysis, test data hypercube and related breakdowns.
June - October 2008: pilot application modules (Eurostat and NSIs), tests, evaluation report; analysis for the full Census Hub.
Census WG, September 2008: more NSIs invited to take part.
2009: IT contacts appointed and informed by Eurostat on the project developments. Technical workshops and missions.
: Development and enhancement of the application, design of all the data structures and codelists.
19-20 October 2010
IT Directors’ Group Meeting

4. The Hub: Innovative transmission and dissemination
Based on data sharing, where a group of partners agree on providing access to their data according to a standard process and common logical organisation
The SDMX Hub approach offers several advantages:
decoupling of NSI systems from the central hub via standard techniques for the exchange
NSIs in full control of the data and access privileges
Relatively limited start-up effort
Re-usability of the IT infrastructure
New system to achieve the dissemination of the 2011 Census data, based on the concept of data sharing, where a group of partners agree on providing access to their data according to standard processes formats and technologies.
19-20 October 2010
IT Directors’ Group Meeting

5. The Census Hub architecture
Hub with pull transmission and data sharing
SDMX Reference Infrastructure
The census hub architecture combines 3 major approaches – data sharing and pull mode of data retrieving; Hub and SDMX infrastructure
To achieve this Census Hub follows the SDMX standards. SDMX provides guidelines and tools to support the "pull" mode of data sharing, where the collecting organisation retrieves the data from the providers' websites.
Data are stored in the NSIs data warehouse and are fetched when a data user requires them.
To ensure this transmission SDMX infrastructure is used. It supports both the Hub and the pull approach through the use of SDMX Query[1] messages and Web Services[2] technologies
The data are made available to any organisation requiring them, in formats which ensure that the data are described by appropriate metadata, whose meaning is common to all parties in the exchange.
From the data management point of view, the hub is based on agreed hypercubes, but here the hypercubes are not sent to the central system. Data are stored in the NSIs data warehouse and are fetched when a data user requires them.
For this purpose, SDMX standards will be used. SDMX details an architecture that supports both the Hub and the pull approach through the use of SDMX Query[1] messages and Web Services[2] technologies
SDMX provides guidelines and tools to support the "pull" mode of data sharing, where the collecting organisation retrieves the data from the providers' websites.
The data may be made available for download in a SDMX-conformant file, or they may be retrieved from a database in response to an SDMX-conformant query.
In both cases, the data are made available to any organisation requiring them, in formats which ensure that the data are consistently described by appropriate metadata, whose meaning is common to all parties in the exchange. In this case as well one can develop own tools or integrate ready made software is considered advantageous.
19-20 October 2010
IT Directors’ Group Meeting

6. The SDMX Hub Pilot project architecture
NSI
SDMX infrastructure
NSI
Census
Data warehouse
Eurostat
SQL
Builder
SDMX
Query Parser
Web
Service
Data
base
Hub
SDMX
Generator
Use of SDMX standards
Hub approach
The Web service receives a SDMX query, handles errors, validates the query, then forwards the SDMX query to the SDMX query parser
The SDMX query parser breaks down the query and sends it to the SQL query builder
The SQL Query Builder creates one or more SQL queries and sends them to Database
The SDMX Generator receives a resultSet from the Database, assembles it in an SDMX-ML data message and forwards it to the web service
The Web service responds with an SDMX-ML message
To develop the central applications hosted in Eurostat
Hub
Tool to allow NSIs to update the LAU code list directly
To support the implementation of the national SDMX IT infrastructures
Advice on how to implement the SDMX infrastructure
SDMX training and technical workshops
Technical IT Working Group
To facilitate software sharing between countries
Developed and made available for free download a NSI SDMX infrastructure by Eurostat
Through the ESSnet on SDMX
19-20 October 2010
IT Directors’ Group Meeting

7. Where are we – development?
Approval of Census Hub legislation
DSD for one hypercube already integrated into the Hub
DSDs for all hypercubes created →this allows test with real hypercube structures
Tool for management of geo-codes (LAU) finishing development
Multilingual support
Legislation on quality indicators under analysis
Installation of IT infrastructure in more countries
19-20 October 2010
IT Directors’ Group Meeting

8. Where are we - ESS?
Countries integrated within the Hub (8): Czech Republic, Germany, Ireland, Italy, Luxembourg, Poland, Portugal, Spain
Expressing their interest or working on it, but not yet in the Hub (13):  Belgium, Bulgaria, Cyprus, Estonia, France, Greece, Hungary, Lithuania, Malta, Norway, Slovenia, Switzerland, United Kingdom
Asking for more information (10): Austria, Denmark, Finland, Iceland, Latvia, Liechtenstein, Netherlands, Romania, Slovakia, Sweden
1-2 June 2010: first Census IT technical WG
30 Sep 2010 – 1 October: SDMX IT technical workshop (Prague)
SDMX training, technical meetings
Second Census IT technical WG: to be convened
19-20 October 2010
IT Directors’ Group Meeting

9. The SDMX Census Hub National Statistical Institute
Eurostat Census Hub
National Statistical Institute
The workflow is:
Step 1: a “data user” browses the Hub to define a dataset of interest via structural metadata. He browses the dimensions and selects a dataset. Then he chooses the organization of the output layout specifying which dimension wll match X-axis and Y-axis and which dimension will vary item after item to generate new tables
Step 2: The Hub converts the user request into an SDMX Query and sends the SDMX Query to an interested NSI Web Service
Step 4: The NSI Web Service converts the SDMX Query in a set of SQL queries and sends them to the NSI data warehouse
Step 5: The NSI data warehouse sends the result to the NSI web service
Step 6: The NSI Web Service converts the result in a SDMX-ML Data message and sends it to the Hub
Step 7: The same steps are repeated if the user has requested data from different MSs
Step 8: the Hub puts together all the SDMX-ML data messages proceeding from the interested NSIs and presents the result to the “data user” in the web browser in readable format.
The pilot hypercube is very simple one in order to let NSIs produce it in a short period (Sex, Age, Currently Activity Status, Geo)
Data are not expected to be exact, just simulated to ensure a proof of concept, so a dummy data file is provided
NSIs’ experiences: several NSIs’ contact are available to communicate their experiences
ESSnet on SDMX: in practice the SDMX Istat Framework
A technical bilateral meeting could be arranged in the NSI,s premises with both (IT staff and statisticians)
SDMX NSI Reference Architecture by Eurostat: can be used per Building Blocks or as a whole through the installation wizard 9 9