1. Vision Infrastructure Project (VIP) Enhanced Dissemination Chain
2nd SISAI meeting, June 2012
B4 – IT for statistical production
Unit B5 – Management of statistical data and metadata
Unit B6 – Dissemination

2. VIP 'Enhanced Dissemination Chain'
Vision Infrastructure Projects
aim to improve the vertical and horizontal integration of the European Statistical System; this ESS integration is one of the central goals of the ESS vision.
Commission Communication COM(2009)404 on the production method of EU statistics: a vision for the next decade
As new technologies become available, there is a clear drive to maximise their use and to gear statistical methods toward them. New forms of communicating with users and producers such as web 2.0, as well as other new developments in information technology, are likely to result in profound changes in communication channels and
data handling and storage.
VIPs deal with improvements which are the precondition for achieving more vertical and horizontal integration of the European Statistical System:
- horizontal integration: depart from the approach where statistics are handled domain by domain
- vertical integration: joint structures / tools / processes + collaborative networks within the ESS
The improvements enable more integrated dissemination of European data and indicators.
VIPS fall under the 2009 Commission Communication on the production method of EU statistics.
The Communication offers a vision for reforming the production method of European statistics, departing from the classical stovepipe model.
Particular attention goes to the usage of the internet and web based technology in a broad sense, as these are nowadays the main vehicle for statistical dissemination.
Technology allows achieving dissemination targets such as full access to data, to all users at the same time & under same conditions, free of charge.

3. VIP 'Enhanced Dissemination Chain'
Enhanced Dissemination Chain – 2011 to 2013
Short term
Eurostat’s Dissemination Chain: improvement, simplification & better integration
Disseminated data: quality improvements
EU policy indicators: faster production
Involvement of ESS (Member States) in the Dissemination Chain
Medium term
Further development of Eurobase towards a more integrated data warehouse for Eurostat and the ESS
Better integration of Eurostat & national dissemination
A 2 years project, October 2011 to October 2013
Short term = within project timeline
1. Simplification and better integration, diminish the risk of disruption of
a) the daily data updates and
b) the online data access,
PLUS
increase the performance of dissemination
2. Data quality of the European data which is disseminated, in particular through the use of ex-post checks on all published data (Eurobase functionalities)
3. Producing and disseminating policy-oriented indicators includes the development of
(a) a new production line for indicator tables, to facilitate flexible combination of data from several domains and
(b) new ways of presenting indicators and dashboards on the Eurostat website,
would be used for the EU2020 indicators, the Sustainable Development Indicators, ...
4. Share software components & tools within the ESS
Medium Term – definition of medium term
1. Data warehouse
Overall context: enterprise architecture; possible strategy.
2. Better integration within the dissemination of Member States.
This covers further work to facilitate NSIs showing European data and metadata on their websites.

4. Eurostat Dissemination Chain Simplified view of the data flows
Comext
Comext
Dissemi-
nation
Internet tools:
Data Explorer,
TGM, etc.
New
Query
Generator
Eurostat
production
databases
Eurobase
This is an extremely simplified representation of Eurostat's dissemination chain.
In the current context, the Eurostat dissemination chain is a complex patchwork of software components with scattered responsibilities.
Dissemination Chain covers:
the reference database Eurobase,
the reference database Comext (external trade, industrial production statistics)
Comext dissemination and
the Eurostat website, including its data access and visualisation tools.
Eurobase - main reference database containing data for about 5000 non Comext datasets
Comext - reference database for external trade and industrial production data ( datasets)
Comext dissemination - database containing data from both Eurobase ( datasets) and Comext (180 datasets)
New Query Generator (NQG) - the tool used to customise a dataset’s default view
Data Explorer - the main application used to access datasets (data browser)
TGM – Table, Graphs & Maps
22/03/2012

5. VIP 'Enhanced Dissemination Chain'
Actions
Deliverables 1
Development towards a Eurostat / ESS
data warehouse
Strategy
(context: enterprise architecture)
Checking algorithms in Eurobase
Simplification and integration of the Eurostat Dissemination Chain
More functionality for producing and disseminating Eurostat policy indicators 2 3
Service Impact Manager
Possible direct connection between
Data Explorer & Eurobase
New production line for
indicator tables
Flexible dashboards for
policy indicators
What will happen in the 2011 to 2013 period? 4 actions
Actions with MS vs. pure Eurostat actions
Details in the following 4
More integration of Eurostat data and metadata in NSI dissemination
Embeddable visualisation tool
SDMX-compliant web services

6. VIP 'Enhanced Dissemination Chain'
Action 1: Development of Eurobase towards an Eurostat / ESS data warehouse
Task 1a: Strategy on opening Eurobase towards an ESS warehouse
Overall context enterprise architecture
Internal discussions, early stage
Task 1b: Ex-post data quality controls for data published in Eurobase
Input into Eurobase is validated by production units
But: errors are possible
Negative values, unexpected jumps, outliers, obvious inconsistencies …
Task 1a:
Series of internal discussion in the overall context of enterprise architecture
MS / ESS to be kept informed on this process
Task 1b:
Work to ensure and improve quality within the reference database Eurobase
Existing input checks – but there are limits what can be checked at the input
Identify obvious errors – 'real' validation is done in production units, this action just to control if something went wrong
Related work to improve quality: presentation of flags, harmonisation of table titles, code lists (outside VIP)

7. VIP 'Enhanced Dissemination Chain'
Action 2: Simplification and better integration of the DISS-CHAIN
Task 2a: Service Impact Manager (SIM)
Task 2a:
The SIM application aims at:
- The centralization of all the monitoring information of each Dissemination Chain component (retrieving log files, monitoring statistics, backlog entries or audit tables from databases for all different environments),
- The processing of the information collected based on defined rules created from the in-depth knowledge of current support team(s),
- Acting as a point of reference regarding follow up of
1) Impact assessment on Eurostat business processes, by means of easy to use dashboards,
2) Eurostat dissemination chain related incident alerts,
3) Open vs. resolved Eurostat dissemination chain related incidents,
4) Eurostat dissemination chain related knowledge base.
These are the current targets and proposals, and may be subject to updates or scope changes

8. VIP 'Enhanced Dissemination Chain'
Action 2: Simplification and better integration of the DISS-CHAIN
Task 2b: Feasibility of the direct connection between Data Explorer and Eurobase
Short-term: as-is analysis, and proposals for improvements of the current DISS-CHAIN
Long-term: development of a new system
Task 2b:
This is to be considered as an analysis of the as-is situation of the dissemination chain, along with proposals for evolution. The implementation of proposals may stretch beyond the scope and timeframe of the VIP-EDC. 8 8

9. VIP 'Enhanced Dissemination Chain'
Action 3: Improve functionality for producing and disseminating policy-oriented indicators
Task 3a: New production line for indicator tables
Faster and more flexible provision of indicators
Task 3b: Flexible dashboards for policy indicators
Existing dashboards: PEEIs, HICP
Specific solutions
'Reference' dashboard
One technical solution ready to be used by different indicator sets
Feasibility study on-going
Task 3a:
Quite frequent requests to make new policy indicators available online very quickly.
Current production process relatively slow.
Use functionalities available in Eurobase: direct preparation of tables using calculation and other functions.
Technical issues and business process and governance issues.
Task 3b:
Eurostat currently has a number of tools to support the visualisation of statistical data, yet for different sets of functionalities and / or based on different technologies. Simultaneously, other institutions – including other Commission DGs – start also to provide dashboards on their websites. The capacity to deliver dashboards for different domains with reduced time for delivery and software maintenance overhead is becoming a key consideration for Eurostat.
Dashboards will:
- Be available on Eurostat's portal,
- Display data in elaborated ways, possibly relying simultaneously on multiple indicators. Also, dashboards will support advanced display features, e.g. for multiple countries comparison,
- Support user interactivity for manipulating and presenting the data,
Etc.

10. VIP 'Enhanced Dissemination Chain'
Action 4: Enhance Eurostat dissemination towards better integration with MS's dissemination
Task 4a: Simple visualisation tool to be embedded in NSIs' websites
Widgets
Development together with MS (Task Force)
Examples (priority widgets)
Task 4a:
Widgets are
Small web applications
To be embedded in any website
“To be embedded in any website” = in the beginning, only NSI and Commission services, later this could be a much broader audience, and this could also take the form of an iOS app e.g.
Showing a relatively small amount of Eurostat data
“Showing a relatively small amount of Eurostat data” = this is about presenting key facts
In a simple and understandable way
Easy to configure
a) Which data
b) Which display
Focus on 18 indicators, to start with.

11. VIP 'Enhanced Dissemination Chain'
Action 4: Enhance Eurostat dissemination towards better integration with MS's dissemination
Task Force meeting 29 & 30 March with 6 NSIs agreed on 2 top priority widgets
Example of 2 widgets as they would be shown inside a Eurostat Portal page, in a side column.
Layout can be adapted to fit NSI's websites.
Now work on architecture of the widgets. 11 11

12. VIP 'Enhanced Dissemination Chain'
Action 4: Enhance Eurostat dissemination towards better integration with MS's dissemination
Task 4b: Dissemination web services
Additional dissemination product
Technology to exchange data directly over the internet (machine to machine)
Beta-release available
Beta release:
- The product is available to anyone wanting to use it, as is and with no warranty that the currently supported service operations will be continued in the future.
- A number of improvements are needed.
The next version will introduce several in-depth changes. Analysis on-going until 2012Q4, implementation and delivery by 2013Q2. 12 12

13. VIP 'Enhanced Dissemination Chain'
Conclusions – expected project results
Eurostat: improvements of the DISS-CHAIN
Technical
Architecture
Functionalities
Data
Quality
Presentation
ESS: integration with MS's dissemination
Widgets
Dissemination web services
Project October 2011 – October 2013 13 13