1. Standardizing and industrializing a business process – the dissemination use case
Alessio Cardacino -
ESTP Course “Information standards and technologies for describing, exchanging and disseminating data and metadata”
THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

2. SDMX architectures and tools supporting data dissemination and reporting processes

3. SDMX database-driven Architecture

4. SDMX Hub Architecture National Statistical Institute
Eurostat Census
Hub
Census Hub Application

5. Eurostat SDMX Reference Architecture
Data user
Data Collectors
Test Client
Web Service
Provider
Web Client
Mapping
Assistant
Auth. DB
Prerequisite: data to be disseminated/reporting has to be stored in a database or PC- Axis files
Mapping Store – repository of SDMX structural metadata and mapping information.
Mapping Assistant - provides mapping information between SDMX artefact and the local data storage scheme.
Test Client - an application to test the mappings performed through the Mapping Assistant. It also acts as an “SDMX query” builder and client against the Web Service. It can be used successful in the “Push” mode.
Web Service - a service that accepts both data and structural metadata SDMX Query messages.
Web Client - a web application that allows users to create SDMX queries to be used for retrieving data from a Mapping Store database and dissemination databases using a SDMX-RI Web Service instance.
TestAuthConfig - application for configuring authentication and authorization information for the Web Service. It allows adding, removing and updating users and maintaining the user’s list of allowed dataflows
NSI
Mapping Store DB
TestAuthConfig
Dissemination RDBMS
Dissemination
PC-AXIS

6. SDMX Istat toolkit
A set of pick-and-choose building blocks allowing a statistical office to handle (macro)data and metadata (subject-matter domain independent)
It extends the SDMX-RI in order to streamline the dissemination and reporting business processes:
push and pull
“distributed” data warehouse SDMX-based
web presentation of data and metadata
It is built using the SDMX Common API (SdmxSource.NET)

7. Implementing the data dissemination process through the «SDMX Datamanager » web application

8. SDMX Modelling of the primary hypercube
Hypercube definition:
A single hypercube containing data on:
Industrial turnover index, Italy (Base 2010=100)
Industrial turnover index, Italy (Base 2015=100)

9. Industrial turnover index table

10. Creation of the primary data hypercube from the STS+IT1+1
Creation of the primary data hypercube from the STS+IT1+1.0 DSD through the «SDMX Datamanager» web application

11. Access to the SDMX Datamanager
Username: admin Password: admin

12. The starting window (Builder)
Click here to load
structural metadata
Here Cubes are shown
Here DSDs are shown

13. Load structure metadata: Category schemes
The first action that must be performed is to upload two or more SDMX Category schemes: 1 will be used to categorize the cubes, the others to categorize SDMX Data flows:
Click on the [Import SDMX Structure] button
Select the file “BL_DW+IT1+1.0.xml” (used for categorizing the cubes)
Select the file “DDB_DW+IT1+1.0.xml” (used for categorizing the Dataflows)
Button for Importing SDMX Structure Metadata (artefacts): CategorySchemes and DSDs
Button for adding SDMX-ML structure files containing artefacts to be imported
Button for uploading (importing) the SDMX artefacts indicated in the list
List of imported categoryschemes before the selection of the categoryscheme to be used, in the builder section, to categorize the cubes
(1)
(2)
(3)
(4)

14. Import of the DSD
In the «Manager Structure» interface of the «Builder» section, click on the «Import Structure» button.
In the pop-up window, click on the «Add File» button, select the STS+IT1+1.0, containing the DSD and the referenced artefacts, and click on the «Upload» button.

15. Selection of the categoryscheme for categorizing the cubes
When the first cube is created, after having selected the DSD to be used for defining its structure, is shown a popup window containing the list of the available categoryschemes, from which has to be selected that to be used for categorizing the cubes

16. Primary hypercube creation
In the «Manager Structure» interface of the «Builder» section:
select the STS+IT1+1.0 DSD (that has to be final)
Click on the «Select Category» button and in the popup window select the «Industry and Construction» category

17. Primary hypercube creation
3. Fill in the «Datacube unique code» textbox the INDTURN code
4. Click on the «Add Name» button, and in the popup windows fill in the italian and english name of the hypercube:
Industrial turnover index

18. Primary hypercube creation
Select the conditional Attributes to be included in the hypercube structure (Dimensions and mandatory Attributes are selected as a default) and click on the «Create Datacube»

19. Primary hypercube creation

20. Csv data upload on the primary hypercube through the «SDMX Datamanager» web application

21. Mapping of the csv data structure
The columns of the csv file must be mapped against the data structure of the hypercube in order to allow the data import.

22. Mapping of the csv data structure : steps
In the «Manager mapping» interface of the «Builder» section, click on the «New MappingSet» button.
Select the hypercube to be mapped:BL_INDTURN B

23. Mapping of the csv data structure : steps
3. Select the csv file to be mapped: IND_TURNOVER.csv

24. Mapping of the csv data structure : steps
4. Perform the mapping: 5. Save mapping

25. Load the csv data file
1. In the «Data import» interface of the «Loader» section, select the mapping previously created:
2. Select the file to be uploaded: IND_TURNOVER.csv

26. Load the csv data file
3. Upload the file by clicking on the button «Upload»
4. Preview the data to be imported

27. Load the csv data file 5. Click on the «Import button»
6. At the end of the «Import» task, the system returns a report indicating the state of the upload (successful or not) and the eventual issues with the indication of the affected records

28. View the imported data
7. View the imported records in the «Manager structure» interface of the Builder section, by clicking on the «Cube Details» button

29. Definition and publication of the Dataflows/views to be disseminated through the ««SDMX Datamanager» web application

30. Hypercubes to be disseminated
Starting from the primary hypercube («Industrial Turnover index»), have to be disseminated in SDMX format and via web service the following hypercubes, without performing any duplication of data in the definition of the hypercubes:
Industrial turnover Index, base 2010=100
Industrial turnover Index, base 2015=100

31. SDMX Dissemination though the ISTAT’s SDMX Datamanager and the SDMX-RI’s NsiWebService
SDMX Datamanager has been developed by ISTAT using standard SDMX libraries provided by Eurostat (SdmxSource.NET)
In order to provide the data through the SDMX web service are automatically generated by the Datamanager SDMX mappings of the local data structures (DDB hypercubes) against the DSDs referenced by the published dataflows that are derived by the DSD through which as been generated the primary cube

32. Definition of the Dataflows
Under the «Industry and Construction» category, have to be created the following dataflows:
Dataflow Id
Dataflow name (en)
INDTURN_2010
Industrial turnover Index, base 2010=100
INDTURN_2015
Industrial turnover Index, base 2015=100

33. Dataflow creation of INDTURN_2010
In the «Query builder» section, click on the «New Cube View» button:

34. Dataflow creation of INDTURN_2010
2. In the «Dataflow» tab, insert the following information:

35. Dataflow creation of INDTURN_2010
3. In the «Query» tab, select the BL_INDTURN hypercube
4. Select the following cube columns:

36. Dataflow creation of INDTURN_2010
5. Insert the following filter conditions:

37. Dataflow creation of INDTURN_2010
6. Query preview:

38. Dataflow creation of INDTURN_2010
7. Dataflow categorization:
8. Save Dataflow by clicking on the «Save View» button

39. Dataflow creation of INDTURN_2010
9. View in the list and Set in production:

40. Put in production Dataflow/Views
From the Dataflow/Views list, click on
Then insert eventually header information about sender and/or receiver
for the selected dataflow

41. Dataflow INDTURN_2010 in production

42. Dataflow creation of INDTURN_2015

43. Dataflow creation of INDTURN_2015
Selected cube columns:

44. Dataflow creation of INDTURN_2015
Filter conditions:

45. Dataflow creation of INDTURN_2015
Dataflow categorization:
Save Dataflow by clicking on the «Save View» button

46. Dataflow creation
List of created dataflows:

47. Browse the SDMX WEB Service content (the created dataflows) through the Eurostat’s NSI Client

48. Eurostat’s NSI Client data browsing
Selection of a dataflow (INDTURN_2015) and dimension filtering:

49. Eurostat’s NSI Client data browsing
Tabular view of selected data

50. Browse the SDMX Web Service content (the created dataflows) through the ISTAT’s SDMX Data Browser

51. ISTAT’s Data Browser
Selection of a dataflow (LOMB_16_FGIUR) and dimension filtering:

52. ISTAT’s Data Browser
Tabular view of selected data