Francesco Rizzo (ISTAT - Italy) SDMX ISTAT FRAMEWORK GENEVE May 2007 OECD SDMX Expert Group

Summary SDMX Istat Framework Geneva May 2007 OECD SDMX Expert Group Why Istat is starting to use SDMX SDMX Istat Framework – an overview SDMX data Web Service (demo) SDMX Web Navigator (demo)

Why Istat is starting to use SDMX SDMX Istat Framework Geneva May 2007 OECD SDMX Expert Group The Istat Information System is made up of several production Directorates sub-systems that, independently one from another, cover the full life cycle of statistical data, from collection to dissemination. Data Collection Data Editing Data Aggregation Production Directorates metadata web navigator thematic DB

validated microdata Geneva May 2007 Meeting on the Management of Statistical Information Systems Data Collection Data Editing Data Aggregation Production Directorate thematic DB Data Collection Data Editing Data Aggregation Production Directorate thematic DB metadata web navigator Integrated Output Management System web navigator Integrated Output Management System project

Why Istat is starting to use SDMX SDMX Istat Framework Geneva May 2007 OECD SDMX Expert Group For this reason Istat is following the evolution of the SDMX initiative with interest : Tactically: through the participation in the EUROSTAT SODI task force Strategically: through the building of a special working group whose main objective is to analyze and verify the use of SDMX in the internal architecture of ISTAT Information System

Why Istat is starting to use SDMX SDMX Istat Framework Geneva May 2007 OECD SDMX Expert Group The participation in the EUROSTAT SODI initiative has involved Istat to gain experience in the use of XML and Web Services technologies for integration purpose inside Statistical Information Systems. In order to facilitate this objective, we are developing several software modules organized in a framework (current version is 1.0)

SDMX Istat Framework version 1.0 SDMX Istat Framework Geneva May 2007 OECD SDMX Expert Group Characteristics: compatible to SODI needs useful to introduce SDMX for both internal purposes and external information exchange can be used entirely from the reporting phase to the dissemination phase, or alternatively using modules separately, integrating them into one information system in the future it might be possible to distribute the framework under Public Licence

time series database.dat Loader ConIstat modules.ist ConIstat web navigator Check fix formatted record file GENEVE May 2007 OECD SDMX Expert Group mapping SDMX web navigator ESA STS STS-ESA SODI RSS SDMX Queries new modules SDMX data web service RSS builder & query builder under construction modules Reference metadata manager Reference metadata database Reference metadata SDMX Query Reference metadata web navigator Reference metadata web service Reference metadata RSS Word extractor IMF Eurostat gesmes

SDMX Istat Framework version 1.0 SDMX Istat Framework Geneva May 2007 OECD SDMX Expert Group Overview of changes in the Loader module: New dissemination functions were added in order to satisfy some SODI requirements: to publish an RSS; to publish on or more SDMX Query file(s); to publish one or more SDMX Compact file(s) (optional)

Response-time test: an entire Dataflow containing 685 time series, 120 months long (developing observation) < 20 seconds demo SDMX Istat Framework version 1.0 SDMX Istat Framework Geneva May 2007 OECD SDMX Expert Group Overview of the SDMX Data Web Service module: SDMX Compact builder XML validate query Parser sql query builder Database SDMX Query SDMX Compact

SDMX Istat Framework version 1.0 SDMX Istat Framework Geneva May 2007 OECD SDMX Expert Group Overview of SDMX Web Navigator module: This module acts as a client towards the web service and represents a web graphic interface that allows the following functions: building a SDMX Query using DSDs as analysis dimensions sending the SDMX Query to the web service for querying the database Testing the SDMX Queries that a user can build starting from SDMX Queries created with the graphic interface demo

SDMX Istat Framework version 1.0 SDMX Istat Framework Geneva May 2007 OECD SDMX Expert Group Overview of Manager and Navigator SDMX reference metadata module: Production Directorates Database Reference metadata manager Word extractor Eurostat IMF SDMX Reference metadata web service XML Reference metadata web navigator XSLT HTML

SDMX Istat Framework version 2.0 SDMX Istat Framework Geneva May 2007 OECD SDMX Expert Group A generalized SDMX Query parse module (beta version): This module accepts only queries with DataWhere section but also accept queries regarding: different Dataflows subsets of different DataFlows A SDMX Query is processed by the following steps: rewrites the XML stream using Boolean logic rules, and adding to each node an attribute that acts as a “unique key”. The scope of the unique key is to set a hierarchy between “parent nodes” and “child nodes”; converts the XML stream in a memory tabular data structure whose columns represent all types of nodes (Time, Dimension, Attribute, Dataflow) except for And and Or; re-organizes the in-memory tabular data structure so that all columns have an And relation, and all rows have an Or relation; converts the in-memory tabular structure in a SQL query