1. SDMX and DDI Working Together Technical Workshop 5-7 June 2013
Marco Pellegrino, Denis Grofils
Eurostat

2. Outline Where are we (background)?
Where are we going (plans and projects)?
DDI-SDMX dialogue
Agenda items

3. Where are we?
Dramatic changes in the environment of official statistics producers (e.g. data deluge)
Modernization of statistical information system seen as a question of survival for the sector of official statistics
Standardization viewed as a key enabler for modernization
 "Standards-based” industrialization of statistical production

4. Standardization in the ESS
The ESS has a long tradition in harmonising statistical products and regulating requirements within the different statistical domains
SDMX: a success story
Standardization allows cross-domain synergies by enabling sharing of data & software
ESS Vision
Increase the level of integration of the ESS (vertical & horizontal)
Maximize sharing & re-use
Standardisation in early stages of production: TO DO!

5. ESS Architecture, current situation
ESTAT

6. ESTAT

7. ESS VIPs and Cross-cutting projects
CC – "Technical"
CC – "Programme"
ADMIN
Information models
Communication
NAPS
Communication Network
Governance
ESS DW PRIX and TRIS
Data warehouse
Human resources
EsBRS
Shared services
Financial resources
SIMSTAT
Legal framework
ICT
Programme office
Common validation policy

8. ESS.VIP business and information principles
Maximum reuse of existing process components and segments
Metadata driven processes allowing adaptation/parameterisation and extension to other contexts
New business process built as a sequence of modular process steps / services
Information objects structured according to available information models and stored in corporate registries/repositories in view of reuse
Adherence to industry and open standards as available

9. Two important projects (1/2)
ESS.VIP Cross-cutting Project on Information Models and Standards (IMS)
To ensure that the European Statistical System (ESS) has access to a set of agreed-upon standards supporting the modernisation of statistical services
To increase coherence between standards, at the same time ensuring that these are consistent with best practices and recommendations from the international community of official statistics.
To define information models that can be used across the ESS to model structural metadata for different types of data, taking into account existing standards and on-going developments
To provide support mechanisms (e.g., capacity-building and training) for the practical implementation of these standards and models
IMPORTANT SYNERGY BETWEEN THE TWO

10. Two important projects (2/2)
UNECE Project Frameworks and Standards for Statistical Modernisation (FSFSM)
To ensure that the international statistical community has access to the standards needed to support the modernisation of statistical production and services
To increase coherence between these standards
To provide support mechanisms for the practical implementation of these standards within national and international statistical organisations
To ensure effective promotion and maintenance of the GSBPM and the GSIM, including the release of new versions as appropriate

11. SDMX-DDI dialogue
Launched in 2010 with 3 goals:
To avoid duplication of efforts and thus avoid confusion about which standards should be used for specific types of applications
To provide reassurance to the user communities of DDI and SDMX that the end-to-end statistical process can be managed, and that standards bodies are both considering the needs of users in this area
To provide specific technical guidance about the use cases and implementation of the standards for specific purposes
 Endorsed by DDI Alliance and SDMX Sponsors / Secretariat (mandate TWG)

12. SDMX & DDI SDMX: Statistical Data and Metadata eXchange
Standard for the exchange of statistical data and metadata
“the preferred standard for exchange and sharing of data and metadata in the global statistical community” UN Statistical Commission 2008 – Widely used in the ESS
Extended to support unit-level data
DDI: Data Documentation Initiative
Standard for the documentation of data
Initially focused on archiving micro-data in the area of social sciences – Widely used in national data archives
Extended to support the full life-cycle of data

13. Generic Statistical Business Process Model
DDI
DDI
SDMX

14. GSBPM, DDI and SDMX: towards a complete system?

15. GSBPM, DDI and SDMX: towards a complete picture?

16. Characterizing the Standards: DDI
DDI Lifecycle can provide a very detailed set of metadata, covering:
The study or series of studies
Many aspects of data collection, including surveys and processing of microdata
The structure of data files, including hierarchical files and those with complex relationships
The lifecycle events and archiving of data files and their metadata
The tabulation and processing of data into tables (Ncubes)
Allows for a link between the microdata variables and the resulting aggregates

17. Characterizing the Standards: SDMX
Describes the structure of aggregate/dimensional data (“structural metadata”)
Provides formats for the dimensional data
Provides a model of data reporting and dissemination
Provides a way of describing and formatting stand-alone metadata sets (“reference metadata”)
Provides standard registry interfaces, providing a catalogue of resources
Provides guidelines for deploying standard web services for SDMX resources
Provides a way of describing statistical processes

18. SDMX Process Metadata Data validation and editing, SDMX Registry,
DSD, data set,
MSD, metadata set,
Web services
Process Metadata
DDI has much more detailed metadata at the level of the study, because it is intended to describe the full process of data production (the data lifecycle)
DDI provides more complete descriptions of the processing of data
SDMX provides more architectural components, to support reporting/collecting and exchange
SDMX provides generic mechanisms to support foreseen and non-foreseen use cases (categorisation, HCL, MSD)
Similarities:
Both standards use a similar mechanism for structuring URN identifiers
Both standards use a similar model for identifiable, versionable, and maintainable things
Both have a concept of an owning agency
There is a very similar set of rules about versioning and maintenance
Both standards use “schemes” as packages for lists of like items
Both standards are designed to support reuse, and have similar referencing models

19. DDI and SDMX
DDI offers a very rich model for the documentation of micro-data
SDMX offers a very integrated exchange platform for statistical outputs (IT architectures, tools, web services)
When people think about using SDMX and DDI together, they make assumptions
Microdata (and tabulations) can be described using DDI
A transformation could be applied to produce SDMX to describe the aggregates/tables
There is a straight mapping from DDI to SDMX
Interestingly, this conceptual model is not how the use of DDI and SDMX together is being approached in reality
The Devil is in the details!
The combined use of both standards could allow a higher level of integration of the complete production process
But: The devil is in the detail!

20. Analysis of use cases
Set of relevant use cases where the two standards could be compared:
Survey data collection
Administrative and register data
Combined use of DDI and SDMX
Micro-data access and on-demand tabulation of micro-data
Metadata and quality reporting

21. The challenge
It's not about which flavor of XML we use (XML doesn’t really matter)
It’s about data and metadata!
If I want to use DDI to describe my data, and you want to use SDMX, how can we ensure that we are getting the same data and metadata?
It's about the convergence of information models and the availability of an integrated IT environment

22. Combined DDI-SDMX approaches
Mixing the two standards within an implementation, allowing for the expression of the same metadata set in both standards, so that the information could be transformed from one format to the other.
Metadata stored and indexed in such a fashion that it can be expressed either as SDMX or DDI on an as-needed basis.
Metadata Repository and Registry project at ABS.
The actual format used for metadata storage may be neither SDMX nor DDI, so long as it can be expressed using both standards.
GSIM to be implemented through a combination of SDMX and DDI?

23. Generic Statistical Information Model (GSIM)
SDMX
DDI
ISO 11179
Etc.

24. GSIM Conceptual model Implementation standards 2424 DDI SDMX
Other relevant standards
Geospatial standards

26. Agenda Introduction Business needs for DDI and SDMX
Technical overview of DDI and SDMX
Conceptual mapping between GSIM, DDI and SDMX
Use cases Statistical registers and administrative data Survey data management (combined use of DDI and SDMX) DDI and SDMX as tools for Metadata and Quality Reporting
Discussion: How to move forward
Conclusion