3 June 2013 SDMX Technical Working Group Luxembourg 1 WP Item 6 Expressions and Calculations

3 June 2013 SDMX Technical Working Group Luxembourg 2 Background The SDMX 2.0 and 2.1 versions already envisaged the introduction of a standard language for validations and calculations The SDMX 2.1 package n. 13 (Transformations and Expressions) is a generic model aimed to track the validation and the calculation of data, derived from the CWM (Common Warehouse Metamodel), a OMG standard (Object Management Group) However this model is not operational in-itself, because it requires a language to specify the validation and calculation expressions

3 June 2013 SDMX Technical Working Group Luxembourg 3 Some outcomes of the TWG WP5 A language for validations and calculations is needed to support the statistical processes Other standards already have a calculation language (e.g. XBRL Formula) Both SDMX and other standards like DDI and GSIM, which don’t have it, are interested in adopting a calculation language Some institutions already adopted a “reference” language, (e.g. Eurostat, Unesco, Banca d’Italia …), in some cases for internal purposes and sometimes also to communicate with their reporting entities Other institutions are willing to adopt a standard language (e.g. FAO …)

3 June 2013 SDMX Technical Working Group Luxembourg 4 The assignment At the end of 2012 the SDMX Secretariat launched the WP Item 6 (Expressions and Calculations), aimed to introduce in the standard a language for validations and calculations, so making the SDMX 2.1 package n. 13 (Transformations and Expressions) fully operational The Secretariat asked to the TWG of elaborating a note, to be shared with the SWG, for describing the work and defining an implementation plan The Secretariat also asked to raise the priority of this item as possible

Paris Meeting Dec The TWG discussed this topic in Dec. 2012: The basic goals of the WP Item were identified The aim and the general model of the SDMX Transformations and Expressions were assessed Three existing business cases were presented –Bank of Poland use of XBRL Formula –Eurostat VIP on Validation project –Banca d’Italia EXL Some requisites of the language were discussed The interested TWG members were identified (the relevant documentation is in the EBIS site, folders: TWG Documents/Deliverables/Expressions and Calculations TWG Documents/TWG Meetings/ )

Some general requisites User orientation Integrated Approach IT implementation independence Active Role for processing Extensibility and customizability Language Effectiveness

The Note for the Secretatiat A first version of the Note was draft according to the outcome of the Paris meeting and was circulated in the TWG and in the SWG It emerged the need of clear business cases, which were discussed in a joint TWG-SWG teleconference; It was decided to: –Add the description of the business cases that the language should support, which is mainly the definition and the exchange of validation rules; –Clarify that an “agnostic” solution should be pursued, to be potentially employed in different standards (e.g. SDMX, DDI, GSIM); –Form a joint TWG-SWG subgoup –Prepare a realistic plan for the implementation of the language, according to the SWG and TWG resources available to dedicate a significant amount of time to work concretely on this topic

The business cases Most of the institutions need a standard language for internal validation Many institutions need a standard language also to share validation rules with reporters Some organizations also share with reporters common data production rules, to guarantee data quality and consistency and lower the reporting burden Some organization uses a reference language also for internal compilation, for example: –For achieving an automated compilation (industrial process) –For enforcing an harmonized compilation made by different people (e.g. banks’ supervision analysis) –For preserving the know-how of the compilation agorithms (if employees change, if tools change)

The priority For most organizations the Validation is the priority: In the first step, aim at the Validation, to achieve faster the main goal Support validations also with the needed algorithm for calculations Adopt a language open to improvements Face the compilation in following steps (facing also more complex algorithm)

The effort available The interested SWG members were identified to form a joint TWG-SWG subgroup It was asked to the interested members the FTE available to WP6 in the next year and a half (from June 2013 to Dec. 2014) At the moment, IMF, Eurostat and Banca d’Italia gave their concrete availability At the moment, the effort available is: –1,1 FTE of person-power –Consultancy in 2014 for about 0.5 FTE for 6 months

Goals of the work  Make it possible to: define the algorithms for data validation and calculation exchange such algorithms between different organizations use such algorithms in the collection and production processes

May a language be agnostic? Agnostic: independent of the particular standard and applicable to all of them (SDMX, DDI, GSIM) Property of closure: a language must operate the model artefacts to produce other model artefacts A language built for SDMX wouldn’t fit DDI & GSIM and vice-versa, because the artefacts are different How can a shareable language be built ?

The first idea If it existed a formal and unequivocal mapping, the same language could be applied to the matching artefacts Last february a task force has been formed within the DDI-SDMX dialogue to map GSIM with SDMX and DDI Some SDMX TWG and SWG members has been involved and a mapping has been done At the moment, the mapping isn’t formal and unequivocal enough Moreover, alternative mappings are possible as a consequence of alternative profiles of application of the standards

The idea of the “agnostic” model Refer the language to an agnostic information model, in principle independent of SDMX, DDI, GSIM The different standards may adopt the language by mapping their artefacts to the agnostic model The model should be as much minimal and simple as possible, just to support validations and calculations

3 June 2013 SDMX Technical Working Group Luxembourg 15 Possile Workstreams –Language for defining expressions (expression language) – that is the list of the operators available and their formal grammar –SDMX IT formats and messages for exchanging the definitions of the expressions –SDMX Web Services to interact with the registry (dictionary), for storing the expressions and retrieving them

Possible working steps for the language Identify the use cases –Data to be supported –Algorithms to be supported Design the reference model Define the language general behaviour –Different dimensionality –Missing / unknown data –Aggregation on hierarchies –Historicity of expressions –Multi-measure and attributes behaviour –… Identify the operators and their syntax

Proposed Plan Use cases for Validation: –From each institution: end of August 2013 –Merged in a common list: end of October 2013 –Finalization of the cases: next SDMX TWG meeting Reference Model Design: Jan – March 2014 Language general behaviour: Jan – March 2014 Operators and their syntax: –Basic set: April – June 2014 –Integration and finalization – July-Oct 2014 –Documentation – Oct-Dic 2014

SDMX activities SDMX IT formats and messages: –Design: Jan – June 2014 ??? –Formalization: July – Oct 2014 ??? SDMX Web Services: –Design: Jan – June 2014 ??? –Formalization: July – Oct 2014 ???

Requisites for the Governance Standard language under a common governance Need of customized parts under the private governance of single institutions, which integrate the language for their own purposes Coordination rules between the standard part and the customized parts

Wednesday 9.30 – special session Presentation of the work made in Bank of Italy Presentation of the work made in Eurostat Presentation of the work made in Unesco Discussion Organization of the future work and agreement on the work plan

December 2012 SDMX Technical Working Group Paris WP Item 6 Expressions and Calculations