1. (COmmon Reference Environment)
CORE ESSnet
(COmmon Reference Environment)
progress report
Carlo Vaccari
Istat - Italy
MSIS Meeting - Luxembourg May 1

2. Outline Introduction and history CORE objectives CORE where we are
Architecture implementation
Information model
CORE and SDMX
CORE and GSIM
MSIS Meeting - Luxembourg May

3. CORA ESSnet Financed by Eurostat under 2009 Statistical Workprogramme
Countries involved: it (coordinator), ch, dk, lv, nl, no, se
Duration: October October 2010
MSIS Meeting - Luxembourg May

4. CORA Technical Architecture
CORA Model: two dimensions
Functional dimension
Construction dimension
Adoption of GSBPM 4.0
9 subprocesses of level 2 1
Specify
Needs 2
Design 3
Build 4
Collect 5
Process 6
Analyse 7
Disseminate 8
Archive 9
Evaluate
MSIS Meeting - Luxembourg May

5. Construction Dimension: Layers
Figures
Time Series
Statistic
Population
Unit
Variable
Value
A domain of interest documented by
statistical products
Statistical series over time
Integrated or simple statistical product for a
given time
A population at a given time
A statistical unit at a given time
A statistical variable at a given time
A logical representation of the value of a
variable
MSIS Meeting - Luxembourg May

6. CORA Model Grid
Statistical processes compliant to CORA model are intended to be designed by statisticians
MSIS Meeting - Luxembourg May

7. After CORA…CORE!
COmmon Reference Environment (CORE), financed by Eurostat under 2010 Statistical Workprogramme
Countries involved: it (coordinator), fr, nl, no, pt, se
Duration: December January 2012
MSIS Meeting - Luxembourg May

8. CORE Principal Outcomes
Environment for the definition and execution of statistical processes
Definition of a process in terms of services selected from an available repository
Execution of the composed workflow
MSIS Meeting - Luxembourg May

9. CORE Outcomes: Design CORA model → CORA information model
Design of CORE services and processes
MSIS Meeting - Luxembourg May

10. CORE Outcomes: Implementation
Selection of available middleware solutions for process execution
Realization of an environment able to permit the execution of processes:
Interfaces (GUIs) for defining CORE processes for statistical users
Integration APIs
Repository of integration layers
MSIS Meeting - Luxembourg May

11. CORE Outcomes: Testing
Realization of processes starting from services implementing some GSBPM phase
Evaluation of costs related to integration
Prototype implementation (to be engineered)
MSIS Meeting - Luxembourg May

12. CORE Architecture (1)
GUIs to support modelling of CORE processes according to the CORA grid
Modeling & control flow constructs
Drag & drop facilities for process design
Global schema
Implementation: we are evaluating the usage of an open process editor tool (Oryx - potsdam.de/Oryx/WebHome )
MSIS Meeting - Luxembourg May 12

13. CORE Architecture (2) Process Runtime Controlled execution of services
Implementation: integration of existing workflow solutions, currently in evaluation phase
Service Runtime
Integration APIs (in-out data transformation)
Service execution
Implementation: CSV and SQL data transformations are currently being implemented
Service Repository
Deployment of services
MSIS Meeting - Luxembourg May

14. CORE Information Model (1)
First draft of CORE information model
Del. 2.1 released: requirements for the model of the interface through which statistical services will communicate
Information Model to be released, currently in discussion phase
MSIS Meeting - Luxembourg May

15. CORE Information Model (2)
Design Principles (in discussion):
Rectangular data sets (rows & columns)
Strong typing (data, rules, parameters)
Dataset kinds (eg micro/aggreg)
Free-style arguments (eg scripts tool dependent)
Other (service arguments and infos)
MSIS Meeting - Luxembourg May

16. CORE and SDMX Both initiatives foster standardization CORE SDMX
Focus on standardization of processes and data exchanges (mainly) intra-NSI
SDMX
Focus on standardization of processes and data exchanges (mainly) inter-NSIs (or between NSIs and international organizations)
MSIS Meeting - Luxembourg May 16

17. CORE and SDMX - 2 CORE SDMX
Focus on all phases of statistical processes
Both micro and macro data considered
SDMX
Focus (mainly) on dissemination phase
Mainly macro data considered
MSIS Meeting - Luxembourg May 17

18. Information Model Both propose an information model
CORE information model
Takes explicitly process dimension into account through GSBPM
Data dimension
SDMX information model
Mainly focused on data dimension
MSIS Meeting - Luxembourg May 18

19. CORA Information Model
MSIS Meeting - Luxembourg May

20. SDMX Data & Metadata Information Model
Data or Metadata Structure Definition
uses specific data or metadata structure
can have child categories
can be linked with categories from multiple category schemes
Data or Metadata Set
Data or Metadata Flow
Category
conforms to business rules of the data or metadata flow
comprised of subject or reporting categories
can get data from multiple data providers
Data Provider
Provision Agreement
Category Scheme
MSIS Meeting - Luxembourg May

21. On Information Models Different abstraction levels CORE SDMX
“Higher” modelling level
E.g.: statistics as tabular data
SDMX
“Lower” modeling level
E.g.: aggregated data set with dimensions, attributes and measures
MSIS Meeting - Luxembourg May

22. Open Issues - 1
Can we use SDMX for micro and macro data exchanges in a CORE process?
Need for mapping of information models
MSIS Meeting - Luxembourg May

23. On-Going Work - 1 CORE implementation scenario within Istat
Main phases: Sample selection and allocation
CORE wrapping of available SAS and R procedures
MSIS Meeting - Luxembourg May

24. On-Going Work - 2 Design and implementation of CORE Integration APIs
Possible in/out SDMX translations
CORE TOOL
SDMX
SDMX
IAPI
TOOL
IAPI
MSIS Meeting - Luxembourg May

25. Open Issues - 2 What about metadata?
CORE: Data and metadata managed in the same way
SDMX:
Distinction between structural metadata and reference metadata
Dedicated effort for metadata management
MSIS Meeting - Luxembourg May

26. Collaboration between CORE/SDMX ESSnets
CORE planned deliverable on “Feedbacks on SDMX Usage in CORE”
Periodical meetings inside Istat between coordinators of the two ESSnets
Exchanges of resources between the two ESSnets
MSIS Meeting - Luxembourg May

27. CORE and GSIM
GSIM: Generic Statistical Information Model deliverable from OCMIMF Operationalising a Common Metadata/Information Management Framework activity inside Statistical Network
Ambiguity on the acronym: reference to “generic statistical information model” in CORE ESSnet proposal
In March started activity to clarify relationships (thanks to J.P.Kent and A.Hamilton)
MSIS Meeting - Luxembourg May

28. CORE and GSIM
First analysis and discussions: the deliverables from the two initiatives are complementary in intent and do not overlap in concept
Necessary to avoid gaps and/or duplications and ensure the complementary relationship
MSIS Meeting - Luxembourg May

29. CORE Information Model
CORE will define a very generic information model (CORE-IM) for the interface through which statistical services will communicate with each other within the framework of the CORA model
As a communication protocol, CORE-IM focuses on the “postal envelope” used when passing information between services, rather than focusing in detail on the information being communicated (ie what is inside)
MSIS Meeting - Luxembourg May

30. CORE and GSIM
CORE-IM current hypothesis: to support a flag to indicate if the information being communicated is described within GSIM
→ without claiming to align the semantics of the content (eg, “classification”), but only to alert a consuming service which “understands” GSIM that it can relate the content to GSIM
MSIS Meeting - Luxembourg May

31. Complementary nature
CORE-IM supporting semantic interoperability at a very high, abstract level (“here is an information object, along with the ‘envelope’ information about it”) where GSIM can provide greater semantic precision to a subset of information objects communicated using CORE
CORE supporting communication between services
→ substantial interoperability benefits
Information aligned with GSIM semantics
→ further level of interoperability
MSIS Meeting - Luxembourg May

32. Complementary nature
GSBPM reference model for statistical business processes
GSIM reference model for information input to, used by and produced by those processes
Models are independent → it's possible to use one without the other
CORE-IM recognizes and uses GSBPM and (hopefully) will do the same with regard to GSIM, giving them a potential contact point
MSIS Meeting - Luxembourg May

33. Coordination in practice
Need to maximize the extent to which these synergies are achieved in practice:
Members in common (no, se)
ABS leader for the OCMIMF and observer in CORE
CORE members external reviewers for GSIM material
CORE WP2 “co-ordination input” from the OCMIMF collaboration team in regard to deliverables
Half day session at METIS workshop (October) presenting CORE and OCMIMF works to external metadata specialists
Common documents in preparation
...
MSIS Meeting - Luxembourg May