1. Eurostat business process (data processing) & CVD October 2007

2. 2 Content Shown today: 1.Eurostat business process (data processing) & CVD – main presentation shown today 1.Proposed business model 2.Its correspondence to CVD architecture For reference: 1.List of sub-processes and sub-sub-processes 2.CVD modules and their relation to business sub-processes and sub-sub-processes 3.CVD modules brief description Implementation modes and availability schedule

3. 3 Eurostat business process (data processing) Process Sub-process (or sub-sub-process) Sub-process (or sub-sub-process) without software development component

4. 4 Processes Manage meta- information 5 Disseminate 4 Validate 2 Analyse 3 Collect 1 1. Proposal for discussion 2. Pick-and choose & mix and match No order in execution although the numbering follows typical order logic.

5. 5 Data files MH TDS Statistical data and metadata Internet Portal NUI DL CVD MANAGER Pre-treated data Validated data Processed data Reference Environment data BB Domain specific software DR / DL BB Domain specific software DR / DL BB Domain specific software ASSIST User support EDAMIS CVD ARCHITECTURE

6. 6 Notes Each BB can be run in batch (CVD) and interactive mode (stand alone) TDS, EDAMIS, NUI, MH, DL are (or will be) compulsory BBs a set of tool to mix and match Domain specific software – procedures that are unique to a few statistical applications and benefits from developing a generalised solution considered nonexistent

7. 7 Cooperate with providers 1.4 Acquire domain intelligence 3.1 Set up collection 1.1 Run collection 1.2 Load data 1.3 Edit 2.1 Detect & treat outliers 2.2 Impute 2.3 Derive new variables 2.4 Integrate and load data 2.5 Prepare tables for Dissemination 3.5 Interpret and explain 3.4 Check quality 3.3 Produce statistics or indicators 3.2 Manage customer queries 4.2 Produce products 4.1 Collect 1 Disseminate 4 Analyse 3 Validate 2 Manage meta- Information 5

8. 8 Data files MH TDS Statistical data and metadata Internet Portal NUI DL CVD MANAGER Pre-treated data Validated data Processed data Reference Environment data BB Domain specific software DR / DL BB Domain specific software DR / DL BB Domain specific software COLLECT VALIDATE ANALYSE DISSEMINATE ASSIST User support EDAMIS MANAGE META-INFORMATION

9. 9 Summary of CVD modules & business process  +   Module especially designed for the sub-process Module designed for other sub-process but could be used for this sub-process as well if the functionalities are appropriate Modules that are used throughout many processes Other uses may be possible in specific cases

10. 10

11. 11 END OF MAIN MODULE

12. 12 Navigation Previous slide Most of boxes and frames on presentation contain links Names of the CVD modules and BBs are usually link enabled Names of the processes as well

13. 13 START OF SUB-SUB-PROCESS LIST

14. 14 Manage provider relationship 1.4.1 Maintain provider information 1.1.7 Manage provider burden across surveys 1.4.2 Train staff on collection 1.1.6 Run collection test 1.1.5 Set up collection security 1.1.4 Configure collection systems 1.1.3 Pre-validate data 1.3.2 Allocate collection responsibilities 1.1.2 Produce collection strategy and schedule 1.1.1 Monitor & report on collection 1.2.5 Follow up non-responses 1.2.4 Collect data 1.2.3 Request data 1.2.2 Contact provider with pre-collection information 1.2.1 Load data & metadata to data environments 1.3.3 Receive electronic data 1.3.1 Cooperate with providers 1.4 Load data 1.3 Run collection 1.2 Set up collection 1.1 From data arrival to data ready for processing (raw data) COLLECT

15. 15 Evaluate imputation results 2.3.4 Run imputation 2.3.3 Identify items for special treatment 2.3.2 Impute 2.3 Revise existing data 2.3.1 Detect and Treat Outliers 2.2 Integrate & load data 2.5 Derive New Variables 2.4 Edit 2.1 Detect outliers 2.2.1 Manually edit variables 2.1.3 Treat outliers 2.2.2 Provide feedback to providers 2.5.3 Evaluate quality of incoming data 2.5.2 Derive variables / indicators 2.4.1 Prepare & load data 2.5.1 Resolve versioning 2.1.1 Auto edit variables 2.1.2 From raw, collected data to validated data VALIDATE

16. 16 Prepare tables for dissemination 3.5 Produce statistics & indicators 3.2.1 Research data sources & methodology 3.1.3 Produce seasonal adjustment 3.2.2 Assess quality measures against quality standards 3.3.4 Compare with previous periods 3.3.2 Check non-sampling errors 3.3.1 Apply confidentiality rules 3.5.1 Produce statistics & indicators 3.2 Interpret and explain 3.4 Check quality 3.3 Acquire domain intelligence 3.1 Collect external information 3.1.1 Collect internal data & information 3.1.2 Produce reports 3.1.6 Evaluate & synthesise knowledge 3.1.5 Manage domain knowledge 3.1.4 Prepare microdata files 3.2.3 Carry out in-depth statistical analysis 3.4.2 Analyse time series dimension 3.4.1 Verify against expectations & intelligence 3.3.5 Confront with other data sources 3.3.3 Produce quality measures for statistics 3.2.4 Approve explanation and statistics 3.4.4 Identify story / commentary to the data 3.4.3. Carry out edit and consistency checks 3.5.2 Finalise tables 3.5.3 Approve tables 3.5.4 From validated data to analysed data and tables ANALYSE

17. 17 Get customer feedback 4.2.4 Set up for production 4.1.1 Transfer data from internal to external environment 4.1.2 Media relations 4.1.3 Other DG / NSI relations 4.1.4 Lift embargo and release products 4.1.5 Analyse and resolve query 4.2.3 Review and record customer query 4.2.1 Allocate query 4.2.2 Produce products 4.1 Manage customer queries 4.2 From tables and analysis to customised disseminated products DISSEMINATE

18. 18 Produce information and explanation 5.2 Determine information and explanation 5.1 Appraise the long- term value of metadata 5.5 Prepare metadata for repository 5.3 Load repositories 5.4 MANAGE META-INFORMATION

19. 19 END OF SUB-SUB-PROCESS LIST START OF BB & SUB-PROCES CROSS REFERENCE

20. 20 1.1 Set up collection EDAMIS Train staff on collection 1.1.6 Run collection test 1.1.5 Set up collection security 1.1.4 Configure collection systems 1.1.3 Allocate collection responsibilities 1.1.2 Produce collection strategy and schedule 1.1.1 Maintain provider information 1.1.7

21. 21 1.2 Run collection EDAMIS Monitor & report on collection 1.2.5 Follow up non- responses 1.2.4 Collect data 1.2.3 Request data 1.2.2 Contact provider with pre-collection information 1.2.1

22. 22 1.3 Load Data EDAMIS Editing BB / EDAMIS Loader BB Pre-validate data 1.3.2 Load data & metadata to data environments 1.3.3 Receive electronic data 1.3.1

23. 23 2.1 Edit EDAMIS Reader BB Editing BB GSAST Loader BB Resolve versioning 2.1.1 Auto edit variables 2.1.2

24. 24 2.2 Detect and treat outliers Outliers BB Reader BB GSAST Derivation BB GSAST Loader BB Detect outliers 2.2.1 Treat outliers 2.2.2

25. 25 2.3 Impute Imputation BB Reader BB Derivation BB GSAST Loader BB Evaluate imputation results 2.3.4 Run imputation 2.3.3 Identify items for special treatment 2.3.2 Revise existing data 2.3.1

26. 26 2.4 Derive new variables Derivation BB Reader BB GSAST Loader BB Derive variables / indicators 2.4.1

27. 27 2.5 Integrate and load data Loader BB GSAST Editing BB Evaluate quality of incoming data 2.5.2 Prepare & load data 2.5.1

28. 28 3.2 Produce statistics or indicators Reader BB Derivation BB GSAST Economic indices BB Seasonal adjustment BB GSAST Derivation BB GSAST Derivation BB Loader BB Produce statistics & indicators 3.2.1 Produce seasonal adjustment 3.2.2 Prepare microdata files 3.2.3 Produce quality measures for statistics 3.2.4

29. 29 3.3 Check Quality Editing BB Derivation BB ANALYTICAL GSAST Reader BB Outliers BB Economic indices BB NUI Assess quality measures against quality standards 3.3.4 Compare with previous periods 3.3.2 Check non- sampling errors 3.3.1 Confront with other data sources 3.3.3 {

30. 30 3.4 Interpret and explain Analytical BB GSAST Reader BB Seasonal adjustment BB Economic indices BB NUI Carry out in- depth statistical analysis 3.4.2 Analyse time series dimension 3.4.1 Identify story / commentary to the data 3.4.3 {

31. 31 3.5 Prepare tables for dissemination Confidentiality BB Reader BB Editing BB GSAST Derivation BB Loader BB Apply confidentiality rules 3.5.1 Carry out edit and consistency checks 3.5.2 Finalise tables 3.5.3

32. 32 4.1 Produce products NUI Set up for production 4.1.1 Transfer data from internal to external environment 4.1.2 Lift embargo and release products 4.1.5

33. 33 4.2 Manage customer queries ASSIST Get customer feedback 4.2.4 Analyse and resolve query 4.2.3 Review and record customer query 4.2.1 Allocate query 4.2.2

34. 34 5 Manage meta-information MH Prepare metadata for repository 5.3 Load repositories 5.4

35. 35 END OF BB & SUB-PROCES CROSS REFERENCE START OF BB DESCRIPTION

36. 36 Target Data Storage (TDS) not a software unique structure of the database contains both statistical data and metadata (all kinds) uniqueness allows to implement coherence rules for the data and metadata throughout the CVD processes structure allowing new types (considering way they are used) of metadata to be added

37. 37 CVD MANAGER To implement a workflow approach for the production process based on a design of the particular production process To control and schedule the invoking of the CVD components within the various stages of statistical production process At each stage of the production process will interact with the human domain manager or with a software component to: Launch software Control output Request input Provide status reports on whole process and its individual components

38. 38 EDAMIS supports the transmission of statistical data from Member States to Eurostat ensures secure and well monitored transmission of data through a single reception point delivery of data to production environments user access management links to structural metadata basic validation format conversion

39. 39 Loader BB loads data and reference metadata in the update or replace mode at the same time assuring its coherence with existing metadata algorithm contains coherence rules can be used any time during the processing for both data and reference metadata

40. 40 Reader BB reads data and metadata and assembles for further processing by other BBs (various formats) can be used any time during the processing for both data and metadata

41. 41 Editing BB executes editing rules optionally with reference data (lookup tables) intra-cell, intra-record (horizontal) and inter-record (vertical) rules reports on the rules execution allows interactive review of messages can be provided to MS for editing at source

42. 42 Outliers BB basic and statistical methods to identify outliers methods: –Hidiroglou-Berthelot and σ-gap –top and bottom – number or percentiles and conditions Reports on the execution in future multidimensional distance measures

43. 43 Imputation BB t.b.d. note: possibly based on BANFF software, any system should be really very similar to BANFF Implementation of various mathematical imputation methods last BB to be developed Scope not yet established

44. 44 Derivation BB Derives new variables optionally with reference data (lookup tables) intra-cell, intra-record (horizontal) and inter-record (vertical) derivations reports execution allows interactive review of messages Uses the same engine (subset) as editing BB

45. 45 Economic indices BB Calculates indices used in economy –Weighted arithmetic mean –Weighted geometric mean –Weighted harmonic mean –Laspeyres –Paasche –Lowe –Edgeworth –Bowley –Fisher –Laspeyres (Geometric) –Paasche (Geometric) –Törnqvist-Theil –Laspeyres (harmonic) –Paasche (harmonic) –Chain index –EKS(-S)

46. 46 GSAST Generic system for treating micro-data and operations of micro and macro-data from surveys Based on SAS base, BI server and Enterprise guide functionalities Also for unique or unusual processing requirements

47. 47 Seasonal adjustment BB Calculates seasonally adjusted time series. Based on X12 and Tramo Seats methods

48. 48 ANALYTICAL various mathematical and visual analysis and review of the data visualisation through graphing of the data statistical analysis (the exact scope not yet determined – possibly through SAS)

49. 49 Confidentiality BB performs confidentiality verification of tables applies various masking techniques assuring confidentiality of published statistics Based on CSB μ-argus and τ-argus

50. 50 NUI To provide access to the statistical reference databases of Eurostat. single tool for all data and metadata based on the principles of graphical tools highly interactive operation metadata is presented to the user shows relation of different types of metadata can be used inside Eurostat

51. 51 ASSIST User support tool Parallel to e-mail system (with attachments) Service request Request follow-up Searchable, central public knowledge database Decentralised help centres / persons Sub-systems by subject matter, geography or any other classification Access management (to appropriate parts of the system by administrative privileges or subject matter)

52. 52 MH – metadata handler (1 of 3) System for handling all the production aspects of classifications, associations and other statistical metadata. Updates on the Nomenclatures Codes Updates on label values of Nomenclature Codes Updates on the Relations between codes Updates on label values of Relations Export classifications, relations to files Create aggregates from relationships

53. 53 MH – metadata handler (2 of 3) Check Relationship Completeness Footnotes on labels Materialized View classifications & relationships: allows to create a subset of a classification or a relation by defining: Selection rules (wildcard expressions) SQL statements (SQL Generation wizard) Dictionary Automatic Creation and Update of relationships Creation through other existing relationships Update through Successor/Predecessor Multidimensional Nomenclatures Simple or Subkeys as code

54. 54 MH – metadata handler (3 of 3) Allows management: Dataset trees Composite and normal datasets (creation, update, etc…) Visibility and accessibility flags on objects (datasets, dictionaries, classifications, etc.) Classification’s default attribute Transposition of datasets (micro-data) implementation list (dictionary) Access Control Lists methods Confidentiality scripts attachments cells attachments (footnotes) presence table.

55. 55 END OF BB DESCRIPTION