1. EDAMIS The new EDAMIS 4 EDAMIS and VALIDATION SERVICES User Group
Javier MANSO DEL VALLE
Directorate B - Methodology; corporate statistical and IT services
Eurostat
October 9th, 2017

2. Agenda Why a new EDAMIS? What is new in EDAMIS 4 ?
Increased security
Secure networks
Easier to integrate
Easier to use
Better performance
Updated Web Forms
For more information

3. Why a new EDAMIS ?

4. Single Entry Policy
Data collected by all organisations of the ESS arrives at a common reception area in Eurostat, where they can be automatically monitored, checked and delivered to the target production environment with a set of common informatics tools.
Since July 1st 2008, it is compulsory to use EDAMIS for the transmission of all regular datasets to Eurostat.
EDAMIS

5. Why a new EDAMIS?
New business requirements coming from ESS Vision 2020
Adapt to changes in new technological landscape

6. ESS Vision 2020 Where do we want to be? How do we get there?
Five key areas for delivering our vision:
Identifying user needs and cooperation with stakeholders
Quality of European statistics
New data sources
Efficient and robust statistical processes
Dissemination and communication on European statistics
ESS = European Statistical System
The ESS Vision 2020 is a common strategic response of the European Statistical System (Eurostat, EU Member States and EFTA countries) to the challenges that official statistics are facing.
It was adopted by the ESS Committee in May 2014.
More information on the ESS Vision 2020:
The ESDEN project contributes to two of the five key areas defined in the ESS Vision 2020.

7. ESS Vision 2020 Key area 3: new data sources
We will exploit the potential of new data sources
Find suitable technology to use new data sources
A digital transformation is taking place across the globe: the data revolution is already presenting official statistics with profound strategic questions.
The ever increasing availability of data is a trend that is of strategic relevance for official statistics. We want to harness new data sources to produce meaningful statistics.
The possibility of using big data in the production of official statistics will be fully explored.

8. ESS Vision 2020 Key area 4: efficient and robust processes
Collaboration based on standards and common technological and statistical elements
Sharing IT services and infrastructure
We will use common methods and tools.
We will explore other areas and use all the opportunities, in particular when dealing with new technology-driven areas like big data, open data and visualization techniques where we expect concrete opportunities to emerge.
We will benefit from exchange of (micro)data, while fully respecting statistical confidentiality.
CONCLUSION: all the elements of the ESS Vision 2020 explained so far call for a modernisation of the data exchange infrastructure.

9. Technological drivers
Adapt to changes in new technological landscape
End of support for Java applets in browsers
Change of underlying technologies
New, re-written back-end in Java J2EE

10. ESDEN project ESDEN is a project of the ESS Vision 2020 Objective:
"Modernisation of the data exchange services across the ESS, increasing the data exchange capacity and upgrading the security to allow a more efficient and secure data exchange in the ESS"

11. What is new in EDAMIS 4 ?

12. What's NOT new? The role of EDAMIS as Single Entry Point
Core functionalities of EDAMIS
Dataset Inventory: organisations, datasets
Reports
Reminders, notifications

13. What is new in EDAMIS 4 ? Increased security

14. Encryption: already in EDAMIS 3
Files encrypted with PGP
Asymmetric encryption
Key pair generated by Eurostat
Private key kept by Eurostat
Public key provided to data providers
Encryption only from the NSI TO Eurostat
EDAMIS maintains a list of public keys
One key pair per domain
What else do we want to add to this timetable?

15. Encryption: improved in EDAMIS 4
Encryption supported TO and FROM Eurostat
Key management
One encryption key for each dataset + receiving organisation
EDAMIS keeps a list of all public keys
Organisations can generate their own keys, and upload public keys in EDAMIS 4
What else do we want to add to this timetable?

16. Two-way encryption in EDAMIS 4
From Member States to Eurostat
Encryption with the public key of the corresponding domain
Public keys of all domains are available in the EDAMIS portal
From Eurostat to Member State
Organisations update their public key in the EDAMIS portal
EDAMIS uses that public key to encrypt data sent to that organisation
NSI
What else do we want to add to this timetable?
NSI

17. Example: EDAMIS 4 Web Portal
ESDEN
EDAMIS 4
Web Portal
What else do we want to add to this timetable?
Application
NSI

18. Encryption in EDAMIS 4 Web Portal
User attaches the file, which is encrypted in the browser
PGP encryption
Public key of the domain
EDAMIS 4
ESDEN
EDAMIS 4
Web Portal
User submits the encrypted file
HTTPS
Internet / TESTA
ESDEN delivers the file
ESDEN keeps an encrypted copy of the file for a defined retention period
User accesses EDAMIS
HTTPS
Internet / TESTA
What else do we want to add to this timetable?
Application
NSI

19. Retention period in EDAMIS
Feature intended for compliance…
Domain managers can define a retention period for their datasets
ESDEN keeps an encrypted copy of all files for the defined period
When the retention period is reached, EDAMIS deletes the file
Expiration period can be 0, EDAMIS will delete the file immediately after delivery
What else do we want to add to this timetable?

20. Retention period in EDAMIS
… also has other uses
Copy of official submissions to Eurostat available
Copy of files received from Eurostat available
Verify the version of the file that was submitted
Recover files that were lost or modified
What else do we want to add to this timetable?

21. Consolidated logging
EDAMIS centralises all information on actions performed on every file
All actions done (e.g. file received, chunks joined, signature checked, file available, file delivered)
All files are hashed, possible to tell whether a file was ever received
What else do we want to add to this timetable?

23. What is new in EDAMIS 4 ? Secure networks

24. EDAMIS and the networks
EDAMIS is available through different networks
The Internet
TESTA
CCN
Global: available all over the world
Low cost
Security level: low, available to the general public

25. TESTA and CCN CCN TESTA Member State CUSTOMS
Managed by European Commission
CCN
CUSTOMS
TESTA
NATIONAL
NETWORK
NATIONAL
NETWORK
NATIONAL
NETWORK
NATIONAL
NETWORK

26. TESTA and CCN TESTA CCN Purpose Generic: exchange of information
Specific: customs and taxations
Point of access
National network
Customs office
Technology IP
IP, but restricted
Authentication
EU-Login
Security
High
Owner
European Commission
European Statistics are produced by Eurostat alone. The production of European Statistics is only possible thanks to the exchange of statistical data between the organisations that are part of the European Statistical System (ESS).

27. Secure connectivity: current status
Connected
to TESTA
to CCN
Unknown
NSIs
DK, ES, HR, HU, LV, NL, SE
Ongoing: AT, PL, UK
Total: 10
AT, BG, CZ, CY, DE, DK, EE, EL, HR, IT, LU, LT, MT, PL, PT, RO, SK, SI
Total: 18
BE, IE
Total: 2
ONAs
ES National Bank
ECB
All customs
Ongoing: BE NBB
All other ONAs

28. Web Portal: separate instances
NSI
DIGIT
Secure environment
EDAMIS 4
Reverse proxy
TESTA
Manual
data exchange
EDAMIS 4
Web Portal
(TESTA)
What else do we want to add to this timetable?
Reverse proxy
Internet
EDAMIS 4
Web Portal
(Internet)
Datasets can be restricted to a particular network
ESDEN

29. What is new in EDAMIS 4 ? Easier to integrate

30. Supported protocols EDAMIS EDAMIS PORTAL EDAMIS sFTP AS4 WEB SERVICES
STANDARD SOFTWARE
EDAMIS
PORTAL
EDAMIS
sFTP
EDAMIS
AS4
SOFTWARE BY EUROSTAT
WEB
SERVICES
ESDEN Client

31. Transmission methods compared
Machine to machine
Supports large files
Encryption facilities
Local installation needed
Uses standard software
Accessible through CCN
Accessible through TESTA
PORTAL
sFTP
AS4
CLIENT

32. Easier to integrate Promote use of standard protocols Allow automation
Connect to national systems
Comply with local policies

33. What is new in EDAMIS 4 ? Easier to use

34. EDAMIS dashboard EDAMIS home page Can be customised with widgets News
Last files sent
Notifications

35. Dashboard
configuration

37. Easier to send files To send a file Portal provides help
Drag and drop file
Click on "Perform transfer"
Portal provides help
Automatically fills metadata from the naming convention
Pre-validation functionality
Validate files before official submission

38. Click on the
"Transmissions" menu

39. In the menu, select
"Send datafile"

40. Drag and drop
your file
in this area

41. … so all metadata fields are filled automatically
The file follows the
EDAMIS naming
convention…
… so all metadata fields
are filled automatically
Click here to
validate your file
Click here to perform
the official transfer

42. Improved search function
Navigation is similar to EDAMIS 3
Search function provides faster access
Search the whole EDAMIS application
Domains, datasets
Users, organisations

43. Use the menu to
access the inventory

44. Use the new
search function

46. Management of rights TCOs can grant rights Users can request rights
Rights become effective
Users can request rights
TCO is notified
TCO can accept or refuse

47. Click to manage
your account

49. User clicks to
request rights

50. Pending requests rights for TCO

51. TCO can accept or refuse the requests

52. Pending Rights at the Dataset level

53. Replacement of Java applets
Java applets replaced by HTML5 and JavaScript
Standard technologies supported by major browsers

54. What is new in EDAMIS 4 ? Better performance

55. New architecture Based on Use of Prepared for scalability
HTML5, Javascript
Java 2 Enterprise Edition
Oracle database
Use of
Reverse proxies (Internet, TESTA)
Load balancers
Prepared for scalability
What else do we want to add to this timetable?

56. Application architecture
Linux VM1
WebLogic Cluster
Managed
server 1
Reverse proxy
Load balancer
Session
replication
Reverse proxy
Load balancer
What else do we want to add to this timetable?
Linux VM2
Managed
server 2

57. Benefits of new architecture
Improved file transfer
Faster
Higher volumes
Ready for growth
Possible to add resources
No need for re-design

58. What is new in EDAMIS 4 ? Web Forms

59. EDAMIS 4 Web Forms Objectives New development Keep same functionality
Minimal impact for production
New development
Java applets no longer used
New graphical interface
What else do we want to add to this timetable?

60. New in EDAMIS Web Forms New, HTML5-based interface
Possibility of pre-validation
Validation prior to official submission
Coherency of validation rules
Common set of rules for Web forms and rest of transmission methods

61. Coherency of validation rules
EDAMIS Web Forms
Data provider
Validation engine
What else do we want to add to this timetable?
EDAMIS STADIUM
European
statistics
Statistical production

62. Click on the
TRANSMISSIONS
menu

63. Click on
"Web Forms"

64. Click on "Create a new transmission" to create a new Web Form entry

66. Click to edit
an existing Web Form

69. For more information

70. For more infos, contact Eurostat
For questions:
The support team can be contacted using the following address
For information and discussions:
Participate to the Yammer group