1. New transmission methods: Use the most adapted transmission methods.
EDAMIS 4
New transmission methods:
Use the most adapted transmission methods.
Directorate B - Methodology; corporate statistical and IT services
Eurostat
June 19th, 2019

2. Agenda Introduction Choose the most adapted transmission methods
Esden Client
sFTP Client
Testing of Esden Client by TCOs

3. Introduction Secure networks

4. EDAMIS and the networks
EDAMIS is available through different networks
The Internet
TESTA
CCN
Global: available all over the world
Low cost
Security level: low, available to the general public

5. TESTA and CCN CCN TESTA Member State CUSTOMS
Managed by European Commission
CCN
CUSTOMS
TESTA
NATIONAL
NETWORK
NATIONAL
NETWORK
NATIONAL
NETWORK
NATIONAL
NETWORK

6. Choose the most adapted transmission methods

7. Transmission methods EDAMIS EDAMIS PORTAL EDAMIS sFTP AS4 access AS4
STANDARD SOFTWARE
EDAMIS
PORTAL
EDAMIS
sFTP
EDAMIS
AS4 access
point
AS4
SOFTWARE
BY EUROSTAT
WEB
SERVICES
ESDEN
Client

8. Choose the most adapted Transmission methods
Criteria to compare the transmission methods and to choose the most adapted ones:
Use standard software
Automatic/manual transmission
Support large files
Provide facilities for encryption
Need local installation and maintenance
Accessibility to secure networks

9. Comparison of the different methods

10. ESDEN Client

11. ESDEN Client Generalities

12. Specific characteristics
Need for installation:
A computer with Java 8+ installed and at least 2 GB of memory.
The computer must have access to the network that will be used: Internet, TESTA or CCN.
For installation, follow all steps from the user guide
Esden Client running:
As command line, only to send (not to receive data files)
As deamon, it is running permanently as a service (it can send and receive datafiles)

13. Sending types 2 possibilities to send datafiles:
The Name of the file following the EDAMIS dataset naming convention (DSNC). In this case, the metadata will be recognised from the file name.
Package the file using the EDAMIS ZIP format. In this case, the zip file should contain:
- A XML containing the metadata (Mandatory)
- The data file (Mandatory)
- An explanatory note (Optional)

14. Sending Datafiles respecting the DSNC
Dataset ID
Domain ID
‘_’
Dataset structure ID
‘_’
Periodicity
Examples:
DEMONAT_RAPID_A
WASTE_DISPO_M
Dataset occurrence ID
Dataset ID
‘_’
Country code
‘_’
Year
‘_’
Period
DEMONAT_RAPID_A_FI_2011_0000
WASTE_DISPO_M_FI_2012_0001

15. Sending ZIP file format
2 possibilities to produce a XML metadatafile:
Extraction of the XML Metadata file from EDAMIS4 Portal
Following the specifications presented in the doc “how-to-exchange-files-using-EDAMIS4-3.1”.

16. Produce a XML metadata file with EDAMIS4 portal
Upload the data file (drag and drop)
Click on this small icon to perform the extraction
Fill in all fields

17. Produce a XML metadata file with EDAMIS4 Portal
Provided directly by the Esden Client
Defined by the selection done in EWP EDAMIS4 Portal

18. ESDEN Client Starting, Sending and receiving datafiles

19. ESDEN Running as deamon
Starting using batch or prompt command

20. Interfaces / Folders To configure ESDEN Client
To send / receive datafile
To Monitor

21. To store the datafile sent
Send /receive folders
To receive datafile
To send datafile
To store the datafile sent

22. Workflow related to the Outbox
Global schema (3 folders to send datafiles)
User choose to send file using DSNC “rules”
User choose to send file using XLM metadata “rules”

23. Workflow related to the Outbox
Transmission respecting the DSNC rules
User will decide if he will encrypt itself or not

24. Workflow related to the Outbox
Transmission using XML metadatafile

25. Receiving datafiles in Esden Client
Transmission id

26. Receiving datafiles in Esden Client

27. Receiving datafiles in Esden Client
Sender country/organisation
from the sender
Dataset/period

28. Esden Client User’s management Dataset management How to monitor the client Error management

29. User’s management in Esden Client
Create a folder in <ESDEN_DATA>/outbox for each user, the folders name must be the user
Define the access rights to the user to his specific folder (read, write and access)
Each user now has rights under the folder with their to send datafile
No synchronization of users between EDAMIS 4 and ESDEN Client is done. Users defined in ESDEN Client may not exist in EDAMIS 4.

30. Dataset management in Esden Client
Dataset management is performed in EDAMIS4. Synchronisation between EDAMIS4 and ESDEN Client is done each time Esden client is restarted.
Public keys linked to the dataset (confidentiality) are automatically imported from EDAMIS4.

31. How to monitor an Esden Client
In order to manage the Disk space, monitoring inbox and sent folders are necessary. Delete any unnecessary file inside these folders.
Transmission errors can be checked in
Files ending in .error in the <ESDEN_DATA>/outbox folders
Files ending in .retry in the <ESDEN_DATA>/outbox folders, this file stay there for a long period
Json file (Log)

32. Example of error message
Sending with DSNC mode: EDAMIS2_A26_M_DE_2018_0000.doc
Specific error message displayed

33. Example of error message
EDAMIS2_A26_M_2018_0015.doc
Specific error message displayed

34. Possible error messages (sending using DSNC)

35. Error message in the log
Extraction from the log file (Json file)

36. How to resent a datafile
Delete the original datafile and the file with extention « .error » from the outbox
Correct the error and put the datafile in the correct folder again

37. sFTP Client

38. Differences between sFTP and Esden Client
Only XML Metadata sending can be used
sFTP doesn’t check everything (period,year,…)
Check to be done by ESDEN Server
Error sent by
Only one sending folder
Only one receiving folder
Encryption only possible by the user

39. sFTP Client Sending / Receiving datafiles

40. Prerequisite
Prior registration for sFTP sending with Eurostat. After registration, the user receives credentials of an sFTP account: a user name and a PW.
Optional: if you intend to sign the files you send over sFTP, you need the PGP key that was generated during the registration.
A standard client that supports sFTP (e.g. OpenSSH, PuTTY, FileZilla).
Having access to one of the sFTP servers of the European Commission: (Internet or TESTA)

41. To use sFTP To send a file To receive a file
Create an XML file with the metadata
(Optional) Sign the file
Package the XML and the data file in a ZIP file
Connect to the sFTP server of Eurostat
Upload it to the folder /fromUser/
To receive a file
Connect regularly to the sFTP of Eurostat
Check the contents of folder /toUser/
Unpack the ZIP files received

42. Create the metadata file
<formatVersion>2.0</formatVersion>
<transmission>
<transmissionId>DK-DST-Client1_ </transmissionId>
<from>
<country>DK</country>
<organisationID>DK1</organisationID>
</from>
<data>
<datasetId>ITGMDE_INTRAEU_M</datasetId>
<year>2019</year>
<period>0001</period>
<fileFormat>gpg</fileFormat>
<dataFilename>micro-data-file T1533.txt.gpg</dataFilename>
<isEncrypted>true</isEncrypted>
</data>
<to/>
</transmission>
</metadata>
Unique id for each transmission
Sender info: country/organisation
File info: dataset, period, format file name,

43. Create the ZIP file

44. Open sFTP Client FileZilla

45. Choose the correct folder
Select folder: fromUser

46. Drag and drop the ZIP file
Sending folder
Drag and drop the ZIP file

47. Receiving folder
Check for arrival of new files

48. Drag and drop the ZIP file
Receiving folder
Drag and drop the ZIP file

49. ESDEN Client Testing proposition of installation by TCOs

50. Testing proposition
Installation (creation of a pair of keys, registration, configuration)
User Management
Creation of different users ( s folder)
Manage the permission on the user’s folder
Transmission of datafile
Using DSNC Mode
Respecting the DSNC
No respecting the DSNC
Managing the error message
Sending confidential datafiles through “Non encrypted” and “Encrypted” folders

51. Testing proposition Control of the datafiles in the “Sent” folder
Using XML Metadata mode
Using correct XML
Using incorrect XML
Managing the error message
Control of the datafiles in the “Sent” folder
Receiving datafiles with ESDEN Client
Checking of the metadatafile XML
Checking of the datafiles
Control of the Datafile traffic in EDAMIS4?