New transmission methods: Use the most adapted transmission methods. EDAMIS 4 New transmission methods: Use the most adapted transmission methods. Directorate B - Methodology; corporate statistical and IT services Eurostat June 19th, 2019

Agenda Introduction Choose the most adapted transmission methods Esden Client sFTP Client Testing of Esden Client by TCOs

Introduction Secure networks

EDAMIS and the networks EDAMIS is available through different networks The Internet TESTA CCN Global: available all over the world Low cost Security level: low, available to the general public

TESTA and CCN CCN TESTA Member State CUSTOMS Managed by European Commission CCN CUSTOMS TESTA NATIONAL NETWORK NATIONAL NETWORK NATIONAL NETWORK NATIONAL NETWORK

Choose the most adapted transmission methods

Transmission methods EDAMIS EDAMIS PORTAL EDAMIS sFTP AS4 access AS4 STANDARD SOFTWARE EDAMIS PORTAL EDAMIS sFTP EDAMIS AS4 access point AS4 SOFTWARE BY EUROSTAT WEB SERVICES ESDEN Client

Choose the most adapted Transmission methods Criteria to compare the transmission methods and to choose the most adapted ones: Use standard software Automatic/manual transmission Support large files Provide facilities for encryption Need local installation and maintenance Accessibility to secure networks

Comparison of the different methods

ESDEN Client

ESDEN Client Generalities

Specific characteristics Need for installation: A computer with Java 8+ installed and at least 2 GB of memory. The computer must have access to the network that will be used: Internet, TESTA or CCN. For installation, follow all steps from the user guide Esden Client running: As command line, only to send (not to receive data files) As deamon, it is running permanently as a service (it can send and receive datafiles)

Sending types 2 possibilities to send datafiles: The Name of the file following the EDAMIS dataset naming convention (DSNC). In this case, the metadata will be recognised from the file name. Package the file using the EDAMIS ZIP format. In this case, the zip file should contain: - A XML containing the metadata (Mandatory) - The data file (Mandatory) - An explanatory note (Optional)

Sending Datafiles respecting the DSNC Dataset ID Domain ID ‘_’ Dataset structure ID ‘_’ Periodicity Examples: DEMONAT_RAPID_A WASTE_DISPO_M Dataset occurrence ID Dataset ID ‘_’ Country code ‘_’ Year ‘_’ Period DEMONAT_RAPID_A_FI_2011_0000 WASTE_DISPO_M_FI_2012_0001

Sending ZIP file format 2 possibilities to produce a XML metadatafile: Extraction of the XML Metadata file from EDAMIS4 Portal Following the specifications presented in the doc “how-to-exchange-files-using-EDAMIS4-3.1”.

Produce a XML metadata file with EDAMIS4 portal Upload the data file (drag and drop) Click on this small icon to perform the extraction Fill in all fields

Produce a XML metadata file with EDAMIS4 Portal Provided directly by the Esden Client Defined by the selection done in EWP EDAMIS4 Portal

ESDEN Client Starting, Sending and receiving datafiles

ESDEN Running as deamon Starting using batch or prompt command

Interfaces / Folders To configure ESDEN Client To send / receive datafile To Monitor

To store the datafile sent Send /receive folders To receive datafile To send datafile To store the datafile sent

Workflow related to the Outbox Global schema (3 folders to send datafiles) User choose to send file using DSNC “rules” User choose to send file using XLM metadata “rules”

Workflow related to the Outbox Transmission respecting the DSNC rules User will decide if he will encrypt itself or not

Workflow related to the Outbox Transmission using XML metadatafile

Receiving datafiles in Esden Client Transmission id

Receiving datafiles in Esden Client

Receiving datafiles in Esden Client Sender country/organisation Email from the sender Dataset/period

Esden Client User’s management Dataset management How to monitor the client Error management

User’s management in Esden Client Create a folder in <ESDEN_DATA>/outbox for each user, the folders name must be the user email Define the access rights to the user to his specific folder (read, write and access) Each user now has rights under the folder with their email to send datafile No synchronization of users between EDAMIS 4 and ESDEN Client is done. Users defined in ESDEN Client may not exist in EDAMIS 4.

Dataset management in Esden Client Dataset management is performed in EDAMIS4. Synchronisation between EDAMIS4 and ESDEN Client is done each time Esden client is restarted. Public keys linked to the dataset (confidentiality) are automatically imported from EDAMIS4.

How to monitor an Esden Client In order to manage the Disk space, monitoring inbox and sent folders are necessary. Delete any unnecessary file inside these folders. Transmission errors can be checked in Files ending in .error in the <ESDEN_DATA>/outbox folders Files ending in .retry in the <ESDEN_DATA>/outbox folders, this file stay there for a long period Json file (Log)

Example of error message Sending with DSNC mode: EDAMIS2_A26_M_DE_2018_0000.doc Specific error message displayed

Example of error message EDAMIS2_A26_M_2018_0015.doc Specific error message displayed

Possible error messages (sending using DSNC)

Error message in the log Extraction from the log file (Json file)

How to resent a datafile Delete the original datafile and the file with extention « .error » from the outbox Correct the error and put the datafile in the correct folder again

sFTP Client

Differences between sFTP and Esden Client Only XML Metadata sending can be used sFTP doesn’t check everything (period,year,…) Check to be done by ESDEN Server Error sent by Email Only one sending folder Only one receiving folder Encryption only possible by the user

sFTP Client Sending / Receiving datafiles

Prerequisite Prior registration for sFTP sending with Eurostat. After registration, the user receives credentials of an sFTP account: a user name and a PW. Optional: if you intend to sign the files you send over sFTP, you need the PGP key that was generated during the registration. A standard client that supports sFTP (e.g. OpenSSH, PuTTY, FileZilla). Having access to one of the sFTP servers of the European Commission: (Internet or TESTA)

To use sFTP To send a file To receive a file Create an XML file with the metadata (Optional) Sign the file Package the XML and the data file in a ZIP file Connect to the sFTP server of Eurostat Upload it to the folder /fromUser/ To receive a file Connect regularly to the sFTP of Eurostat Check the contents of folder /toUser/ Unpack the ZIP files received

Create the metadata file <formatVersion>2.0</formatVersion> <transmission> <transmissionId>DK-DST-Client1_201903201400</transmissionId> <from> <country>DK</country> <organisationID>DK1</organisationID> <senderEmail>user.name@dst.dk</senderEmail> </from> <data> <datasetId>ITGMDE_INTRAEU_M</datasetId> <year>2019</year> <period>0001</period> <fileFormat>gpg</fileFormat> <dataFilename>micro-data-file-20190219T1533.txt.gpg</dataFilename> <isEncrypted>true</isEncrypted> </data> <to/> </transmission> </metadata> Unique id for each transmission Sender info: country/organisation File info: dataset, period, format file name,

Create the ZIP file

Open sFTP Client FileZilla

Choose the correct folder Select folder: fromUser

Drag and drop the ZIP file Sending folder Drag and drop the ZIP file

Receiving folder Check for arrival of new files

Drag and drop the ZIP file Receiving folder Drag and drop the ZIP file

ESDEN Client Testing proposition of installation by TCOs

Testing proposition Installation (creation of a pair of keys, registration, configuration) User Management Creation of different users (emails folder) Manage the permission on the user’s folder Transmission of datafile Using DSNC Mode Respecting the DSNC No respecting the DSNC Managing the error message Sending confidential datafiles through “Non encrypted” and “Encrypted” folders

Testing proposition Control of the datafiles in the “Sent” folder Using XML Metadata mode Using correct XML Using incorrect XML Managing the error message Control of the datafiles in the “Sent” folder Receiving datafiles with ESDEN Client Checking of the metadatafile XML Checking of the datafiles Control of the Datafile traffic in EDAMIS4?