Future of EDAMIS Webforms Item 3.2 Data Transmission Coordinators Group meeting 17-18 June 2019 Luca Gramaglia Eurostat, Unit B5

Situation last year… Request The EVUG members asked Eurostat to evaluate the continued need for Webforms, now that other transmission and conversion means are available. Response Webforms still present advantages over Excel: Data providers cannot change the layout of the Webforms Webforms make it possible to highlight cells that fail validation rules Webforms avoid some technical issues with Excel (e.g. floating point representation) For these reasons, there is currently no concrete plan to phase out Webforms. However, potential solutions to the problems mentioned above are being investigated.

Webforms: current usage All data used in this section covers the period April 1st 2018 – March 31st 2019

Webforms: what are they? Webforms are a data collection tool integrated in EDAMIS. Webforms allow data providers to manually input data in a tabular, Excel-like environment, while guaranteeing that the data arrives in Eurostat in an easily machine-readable format. Webforms also provide the possibility for domain managers to encode validation rules, which can be used by data providers to pre-validate their files prior to submission. Webforms were first introduced in the late 2000's. Over the years, efforts were made to align the Webforms to SDMX. Webforms currently output an SDMX-compatible file.

Webforms: why do they exist? The initial goal behind the development of Webforms was the phase-out the use of Excel to transmit data to Eurostat. The main disadvantages of Excel were the following: The flexibility of Excel made it difficult and resource-intensive to automate data loading and processing. Pre-validation functionalities could only be made available by using Excel formulas and macros. These were difficult to maintain and did not enjoy any corporate support. Excel is proprietary software. Data providers that used open-source alternatives (e.g. OpenOffice) found it difficult to provide data to Eurostat using Excel. Moreover, as a general principle, Eurostat should not force data providers to use proprietary software for data transmission.

Who uses Webforms - Eurostat Webforms are used today for 105 EDAMIS datasets. The overwhelming majority of these datasets are in Directorate E, with E1 in particular being a main user. Unit Number of Webform datasets Percentage of total Webform transmissions E1 62 86.5% E2 23 10.9% E3 16 2.2% F2 1 0.3%

Who uses Webforms – Data providers NSIs are responsible for 76% of Webform transmissions. This is in line with the Eurostat-wide average of 74%.

Webforms: How much are they used?

Webforms: did they succeed? Current usage of Webforms Current usage of Excel # of EDAMIS datasets 106 # of transmissions (last 12 months) 9995 (11.2%) Volume of transmissions 231 MB (0.1%) # of EDAMIS datasets ~180 # of transmissions (last 12 months) 18145 (20.4%) Volume of transmissions (last 12 months): 21555 MB (13.9%)

Webforms: did they succeed? Webforms were not successful in phasing out Excel, mostly due to certain limitations of Webforms: Size limitations: Webforms are only suitable for small datasets Modelling limitations: Webforms are only suitable for datasets containing a small number of orthogonal dimensions. They cannot handle datasets for which more complex data modelling is necessary. Confidentiality: Webforms cannot be used for the transmission of confidential data, as their architecture is not consistent with encryption. Moreover, Webforms can represent an obstacle to automation, as it is not possible to automate Webform transmissions

Webform trends As a result of these limitations, Excel has remained more popular than Webforms as a delivery format. Moreover, In part because of the limitations mentioned above, the general trend observed today is of a move away from Webforms: In the Pesticides domain, Webforms were phased out in favour of Excel because of confidentiality concerns. There has been no new domain using Webforms in the past several years. On the contrary, one domain completely abandoned Webforms (ENERGY – unit E5).

Alternatives to Webforms

Eurostat validation architecture Eurostat’s validation architecture now supports a wider variety of formats (SDMX-ML, SDMX-CSV, plain CSV, FLR, Excel)

SDMX-compatible Excel templates Due to the enduring popularity of Excel, Eurostat developed SDMX-compatible Excel templates. These templates address the main challenges posed by Excel (see slide 3) The templates offer a way to structure Excel files so that they can be automatically converted to a machine-readable format and processed. Thanks to Eurostat's new validation architecture, centralised corporate validation services can be used in conjunction with the templates for validation and pre-validation. Excel remains proprietary software, but the templates can be opened, edited and saved using non-proprietary tools. SDMX-compatible Excel templates can represent an alternative to Webforms!

SDMX-compatible Excel vs Webforms: advantages Cheap: reuses existing solution, no new development needed, no need for training on a specific tool. Confidentiality: The use of the SDMX-compatible Excel templates allows for encryption during transmission. All validation functionalities can be provided with encrypted data as well. Flexibility: No limitations in terms of size. Automation: SDMX-compatible templates would be one of the options available to data providers. Data providers could easily opt for SDMX-ML or (SDMX-)CSV if they wish to automate the process.

SDMX-compatible Excel vs Webforms: drawbacks Less user-friendly validation: Structural validation is not built-in: users can make mistakes if they modify the templates. Validation reports not currently as user-friendly than the ones with Webforms (and for the time being slower) Pre-filling: More difficult to replicate the pre-filling functionalities currently available with Webforms. Visibility: Previously transmitted data not visible in EDAMIS.

Next steps

Testing The alternative scenario is being tested in the ANI domain for three datasets: ANI_GIPCAT_S ANI_HATACTI_M ANI_SLAUGHT_M Based on the outcomes of these tests, a final decision on the future of Webforms in EDAMIS 4 will be taken TCOs are invited to participate in the tests (or invite their colleagues to participate). For the time being (and until a final approach is decided upon, Webforms are only available in EDAMIS3)

Questions for discussion Do TCOs have any comments on the alternative scenario proposed? Do TCOs have any preference between the two scenarios? Are there any advantages / disadvantages not mentioned? How do TCOs judge the balance between the advantages of the alternative scenario (e.g. possibility of automation and encryption) compared to the disadvantages (e.g. user-friendliness) Any suggestions on the testing?

Thank you for your attention!