Data collection of 2012: Data transmission standards and tools ESTP course on Waste Statistics, 24-25 April 2012 Hartmut Schrör, Eurostat hartmut.schroer@ec.europa.eu

Contents Data transmission: What is changing? What is SDMX? Why use it? Tools for SDMX compliant data transmission Webforms SDMX converter How to revise data for years 2004-2008? Reference documents and help 24/04/2012 Data transmission standards and tools

24/04/2012 Data transmission standards and tools

Data transmission: phase-out of Excel tool Outdated metadata in “Edawaste” codes used for waste, waste operations, NACE are no longer standard use of “standard code lists” in WStatR data transcoding would be necessary Labour intensive maintenance country specific regional NUTS 2 breakdown one Excel file required for each reporting country Possible incompatibility with local PC installation Version of Excel authorisation to run macros required 24/04/2012 Data transmission standards and tools

Data transmission: phase-out of Excel tool Macros for export of XML files bugs in program code => duplicates problems with special characters Prevalidations in Excel sometimes not used Member States sent Excel files rather than exported XML files => Prevalidations not performed Standard tools available for SDMX eDAMIS web forms SDMX Converter => duplication of data entry tools should be avoided 24/04/2012 Data transmission standards and tools

Contents Data transmission: What is changing? What is SDMX? Why use it? Tools for SDMX compliant data transmission Webforms SDMX converter How to revise data for years 2004-2008? Reference documents and help 24/04/2012 Data transmission standards and tools

Statistical Data and Metadata Exchange Common data transmission standard supported by Eurostat, ECB, OECD, UN, World Bank, IMF, BIS Purpose: easy exchange of statistical data by using Harmonised metadata (codelists) An open file format (XML) “SDMX-ML”: SDMX compliant XML format => different IT systems (databases, programs...) understand the same “language”. More explanations in the Manual on Waste Statistics, chapter 5 24/04/2012 Data transmission standards and tools

What is a DSD (data structure definition)?

Example of a DSD – waste codes 24/04/2012 Data transmission standards and tools

DSDs… define for each dataset the dimensions composing it e.g. GENER: nace_r2, geo, time, waste, hazard the codes and labels of all elements in a dimension the valid codes in a particular dataset describe each dataset separately (GENER, TREATM, REGIO) are noted in XML files are stored in the SDMX Registry https://webgate.ec.europa.eu/sdmxregistry 24/04/2012 Data transmission standards and tools

The past: simple “CSV” file format geo,time,waste,hazard,nace_r2,obs_value,obs_status,obs_conf BG,2010,TOTAL,TOTAL,A,7022,, BG,2010,TOTAL,TOTAL,C,,M, BG,2010,TOTAL,TOTAL,C10-C12,639,, BG,2010,TOTAL,TOTAL,C13-C15,2213,, BG,2010,TOTAL,TOTAL,C16,1234,,C BG,2010,TOTAL,TOTAL,C17_C18,4321,,C BG,2010,TOTAL,TOTAL,C19,2222,,C BG,2010,TOTAL,TOTAL,C20-C22,5432,,C BG,2010,TOTAL,TOTAL,C23,7777,,D BG,2010,TOTAL,TOTAL,C24_C25,1234,,D BG,2010,TOTAL,TOTAL,C26-C30,4741,, BG,2010,TOTAL,TOTAL,C31-C33,3560,E, BG,2010,TOTAL,TOTAL,E,3026,, BG,2010,TOTAL,TOTAL,E36_E37_E39,5090,, … 24/04/2012 Data transmission standards and tools

The present / future: SDMX-ML format 24/04/2012 Data transmission standards and tools

How to create these files? 24/04/2012 Data transmission standards and tools

Contents Data transmission: What is changing? What is SDMX? Why use it? Tools for SDMX compliant data transmission Webforms SDMX converter How to revise data for years 2004-2008? Reference documents and help 24/04/2012 Data transmission standards and tools

web forms - overview Where to find them functionality – how to enter data, etc. limitations advantages 24/04/2012 Data transmission standards and tools

Where to find the web forms Go to eDAMIS Web Portal https://webgate.ec.europa.eu/edamis logon with CIRCA user ID go to “Transmission”, “web form entry”

Where to find the web forms Select web form WASTE_GENER_A2 WASTE_TREATM_A2 WASTE_REGIO1_A2 (incineration and recovery) WASTE_REGIO2_A2 (landfills)

web form “GENER”

Web forms - functionality typing in data numeric values flags (status, confidentiality) comment on the table locking headers (like “freeze panes” in Excel) saving and re-opening data are not transmitted when saved “draft” status until web form is transmitted to Eurostat copying and pasting e.g. from Excel table layout must be identical test before use 24/04/2012 Data transmission standards and tools

Web forms - functionality Export / import using CSV format export to CSV possible editing in Excel re-import into the web form export to various other formats 24/04/2012 Data transmission standards and tools

Web forms – calculations and checks Calculations of totals performed in the web forms hazardous + non-hazardous waste (EWC-Stat items) NACE activities treatment operations No totals in REGIO tables on treatment facilities and capacities on purpose, because regional data may differ from national data sums to be entered manually 24/04/2012 Data transmission standards and tools

Web forms – checks numeric values only no negative values warning (not ‘error’) if a data cell is empty if it should be empty because the value is not available leave it empty set the “M” flag justify warning clear distinction between real zero and “missing” / “not available” 24/04/2012 Data transmission standards and tools

Why two web forms for “REGIO”? Table on facilities and capacities breakdown into NUTS 2 regions energy recovery incineration without energy recovery other recovery all landfills no NUTS 2 breakdown for types of landfills hazardous waste non-hazardous waste inert waste Difficult to display in one web form 24/04/2012 Data transmission standards and tools

Why two web forms for “REGIO”? treatment item number 1 2 3 4 Treatment categories  Energy recovery (R1) Waste incineration (D10) Recovery (R2 — R11) Landfilling (D1, D5, D12) Population served by collection 3a 3 b landfills for haz. waste landfills for non-haz waste landfills for inert waste landfills total Regions, NUTS 2 level no. of facilities capacity t/a rest capacity m³ closed % Region 1 Region 2 Region 3 … …. National total

two web forms for “REGIO” REGIO1 (all data with NUTS 2 breakdown) energy recovery incineration other recovery REGIO2 Landfills (total at NUTS 2 level, detail at national level) “population served by collection” not to be reported in web forms but in quality report coverage is not a waste operation => does not fit in any codelists used for table definition only one value 24/04/2012 Data transmission standards and tools

web forms – limitations Column width varies according to header, but cannot be changed. long header => wide column no word wrapping in header Calculated totals in GENER and TREATM cannot be modified but totals to be entered in REGIO Only one column for flags flag for “status” (estimated, etc.) and “confidentiality” combined in one column (drop-down list). Reduced set of flags to comply with SDMX standard 24/04/2012 Data transmission standards and tools

flags – valid combinations status flags M: missing E: estimated P: provisional No “R” for revised (rarely used). confidentiality C: primary confidentiality (regardless of what type) D: secondary confidentiality (hidden to protect value with C flag) No “A”, “B” flags valid: M, E, EC, ED, P, PC, PD 24/04/2012 Data transmission standards and tools

web forms – advantages Transmission of the dataset directly from the web form no file to be saved on your PC or elsewhere no separate manual transmission of the tables via eDAMIS No trouble with correct use of codes file formats local PC settings and installed software only a web browser with Java is required => SDMX-ML files are generated and sent off automatically when clicking on “official transfer”. 24/04/2012 Data transmission standards and tools

SDMX Converter – what does it do? Conversion of a text file to SDMX-ML format source: CSV target: XML file using SDMX standard 24/04/2012 Data transmission standards and tools

CSV file format geo,time,waste,hazard,nace_r2,obs_value,obs_status,obs_conf BG,2010,TOTAL,TOTAL,A,7022,, BG,2010,TOTAL,TOTAL,C,,M, BG,2010,TOTAL,TOTAL,C10-C12,639,, BG,2010,TOTAL,TOTAL,C13-C15,2213,, BG,2010,TOTAL,TOTAL,C16,1234,,C BG,2010,TOTAL,TOTAL,C17_C18,4321,,C BG,2010,TOTAL,TOTAL,C19,2222,,C BG,2010,TOTAL,TOTAL,C20-C22,5432,,C BG,2010,TOTAL,TOTAL,C23,7777,,D BG,2010,TOTAL,TOTAL,C24_C25,1234,,D BG,2010,TOTAL,TOTAL,C26-C30,4741,, BG,2010,TOTAL,TOTAL,C31-C33,3560,E, BG,2010,TOTAL,TOTAL,E,3026,, BG,2010,TOTAL,TOTAL,E36_E37_E39,5090,, … 24/04/2012 Data transmission standards and tools

SDMX-ML format 24/04/2012 Data transmission standards and tools

What is required to use SDMX converter? software “SDMX converter” + “Java Runtime Environment” standard code lists used for WStatR CSV input files using standard codes, or using other codes that are mapped to standard codes in SDMX converter DSDs for WStatR available in the SDMX Registry https://webgate.ec.europa.eu/sdmxregistry 24/04/2012 Data transmission standards and tools

Where to get SDMX converter? download the latest version (currently 2.7.2) from CIRCA http://circa.europa.eu/Public/irc/dsis/stne/library?l=/x-dis/tools/sdmx_converter Java based software and documentation install on your PC ask your local co-ordinator / IT support 24/04/2012 Data transmission standards and tools

What to do with the generated files? Send XML files to Eurostat as before via eDAMIS Send quality report along with data files 24/04/2012 Data transmission standards and tools

Contents Data transmission: What is changing? What is SDMX? Why use it? Tools for SDMX compliant data transmission Webforms SDMX converter How to revise data for years 2004-2008? Reference documents and help 24/04/2012 Data transmission standards and tools

Revisions of data 2004 to 2008 Edawaste data entry tool? pre-filled and reusable in many cases, but… recoding would be necessary for NACE (for 2004, 2006 to NACE Rev. 2) waste (mapping to to 2010 waste breakdown) all codes because of SDMX and new standard code lists Update procedure is cumbersome and may lead to errors. 24/04/2012 Data transmission standards and tools

Revisions 2004-2008: suggested solution Eurostat extracts the country / table / year from our database into Excel sends the Excel file to the country for editing Country edits the Excel table (without changing the layout) sends the Excel file back to Eurostat via eDAMIS copies and pastes the table contents back into the database 24/04/2012 Data transmission standards and tools

Contents Data transmission: What is changing? What is SDMX? Why use it? Tools for SDMX compliant data transmission Webforms SDMX converter How to revise data for years 2004-2008? Reference documents and help 24/04/2012 Data transmission standards and tools

Reference documents and help http://tinyurl.com/wstatr2012 CIRCA directory with guidance documents quality report template links to eDAMIS, SDMX Registry… Ask your local eDAMIS co-ordinator estat-support-edamis@ec.europa.eu estat-waste-statistics@ec.europa.eu 24/04/2012 Data transmission standards and tools

Thank you for your attention 24/04/2012 Data transmission standards and tools