Session 4 – Data collection

TUV + external speakers Agenda Data collection TUV + external speakers 13:30 – 15:00 Presentation on methodology and overview of test data used Contractor TÜV Rheinland Ms. van Zijverden and Mr. Hafner Practice report of test data suppliers Mr. Delannoy, FRA Ministry, Mission France Très Haut Debit Mr. Flaviano, ITA NRA AGCOM Mr. van Ostaede, Belgian Consumer Organisation Test Achat Survey and discussion All

Data collection approach Display on the platform Data collected from whom? Data suppliers NRAs Ministries Private crowdsourcing providers How is data collected Various data formats Flexible collection process What data is collected? Spatial resolution / geometry Attributes and meta data

Test data collection Test data campaign in April / May 2016 Data supplier provided data in own format Data supplied EC SMART 2010/0036 EC SMART 2013/0054 France Germany Italy Poland Romania Slovenia (via AKOStest/netTest) UK (open data) Akamai netBravo Netradar Open data Austria (via netTest) Czech Republic (via netMetr/netTest) Ookla (only open data sets) M-Lab More test data delivery is discussed currently MONROE, M-Lab, Cedexis, OpenSignal Provided by: Germany, France, EC/IHS (SMART 2013/0054) Open data: UK Awaiting approval: Poland, Slovenia Provided by: EC/SamKnows (SMART 2010/0036), Italy Awaiting delivery: Germany, MONROE Awaiting approval: RIPE Atlas Provided by: Italy (fixed), Romania, Akamai, Netradar, netBravo, Slovenia (via AKOStest/netTest) Open data: Austria (via netTest), Czech Republic (via netMetr/netTest), Ookla (only open data sets) Awaiting delivery: OpenSignal, Italy (mobile) Awaiting approval: M-Lab, Cedexis

Data collection process First data collection campaign in 2016 Data supplier provides data in own format Contractor structures data according to data model Contractor sends structured data plus explanation back to data supplier Minimum requirement because of data privacy : Data aggregated to one of the defined spatial resolution levels (at least address or grid level) NOTE: if not available, we support data supplier in aggregating raw data

Data collection process Data model will be amended if practice experience from first collection campaign shows need to do so Adapt data model (if and where necessary) Presentation at 2nd Stakeholder Consultation Workshop

Data collection process Data collection campaigns as of second year of development phase (2017) Data suppliers provide data according to approved data model Data suppliers provides data according to data model Data suppliers‘ data is already aggregated similarly to data model and can be provided in original structure 2 alternatives Feedback following each data provision Data suppliers are given feedback, ensuring a smooth process Qualification process: Feedback and (where appropriate) training as qualification process for self-reliable data supply

Geometry – Level of spatial resolution Small regions (NUTS 3) Grid 1Km Higher resolved grids Addresses / Points Collected Small regions (NUTS 3) Public Depending on Memorandum of Understanding Visualised Small regions (NUTS 3) Grid 1Km Higher resolved grids Addresses / Points Expert

Data formats – simplified templates to collect all kind of data sets linked to geometry We ensure a user-friendly process with little data storage requirements Geo-data Tables / Text shp wfs gml tbd tbd xls csv or

Our data model: Thousands of values can be filled in per data set – no raw data Initiative NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Name ID 1 Group: Wired, Wireless,. Availability: Inhabitants, Area.. Infrastruc-ture All time / Unknown All/ Unknown All/ Unknown X 2 Single: LTE/4G, CAT4,.. Takeup Speed down Working Days LAN Operator (physical) 3 Measure-ment Speed up Weekends WLAN Operator (virtual) Latency… Day Peak... Mobile

Our data model: Thousands of values can be filled in per data set – no raw data Initiative NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Name ID 1 Group: Wired, Wireless,. Availability: Inhabitants, Area.. Infrastruc-ture All time / Unknown All/ Unknown All/ Unknown X 2 Single: LTE/4G, CAT4,.. Takeup Speed down Working Days LAN Operator (physical) 3 Measure-ment Speed up Weekends WLAN Operator (virtual) Latency… Day Peak... Mobile

Our data model: Thousands of values can be filled in per data set – no raw data Initiative NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Name ID 1 Group: Wired, Wireless,. Availability: Inhabitants, Area.. Infrastruc-ture All time / Unknown All/ Unknown All/ Unknown X 2 Single: LTE/4G, CAT4,.. Takeup Speed down Working Days LAN Operator (physical) 3 Measure-ment Speed up Weekends WLAN Operator (virtual) Latency… Day Peak... Mobile

Our data model: Thousands of values can be filled in per data set – no raw data Initiative NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Name ID 1 Group: Wired, Wireless,. Availability: Inhabitants, Area.. Infrastruc-ture All time / Unknown All/ Unknown All/ Unknown X 2 Single: LTE/4G, CAT4,.. Takeup Speed down Working Days LAN Operator (physical) 3 Measure-ment Speed up Weekends WLAN Operator (virtual) Latency… Day Peak... Mobile

Our data model: Thousands of values can be filled in per data set – no raw data Initiative NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Name ID 1 Group: Wired, Wireless,. Availability: Inhabitants, Area.. Infrastruc-ture All time / Unknown All/ Unknown All/ Unknown X 2 Single: LTE/4G, CAT4,.. Takeup Speed down Working Days LAN Operator (physical) 3 Measure-ment Speed up Weekends WLAN Operator (virtual) Latency… Day Peak... Mobile

Our data model: Thousands of values can be filled in per data set – no raw data Initiative NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Name ID 1 Group: Wired, Wireless,. Availability: Inhabitants, Area.. Infrastruc-ture All time / Unknown All/ Unknown All/ Unknown X 2 Single: LTE/4G, CAT4,.. Takeup Speed down Working Days LAN Operator (physical) 3 Measure-ment Speed up Weekends WLAN Operator (virtual) Latency… Day Peak... Mobile

Example QoS 3: Initiative from Italy „MisuraInternet“ NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Misura- Internet ITH55 Single measurements (140.000) with link to NUTS 3 Region

MisuraInternet initiative provides data in category QoS 3 practice experienced NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Misura- Internet ITH55 3

Italian initiative supplied data on single technology NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Misura- Internet ITH55 3 Single: DSL/ADSL nly Speed Down Speed Up Latency Data can be provided for single technologies or technology group

Italian initiative supplied data on measurement only NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Misura- Internet ITH55 3 Single: DSL/ADSL Measure-ment Only Speed Down Speed Up Latency Additional indicators refer to measurement only or measurement compared to contracted speed or availability linked to households, addresses, population etc.

Italian initiative supplied data on three selected quality criteria NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Misura- Internet ITH55 3 Single: DSL/ADSL Measure-ment Only Speed Down Speed Up Latency

Additional values used for meta data, not in the case for this test data set Initiative NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Misura- Internet ITH55 3 Single: DSL/ADSL Measure-ment Only Speed Down All time / Unknown All / Unknown Speed Up Latency More information can refer to points of time of QoS, customer technology (wired, wireless) and operators (physical and virtual)

Result of combinations for data supplied by MisuraInternet Initiative NUTS / GRID ID QoS Type Techno-logy Internet Access Provider Additional indicator Quality criteria Time Techno-logy Customer End User Operator Result of combina-tions Misura- Internet ITH55 3 Single: DSL/ADSL Measure-ment Only Speed Down All time/ Unknown All/ X Speed Up Latency Speed Down Min [Mbit/s] Max Average [Mbit/s] Median Number of measure-ments 1,5 51,32 20,57 13 26

Full picture: Thousands of combinations of values can be collected – no raw data Initiative NUTS / GRID ID QoS Type Technology Internet Access Provider Additional indicator Quality criteria Time Technology Customer End User Operator Result of combinations Name ID 1 Group: All/Unknown Availability Households Infrastructure All time / Unknown All/ Unknown All/Unknown X 2 Group: Wired Availability Inhabitants Speed Down Working Days LAN Operator (physical) 3 Group: Wireless Availability Area Speed Up Weekends WLAN … Group: Mobile Availability Addresses Latency Day Peak Mobile Group: NGA Availability Roads Jitter Day Non peak Operator (virtual) Single: DSL/ADSL Take-up Packet loss Single: CATV Measurement Only Data Usage Single: FTTC/VDSL Measurement Comparison Single: FTTH/B Single: UMTS/3G Single: LTE/4G Single: 2G Single: WiMAX/WLAN Single: Satellite No expectation to receive data for all attributes

Full picture: Thousands of combinations of values can be collected – no raw data Initiative NUTS / GRID ID QoS Type Technology Internet Access Provider Additional indicator Quality criteria Time Technology Customer End User Operator Result of combinations Name ID 1 Group: All/Unknown Availability Households Infrastructure All time / Unknown All/ Unknown All/Unknown X 2 Group: Wired Availability Inhabitants Speed Down Working Days LAN Operator (physical) 3 Group: Wireless Availability Area Speed Up Weekends WLAN … Group: Mobile Availability Addresses Latency Day Peak Mobile Group: NGA Availability Roads Jitter Day Non peak Operator (virtual) Single: DSL/ADSL Take-up Packet loss Single: CATV Measurement Only Data Usage Single: FTTC/VDSL Measurement Comparison Single: FTTH/B Single: UMTS/3G Single: LTE/4G Single: 2G Single: WiMAX/WLAN Single: Satellite Data model is compromise between completeness and user-friendliness No expectation to receive data for all attributes

Content – Meta data what we collect Meta data has to be supplied in order to assure similar representation of data according to the data supplier’s measurement focus, published evaluation, and to avoid misinterpretation of data Mapping initiative approach / focus Focus of supplied data Data processing – raw data Spatial resolution of raw data Location accuracy (assessment of accuracy) Filtering of insufficient values (e.g. multiple measurements, values limited by equipment or contract…) Data processing – aggregation rules Completeness of data / samples size Spatial distribution of data Inclusion of all / some ISPs Reference to challenges or potential misinterpretations of own data set

What do we do with data to provide INSPIRE conform meta data? Delivered meta data is converted by contractor into INSPIRE compliant format for data feeds Identification Mapping initiative approach Focus of supplied data Quality & Validity Data processing Challenges / interpretation Temporal Temporal extent of data Constraints Conditions on usage and access to data

Back-up

Example test data Italy Supplied data What was supplied? List of single measurements linked to NUTS 3 regions Values on latency, download and upload Speed Corporate Presentation

Example test data Italy Data processing / aggregation (example latency) Basic data Calculated value groups Step 1: Classification of single measurement values into value groups E.g. first row (ID 690) --- Value delay 93,4 ms --- Classification in groups Latency < 500 ms and < 100 ms Corporate Presentation

Example test data Italy Data processing / aggregation (example latency) Step 2: Aggregation of all Measurements in a NUTS 3 region and calculation of statistical values and percentage of measurements in value groups Corporate Presentation

What is NUTS 3 level? Unterallgäu NUTS 3 DE27C Schwaben NUTS 2 DE27 The current NUTS 2013 classification is valid from 1 January 2015 and lists 98 regions at NUTS 1, 276 regions at NUTS 2 and 1342 regions at NUTS 3 level Unterallgäu NUTS 3 DE27C Schwaben NUTS 2 DE27 Bayern NUTS 1 DE2 Germany NUTS 0 DE

Why do we use NUTS 3 level? NUTS 3 NUTS 3 level as compromise between requirements on comparability and low effort for data suppliers Comparability Convenience European-wide statistical standard One common comparison level Possibility to link to other statistical values Manageable number of polygons to be provided Possibility to get a quick and clear overview User can refer to NUTS 3 as spatial unit, in contrast to grids NUTS 3

Groups are defined by data supplier Result of combinations Min Max Average Median Group: 1,2,n,.. No of Measure- ments No of Operators (physical) Operators (virtual) Groups are defined according to the quality criteria Group 1 could refer to the speed e.g. „≥ 1 Mbit/s“ or to latency e.g. „100 – 500 ms" Values are percentage of available units or measurement results within a group Example: Download speed in Mbit/s 1 2 8 10 16 20 100 50 25 30 Group z (≥ 50 Mbit/s) Group y (30 - 50 Mbit/s) Group x (≥ 1 Mbit/s)