Presentation is loading. Please wait.

Presentation is loading. Please wait.

Analysis of broadband speed study SMART 2013/0056

Published byBertram Harper Modified over 7 years ago

Similar presentations

Presentation on theme: "Analysis of broadband speed study SMART 2013/0056"— Presentation transcript:

1 Analysis of broadband speed study SMART 2013/0056
DG Connect European Monitoring Platform for Mapping of QoS and QoE: Meeting of Technical review panel and external experts March 1st, 2016 Hendrik Rood, senior consultant

2 Background of the study
Background and introduction Background of the study Growing relevance of measuring broadband performance Perception of difference between advertised and actual broadband speeds Increase transparency about possible differences in performance Regulatory tool to identify and analyse possible malpractises SamKnows study commissioned by the Commission in 2011 Measurement results for March 2012, October 2013 and October 2014 October 2014 results not included in our study due to timing Many alternative platforms for measuring broadband speed Country aggregates are frequently cited in benchmarks and scholar studies Uncertainty about the differences and limitations of the other tools Commission awarded Stratix a study to analyse and explain differences SMART 2013/0056 Analysis of broadband speed study March 1, 2016

3 SMART 2013/0056 Analysis of broadband speed study
Background and introduction Objective and scope Collect and compare information on EU broadband speeds from various measurement approaches and explain differences Focus on fixed broadband speed measurement Some insights into mobile broadband measurement and (4G) Fixed Wireless Mainly restricted to upload and download speeds Geographic scope and periods studied EU28 and several other relevant markets (USA, Japan, China, South Korea) Period , with focus on SamKnows measurement periods Methodology and results Deliverables: Research report and database with measurement results SMART 2013/0056 Analysis of broadband speed study March 1, 2016

4 Project approach (phases)
Background and introduction Project approach (phases) Qualitative comparison of the different measurement approaches Interview with representatives of measurement platforms and a stakeholder Akamai, Ookla, RIPE Atlas, M-Labs, SamKnows, OECD Identify similarities and explain differences in methodologies Overview and limited review of initiatives from National Regulatory Authorities Collection and structuring of measurement data Selection of appropriate data sources and requesting data Compiling and structuring measurement datasets Quantitative comparison of the measurement results Analyse, identify and explain differences in measurement results Pairwise comparison using Bland-Altman Plots SMART 2013/0056 Analysis of broadband speed study March 1, 2016

5 Many differentiating factors between methods
Qualitative comparison Many differentiating factors between methods Measurement setup (technical implementation) Web-based / Software client / Hardware client / Platform based Measurement procedure and programming algorithms Single-threaded vs Multi-threaded (saturating the connection) Sampling strategies and sample sizes Purposive sampling / self-selection / random sampling SMART 2013/0056 Analysis of broadband speed study March 1, 2016

6 Focus on four (leading) data sources
Collection and of measurement data Focus on four (leading) data sources SamKnows, global leader (hardware assisted) broadband measurement Purposive sample of 9,467 devices in EU, 25% in UK, 64 million tests monthly Ookla, global leader in web-based network diagnostic applications Self-selection sample of 100 million IPs, 400 million tests monthly Akamai, world’s largest Content Distribution Network Random sample of 700+ million IPs, 100+ trillion requests quarterly M-lab, world’s largest collection of open Internet performance data Self-selection panel of 1 million IPs, 4.5 million tests (monthly) SMART 2013/0056 Analysis of broadband speed study March 1, 2016

7 Country level aggregation varies considerably
Analysis of methodology Country level aggregation varies considerably SMART 2013/0056 Analysis of broadband speed study March 1, 2016

8 Characteristics of our data set
Analysis of methodology Characteristics of our data set SMART 2013/0056 Analysis of broadband speed study March 1, 2016

9 Main characteristics of our data set
Dataset for quantitative analysis Main characteristics of our data set Overview of the countries included in the different datasets SMART 2013/0056 Analysis of broadband speed study March 1, 2016

10 Significant differences in measurement results
Quantitative analysis – first view Significant differences in measurement results *Akamai data is aggregated per quarter, data in figure is for 2013 Q4 SMART 2013/0056 Analysis of broadband speed study March 1, 2016

11 EU aggregated results from our datasets
Quantitative analysis – first view EU aggregated results from our datasets EU-28 broadband measurements average (weighted per installed base) SMART 2013/0056 Analysis of broadband speed study March 1, 2016

12 Data set report per country: example UK
Quantitative analysis – first view Data set report per country: example UK Broadband speed datasets for the UK incl. speed basket weighting SMART 2013/0056 Analysis of broadband speed study March 1, 2016

13 All datasets covered the EU15
Quantitative analysis – first view All datasets covered the EU15 Average monthly download speeds measured in the EU15 SMART 2013/0056 Analysis of broadband speed study March 1, 2016

14 Comparison with measurements for the USA
Quantitative analysis – first view Comparison with measurements for the USA Average monthly download speeds measured in the US SMART 2013/0056 Analysis of broadband speed study March 1, 2016

15 Deeper insight into differences in results
Quantitative analysis Deeper insight into differences in results Analyse consistency of differences in results over baskets Possibly smaller differences for low headline speed categories Less influence from external factors (Wi-Fi, hardware limitations) Identify and analyse possible differences between technologies Identify and analyse effects of sampling bias for SamKnows results Attempt to weigh/adjust SamKnows results for certain countries Where detailed information on market shares is available (DK,LT,NL,PT) Analyse possible effects on country averages (per technology) Pairwise comparison of instruments using Bland-Altman Plots Known statistical technique from “medical instrument comparisons” SMART 2013/0056 Analysis of broadband speed study March 1, 2016

16 Breakdown results to compare like with like
Quantitative analysis - weighting Breakdown results to compare like with like Create comparable subselections (‘baskets’) Based on headline speeds and broadband technology Possibility to weigh samples based on market shares Such level of detail is often not available User information is limited for most alternative data sources No detailed information on market shares per basket (for all countries) Company confidentiality issues at national or regional (NUTS-2) level SMART 2013/0056 Analysis of broadband speed study March 1, 2016

17 SMART 2013/0056 Analysis of broadband speed study
Quantitative analysis - weighting Proportional difference between observed and expected sample frequency for SamKnows’ dataset Results per download speed category per country SamKnows study has bias due to a self-selecting volunteer panel March November 2013 SMART 2013/0056 Analysis of broadband speed study March 1, 2016

18 SMART 2013/0056 Analysis of broadband speed study
Quantitative analysis - weighting Bias towards higher bitrate panel observed for all measurement methods with headline speed Ookla / Speedtest has a bias due to self-selection effect SMART 2013/0056 Analysis of broadband speed study March 1, 2016

19 SamKnows and Ookla allow for weighting on headline speed baskets
Scatter plots for SamKnows (left) and Ookla (right) of the relative standard deviation in download measurements over time in relation to sample size SMART 2013/0056 Analysis of broadband speed study March 1, 2016

20 SMART 2013/0056 Analysis of broadband speed study
Quantitative analysis - weighting Weighting SamKnows data on speed had more impact than correcting for access technology Denmark Lithuania Netherlands Portugal SMART 2013/0056 Analysis of broadband speed study March 1, 2016

21 Comparisons made using Bland-Altman plots
Quantitative analysis – comparisons Comparisons made using Bland-Altman plots Main findings: The weighted measurement speeds for Ookla are structurally lower than for the SamKnows study. This can be explained as an effect of the limitations induced by a software/web-based approach. Akamai’s “peak average download speed” per country appears to be consistently higher than weighted country averages from SamKnows and Ookla. Compared to Samknows weighted country averages there is an (unexpected) absolute offset of 10 Mbps on average Akamai’s “average download speed” per country is significantly lower than Ookla and SamKnows’ weighted country averages. M-Lab measurement results are consistently lower than either SamKnows, Ookla or Akamai. This can largely be explained from M-lab’s divergent measurement methodology using single-threading and including TCP slow-start. SMART 2013/0056 Analysis of broadband speed study March 1, 2016

22 SMART 2013/0056 Analysis of broadband speed study
SamKnows effect of weighting by headline download speed category shows bias Linear Log Bland-Altman plots for pairwise comparisons SMART 2013/0056 Analysis of broadband speed study March 1, 2016

23 SMART 2013/0056 Analysis of broadband speed study
Quantitative analysis – comparisons Ookla effect of weighting by headline download speed category shows bias Linear Log Bland-Altman plots for pairwise comparisons SMART 2013/0056 Analysis of broadband speed study March 1, 2016

24 SMART 2013/0056 Analysis of broadband speed study
Quantitative analysis – comparisons SamKnows vs Ookla download paired by headline download speed: bias downward Ookla Linear Log SMART 2013/0056 Analysis of broadband speed study March 1, 2016

25 SMART 2013/0056 Analysis of broadband speed study
Quantitative analysis – comparisons SamKnows vs Ookla download paired by headline download speed: bias downward Ookla Linear Log SMART 2013/0056 Analysis of broadband speed study March 1, 2016

26 SamKnows vs Ookla DL paired by (headline) download speed category
Quantitative analysis – comparisons SamKnows vs Ookla DL paired by (headline) download speed category Linear Log SMART 2013/0056 Analysis of broadband speed study March 1, 2016

27 SamKnows vs Ookla survey download weighted paired by country:
Quantitative analysis – comparisons SamKnows vs Ookla survey download weighted paired by country: Linear Log More agreement but statistical datasets become much smaller SMART 2013/0056 Analysis of broadband speed study March 1, 2016

28 SamKnows vs Akamai peak download paired by country
Quantitative analysis – comparisons SamKnows vs Akamai peak download paired by country Linear Log Akamai peak download gives higher results for lowspeed countries SMART 2013/0056 Analysis of broadband speed study March 1, 2016

29 SamKnows weighted vs Akamai average download paired by country
Quantitative analysis – comparisons SamKnows weighted vs Akamai average download paired by country Linear Log Akamai average download shows effect of start-up of transmission SMART 2013/0056 Analysis of broadband speed study March 1, 2016

30 SamKnows vs Mlab download paired by country
Quantitative analysis – comparisons SamKnows vs Mlab download paired by country Linear Log Mlab results diverge considerable from other measurement platforms SMART 2013/0056 Analysis of broadband speed study March 1, 2016

31 SMART 2013/0056 Analysis of broadband speed study
Conclusions and further research Conclusions overall Pairwise comparison shows significant correlation, but differences are still substantive. Even after weighting results do not converge strongly Lower than expected software vs hardware outcomes does not explain sufficiently the divergence found Small panel/sample sizes (per country/region) Discrepancies in panel composition Strong recommendation: sufficient size panel is needed Either a-select drawn or enough information for appropriate weighting Unweighted aggregates are at best rough approximations Most unweighted measurements show bias to higher speeds due to panel For use as a policy instrument it is better to focus on weighted data SMART 2013/0056 Analysis of broadband speed study March 1, 2016

32 SMART 2013/0056 Analysis of broadband speed study
Conclusions and further research Further research In this study we only compared aggregated results various “instruments” did not measure the exact same pool of connections Beter understanding by a detailed comparison comparing measurements on a like-for-like base for instance per IP-address, for all relevant measurement methods Such an approach would exclude effects from panel bias and aggregation methods provide a better insight into the actual divergence in particular for M-lab and Akamai in particular. SMART 2013/0056 Analysis of broadband speed study March 1, 2016

33 Any Questions? Hendrik.Rood@stratix.nl

Download ppt "Analysis of broadband speed study SMART 2013/0056"

Similar presentations

Original Figures for Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring

Original Figures for "Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring"
The quality framework of European statistics by the ESCB Quality Conference Vienna, 3 June 2014 Aurel Schubert 1) European Central Bank 1) This presentation.

The quality framework of European statistics by the ESCB Quality Conference Vienna, 3 June 2014 Aurel Schubert 1) European Central Bank 1) This presentation.
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Catherine Trapani Educational Testing Service ECOLT: October.

Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Catherine Trapani Educational Testing Service ECOLT: October.
Q.Int_speed_test Draft Recommendation Eva Ibarrola SG11 Q15 Acting Rapporteur JCA-CIT meeting (electronic meeting)

Q.Int_speed_test Draft Recommendation Eva Ibarrola SG11 Q15 Acting Rapporteur JCA-CIT meeting (electronic meeting)
Title slide PIPELINE QRA SEMINAR. PIPELINE RISK ASSESSMENT INTRODUCTION TO GENERAL RISK MANAGEMENT 2.

Title slide PIPELINE QRA SEMINAR. PIPELINE RISK ASSESSMENT INTRODUCTION TO GENERAL RISK MANAGEMENT 2.
Knowledge is Power Marketing Information System (MIS) determines what information managers need and then gathers, sorts, analyzes, stores, and distributes.

Knowledge is Power Marketing Information System (MIS) determines what information managers need and then gathers, sorts, analyzes, stores, and distributes.
Measuring the experience consumers have when using broadband services Tim Gilfedder Technical Advisor 3 rd July 2015.

Measuring the experience consumers have when using broadband services Tim Gilfedder Technical Advisor 3 rd July 2015.
Energy Efficiency Benchmarking for Mobile Networks

Energy Efficiency Benchmarking for Mobile Networks
What is Business Analysis Planning & Monitoring?

What is Business Analysis Planning & Monitoring?
Key Elements for electronic Local Authorities’ Networks Cagliari, 27 th May 2002 Walter Wenzel CEO of KEeLAN project.

Key Elements for electronic Local Authorities’ Networks Cagliari, 27 th May 2002 Walter Wenzel CEO of KEeLAN project.
South Carolina Oyster Restoration and Enhancement Water Quality Monitoring: Evaluation of a Digital Training Product for South Carolina Oyster Restoration.

South Carolina Oyster Restoration and Enhancement Water Quality Monitoring: Evaluation of a Digital Training Product for South Carolina Oyster Restoration.
The Effect of Computers on Student Writing: A Meta-Analysis of Studies from 1992 to 2002 Amie Goldberg, Michael Russell, & Abigail Cook Technology and.

The Effect of Computers on Student Writing: A Meta-Analysis of Studies from 1992 to 2002 Amie Goldberg, Michael Russell, & Abigail Cook Technology and.
Question Four: Project Risk Management PMBOK definition of Project Risk Project risk management is the art and science of identifying, analyzing, and responding.

Question Four: Project Risk Management PMBOK definition of Project Risk Project risk management is the art and science of identifying, analyzing, and responding.
Telecoms Review Institutional aspects European Parliament, 5 March 2008.

Telecoms Review Institutional aspects European Parliament, 5 March 2008.
Impact of e-commerce on the UK economy Cecil Prescott Head of Research& Development and Information Technology, ONS #UKecomStats.

Impact of e-commerce on the UK economy Cecil Prescott Head of Research& Development and Information Technology, ONS #UKecomStats.
June 2009 Regulation on pesticide statistics Pierre NADIN ESTAT E1- Farms, agro-environment and rural development

June 2009 Regulation on pesticide statistics Pierre NADIN ESTAT E1- Farms, agro-environment and rural development
Workshop on Passenger Mobility Conclusions. EU data requirements – DG MOVE  Environmental, economic and social considerations require close monitoring.

Workshop on Passenger Mobility Conclusions. EU data requirements – DG MOVE  Environmental, economic and social considerations require close monitoring.
Creating the environment for business Assessment of the Implementation by the Member States of the IPPC Directive Advisory Group Meeting Friday 13 th January.

Creating the environment for business Assessment of the Implementation by the Member States of the IPPC Directive Advisory Group Meeting Friday 13 th January.
State aid for broadband Internet development in Lithuania TAIEX Multi-beneficiary Workshop on TAIEX Assistance within Chapter 10 – Information society.

State aid for broadband Internet development in Lithuania TAIEX Multi-beneficiary Workshop on TAIEX Assistance within Chapter 10 – Information society.
Methodology of the European Commission’s project “Mapping study (phase II): Mapping of Broadband Services in Europe - SMART 2014/0016” Christiane Lehmann,

Methodology of the European Commission’s project “Mapping study (phase II): Mapping of Broadband Services in Europe - SMART 2014/0016” Christiane Lehmann,

Similar presentations

Ads by Google