Analysis of broadband speed study SMART 2013/0056 DG Connect European Monitoring Platform for Mapping of QoS and QoE: Meeting of Technical review panel and external experts March 1st, 2016 Hendrik Rood, senior consultant

Background of the study Background and introduction Background of the study Growing relevance of measuring broadband performance Perception of difference between advertised and actual broadband speeds Increase transparency about possible differences in performance Regulatory tool to identify and analyse possible malpractises SamKnows study commissioned by the Commission in 2011 Measurement results for March 2012, October 2013 and October 2014 October 2014 results not included in our study due to timing Many alternative platforms for measuring broadband speed Country aggregates are frequently cited in benchmarks and scholar studies Uncertainty about the differences and limitations of the other tools Commission awarded Stratix a study to analyse and explain differences SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SMART 2013/0056 Analysis of broadband speed study Background and introduction Objective and scope Collect and compare information on EU broadband speeds from various measurement approaches and explain differences Focus on fixed broadband speed measurement Some insights into mobile broadband measurement and (4G) Fixed Wireless Mainly restricted to upload and download speeds Geographic scope and periods studied EU28 and several other relevant markets (USA, Japan, China, South Korea) Period 2010-2013, with focus on SamKnows measurement periods Methodology and results Deliverables: Research report and database with measurement results SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Project approach (phases) Background and introduction Project approach (phases) Qualitative comparison of the different measurement approaches Interview with representatives of measurement platforms and a stakeholder Akamai, Ookla, RIPE Atlas, M-Labs, SamKnows, OECD Identify similarities and explain differences in methodologies Overview and limited review of initiatives from National Regulatory Authorities Collection and structuring of measurement data Selection of appropriate data sources and requesting data Compiling and structuring measurement datasets Quantitative comparison of the measurement results Analyse, identify and explain differences in measurement results Pairwise comparison using Bland-Altman Plots SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Many differentiating factors between methods Qualitative comparison Many differentiating factors between methods Measurement setup (technical implementation) Web-based / Software client / Hardware client / Platform based Measurement procedure and programming algorithms Single-threaded vs Multi-threaded (saturating the connection) Sampling strategies and sample sizes Purposive sampling / self-selection / random sampling SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Focus on four (leading) data sources Collection and of measurement data Focus on four (leading) data sources SamKnows, global leader (hardware assisted) broadband measurement Purposive sample of 9,467 devices in EU, 25% in UK, 64 million tests monthly Ookla, global leader in web-based network diagnostic applications Self-selection sample of 100 million IPs, 400 million tests monthly Akamai, world’s largest Content Distribution Network Random sample of 700+ million IPs, 100+ trillion requests quarterly M-lab, world’s largest collection of open Internet performance data Self-selection panel of 1 million IPs, 4.5 million tests (monthly) SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Country level aggregation varies considerably Analysis of methodology Country level aggregation varies considerably SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Characteristics of our data set Analysis of methodology Characteristics of our data set SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Main characteristics of our data set Dataset for quantitative analysis Main characteristics of our data set Overview of the countries included in the different datasets SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Significant differences in measurement results Quantitative analysis – first view Significant differences in measurement results *Akamai data is aggregated per quarter, data in figure is for 2013 Q4 SMART 2013/0056 Analysis of broadband speed study March 1, 2016

EU aggregated results from our datasets Quantitative analysis – first view EU aggregated results from our datasets EU-28 broadband measurements average (weighted per installed base) SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Data set report per country: example UK Quantitative analysis – first view Data set report per country: example UK Broadband speed datasets for the UK incl. speed basket weighting SMART 2013/0056 Analysis of broadband speed study March 1, 2016

All datasets covered the EU15 Quantitative analysis – first view All datasets covered the EU15 Average monthly download speeds measured in the EU15 SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Comparison with measurements for the USA Quantitative analysis – first view Comparison with measurements for the USA Average monthly download speeds measured in the US SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Deeper insight into differences in results Quantitative analysis Deeper insight into differences in results Analyse consistency of differences in results over baskets Possibly smaller differences for low headline speed categories Less influence from external factors (Wi-Fi, hardware limitations) Identify and analyse possible differences between technologies Identify and analyse effects of sampling bias for SamKnows results Attempt to weigh/adjust SamKnows results for certain countries Where detailed information on market shares is available (DK,LT,NL,PT) Analyse possible effects on country averages (per technology) Pairwise comparison of instruments using Bland-Altman Plots Known statistical technique from “medical instrument comparisons” SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Breakdown results to compare like with like Quantitative analysis - weighting Breakdown results to compare like with like Create comparable subselections (‘baskets’) Based on headline speeds and broadband technology Possibility to weigh samples based on market shares Such level of detail is often not available User information is limited for most alternative data sources No detailed information on market shares per basket (for all countries) Company confidentiality issues at national or regional (NUTS-2) level SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SMART 2013/0056 Analysis of broadband speed study Quantitative analysis - weighting Proportional difference between observed and expected sample frequency for SamKnows’ dataset Results per download speed category per country SamKnows study has bias due to a self-selecting volunteer panel March 2012 November 2013 SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SMART 2013/0056 Analysis of broadband speed study Quantitative analysis - weighting Bias towards higher bitrate panel observed for all measurement methods with headline speed Ookla / Speedtest has a bias due to self-selection effect SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SamKnows and Ookla allow for weighting on headline speed baskets Scatter plots for SamKnows (left) and Ookla (right) of the relative standard deviation in download measurements over time in relation to sample size SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SMART 2013/0056 Analysis of broadband speed study Quantitative analysis - weighting Weighting SamKnows data on speed had more impact than correcting for access technology Denmark Lithuania Netherlands Portugal SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Comparisons made using Bland-Altman plots Quantitative analysis – comparisons Comparisons made using Bland-Altman plots Main findings: The weighted measurement speeds for Ookla are structurally lower than for the SamKnows study. This can be explained as an effect of the limitations induced by a software/web-based approach. Akamai’s “peak average download speed” per country appears to be consistently higher than weighted country averages from SamKnows and Ookla. Compared to Samknows weighted country averages there is an (unexpected) absolute offset of 10 Mbps on average Akamai’s “average download speed” per country is significantly lower than Ookla and SamKnows’ weighted country averages. M-Lab measurement results are consistently lower than either SamKnows, Ookla or Akamai. This can largely be explained from M-lab’s divergent measurement methodology using single-threading and including TCP slow-start. SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SMART 2013/0056 Analysis of broadband speed study SamKnows effect of weighting by headline download speed category shows bias Linear Log Bland-Altman plots for pairwise comparisons SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SMART 2013/0056 Analysis of broadband speed study Quantitative analysis – comparisons Ookla effect of weighting by headline download speed category shows bias Linear Log Bland-Altman plots for pairwise comparisons SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SMART 2013/0056 Analysis of broadband speed study Quantitative analysis – comparisons SamKnows vs Ookla download paired by headline download speed: bias downward Ookla Linear Log SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SMART 2013/0056 Analysis of broadband speed study Quantitative analysis – comparisons SamKnows vs Ookla download paired by headline download speed: bias downward Ookla Linear Log SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SamKnows vs Ookla DL paired by (headline) download speed category Quantitative analysis – comparisons SamKnows vs Ookla DL paired by (headline) download speed category Linear Log SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SamKnows vs Ookla survey download weighted paired by country: Quantitative analysis – comparisons SamKnows vs Ookla survey download weighted paired by country: Linear Log More agreement but statistical datasets become much smaller SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SamKnows vs Akamai peak download paired by country Quantitative analysis – comparisons SamKnows vs Akamai peak download paired by country Linear Log Akamai peak download gives higher results for lowspeed countries SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SamKnows weighted vs Akamai average download paired by country Quantitative analysis – comparisons SamKnows weighted vs Akamai average download paired by country Linear Log Akamai average download shows effect of start-up of transmission SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SamKnows vs Mlab download paired by country Quantitative analysis – comparisons SamKnows vs Mlab download paired by country Linear Log Mlab results diverge considerable from other measurement platforms SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SMART 2013/0056 Analysis of broadband speed study Conclusions and further research Conclusions overall Pairwise comparison shows significant correlation, but differences are still substantive. Even after weighting results do not converge strongly Lower than expected software vs hardware outcomes does not explain sufficiently the divergence found Small panel/sample sizes (per country/region) Discrepancies in panel composition Strong recommendation: sufficient size panel is needed Either a-select drawn or enough information for appropriate weighting Unweighted aggregates are at best rough approximations Most unweighted measurements show bias to higher speeds due to panel For use as a policy instrument it is better to focus on weighted data SMART 2013/0056 Analysis of broadband speed study March 1, 2016

SMART 2013/0056 Analysis of broadband speed study Conclusions and further research Further research In this study we only compared aggregated results various “instruments” did not measure the exact same pool of connections Beter understanding by a detailed comparison comparing measurements on a like-for-like base for instance per IP-address, for all relevant measurement methods Such an approach would exclude effects from panel bias and aggregation methods provide a better insight into the actual divergence in particular for M-lab and Akamai in particular. SMART 2013/0056 Analysis of broadband speed study March 1, 2016

Any Questions? Hendrik.Rood@stratix.nl