1. Benchmark Methodology - Update
February 2017
©2017 Dynatrace Systems Confidential

2. ©2016 Keynote Systems Confidential
Agenda
Current Approach and Challenges
Benchmarking Evolved – Key Changes
Browser Processing
W3C
Visual Complete
Availability
Ranking Metrics
Key Questions
How are sites selected?
How are pages/paths selected?
Why the use of Chrome?
Why not multiple browsers?
Why the use of Mobile Emulation?
©2016 Keynote Systems Confidential

3. Current Approach and Challenges
Modern sites cannot be accurately measured without including Browser Processing
Dynatrace has found that many pages have up to 50% of their time spent in Browser Execution
Not surprising given extremely high amounts of 1st party and 3rd party JS overhead
Not surprising given the complexity of these web applications
Current benchmark is massively underreporting performance by excluding this processing time
There is less correlation between actual perception of performance and end to end metric
End to end is captured when the last network request has returned
Includes all third party tags, all lazy loaded content and all content loaded below the fold
Sites have so much 3rd party overhead that in most cases push out this end to end metric without any real impact on a site visitor
The current benchmark is overreporting load time by only using end to end
Multiple browsers are important for QA testing but do not necessarily reveal stark differences in performance
The goal of the benchmark is to fairly compare members and to provide a view of a “typical” experience
The goal is not to capture every possible user
To that end a single browser (especially the most widely used browser – Chrome) is a good proxy for this “typical” experience
Current Approach
End to end is the core performance metric
Browser Processing not included
Multiple browsers are used
©2016 Keynote Systems Confidential

4. Benchmarking Evolved – Key Changes
W3C (Navigation Timing) Metrics
Browser Processing
Visual Complete
Single Browser
Simpler Rankings
©2016 Keynote Systems Confidential

5. W3C (Navigation Timing) Metrics
By capturing key Navigation Timing metrics, we can slice the data much more precisely than end to end
No single metric is perfect at capturing when a page is ready but many are better than end to end
At minimum, looking at the LoadEventEnd (or commonly called onload) allows us to exclude all lazy loaded or post loaded third parties
©2016 Keynote Systems Confidential

6. ©2016 Keynote Systems Confidential
Browser Processing
The site below is a common example of the time taken in a typical page load that is tied to Browser Processing
Browser Processing in this cases means excessive execution – not allowing the network fetching/threading to continue
To be very clear, this is not parallel processing which happens for every page and site but excessive execution that serially interrupts network fetching/loading
If Browser Processing were excluded (as it is in the current benchmark data) these pages would appear to be much faster than they actually are (Home for example has more time in execution than in network/back-end)
Given the amount of processing is completely dependent on the page’s design – its impact on each page/site will be unique
©2016 Keynote Systems Confidential

7. ©2016 Keynote Systems Confidential
Visual Complete
Even more useful than the W3C metrics is a measure of above the fold render
When is the page rendered above the fold and likely ready to use
This metric is “visual complete”
Once available in Q3 this will become the primary metric for ranking in the benchmark results
Speed Index
Visually Complete
Response Time
©2016 Keynote Systems Confidential

8. ©2016 Keynote Systems Confidential
Simpler Rankings
Data should be easy to understand and the method of calculation transparent
Each benchmark will have a single set of data – no longer combining data into a “meta” ranking
Those who purchase the benchmark will have the ability to sort this data (in a CSV) based on several metrics
Part of the weekly publication
First Paint, Onload, Visual Render, End to End
Dynatrace will also rank based on what it considers to be the most accurate metric available at the time
Likely Onload first and then moving to Visual Render
Variability rankings will be removed
©2016 Keynote Systems Confidential

9. ©2016 Keynote Systems Confidential
Key Questions
How are sites selected?
Dynatrace selects sites based a mix of factors included brand presence, traffic volumes (where known), measurability etc.
Our hope is to represent the core sites in a vertical or industry
How are pages/paths selected?
Many of our benchmarks are Home Pages, as Home Pages are still a primary entry point to the site and can be indicative of overall site performance
For our transactional (multi-step) benchmarks, Dynatrace selects common and comparable sets of pages that represent the most important functionally in the measured industry
Why the use of Chrome?
Chrome continues to dominate in terms of market penetration
There are typically not major differences between browsers in terms of error and performance, allowing Chrome to be a good proxy for a “typical user”
Why not multiple browsers?
If we think about the common performance issues – back-end, network/infrastructure, third parties, core design etc – they do not vary by browser
Similar with errors where most errors are tied to infrastructure/application or third party issues and will be captured by ANY browser
There are corner cases where both errors (typically functional) or performance can be unique in a specific browser version. The best way to capture this is with Real User (RUM) data or functional testing and not with a benchmark.
Why the use of Mobile Emulation?
The goal of the benchmark is to help our customers understand how they can improve their site. By removing the high levels of errors and performance variability seen in all mobile networks we provide a accurate proxy for mobile load times without including errors and issues that are outside the site owners control.
©2016 Keynote Systems Confidential