1. Marquee Video Integration A/B test analysis
4/16/ /29/2015

2. Agenda Executive Summary Testing Overview Marquee on New Car Path
Marquee on Owners Path
Revenue discussion
Key Takeaways & Recommendations

3. Executive Summary
Marquee video ads promotes high click through to OEM site on the New Car Path, even though viewer completion rate is low
Significant positive impact on click through rate on New Car path products, as well as all products on the site level
Although most metrics did not reach statistical significance, directionally speaking we see small decreases in repeat visitation, and increased page load time, but overall revenue is directionally positive
Impact on traffic metric and revenue impact differ by creative; Scion creative performed better than Cadillac
Performance on Owners Path is less favorable

4. A note on “Statistical Significance”
What does it mean?
It means we are confident that the change in metrics was not due to random fluctuation
ie. When shooting free throws, in one instance I score 5 out 10 on my left hand, and 3 out 10 on my right hand, am I really a better left-hand shooter? Or did I just lucked out?
Solution: shoot many sets of 20 shots (10 each hand) and record down the data, and compare my FT % of each set
If my FT % of left hand shooting is tested statistically significant better than my right hand, it means I am confident that the different did not happen randomly. I am really a better left hand shooter
What if something is not “statistically significant”?
2 scenarios:
1) we have not collected enough samples
In this case, we consider the metric is “directionally helpful”
2) we have collected enough samples, but did not show significant changes
In this case, we claim there is no change,
In this analysis, many changes are very small, and therefore did not reach significance.
However, many of the metrics still provide valuable information, so we would like to discuss those. Here is how they are noted.
Statistically significant change will be denoted in green and red
ie. -3.5% or +25%
All directionally helpful changes will be stated in black. Ie. +2.5pp, -1.3%
For the metrics that we had enough sample for, but did not show significant change, we’ve zero-ed them out to be no change
Please see the Appendix for a Metric Summary Sheet

5. Product and Test Overview

6. Marquee Video Introduction
For Sample-
The Marquee Video is designed to be highly impactful, branded video unit that auto-animates down the entire path after make/model is known, including the Hub page.
Call to action: “Click to Play”
Upon user-initiation, the full video can be viewed; however the intent is to immerse shoppers in a branded experience to influence which vehicles they will consider.
Call to action: buttons clicking offsite

7. Test Overview
3 tests are running simultaneously, and mutually exclusive
Only available for desktop users
Test 1 – Cadillac NC Model Path
50% Control: New Car Model Leaderboard running as is
50% Challenger: Marque Video running in place of NC Model LB
Test 2 – Scion NC Model Path
Test 3 – Scion Owners Model Path
50% Control: Owners Model Leaderboard running as is
50% Challenger: Marque Video running in place of Owners Model LB

8. Testing Objective A note on the traffic level
Understand impact of Marquee to KBB.com
Where do consumers go after going down the pricing path (more research, directly to RET page)?
Is there drop-off in conversion down the path?
What are the impacts to load time?
What is impact to impression generation down the path?
Understand impact to other ads on the page
Key performance indicators include ad impressions, ad CTR, ad revenue
A note on the traffic level
This test only involved a very small amount of traffic vs. site overall, and is only representative for the brands tested
Item
Visits
% of Total Site
Site Overall Traffic
10,361,117
Visit involved in Cadillac NC Test
13,421
0.13%
Visit involved in Scion NC Test
4,921
0.05%
Visit involved in Scion Owners Test
19,376
0.19%
Total Visits involved in 3 tests
37,718
0.36%

9. Overall Traffic and Ad Impacts - New Car Path
4/16/ /29/2015

10. New Car Path – Summary with directional read
The Good
Click-to-Play and CTR on Marquee is higher than industry benchmark
Increased time on site vs. control
Increased % visits returning to a NC section on subsequent visits
Increased Ad CTR both on NC Path and on overall site level
The Bad
Video completion rate is lower than industry benchmark
Decreased repeat visitation
Decreased overall PV/Visit, and PV/Visitor for Cadillac
Decreased % of traffic and PV/Visit to UC Shopper and Owner for Cadillac
Difference & Uncertainty
Cadillac video had much higher CTR than Scion
Possible technical issue impacting CTR of Scion Video
Higher completion rates of Scion Videos
Content of the video might dictate impact on traffic and browsing behavior
Increased PV/Visit for Scion, driven by high PV/Visit on UC Shopper and Owners

11. Cadillac Marquee Video performance
Viewability of Marquee is 86.4%, much higher vs 68.6% on Cadillac NC Model Leaderboard, and on par with industry benchmark for 1280x %
40% completed the auto-play video.
1.5% engaged on “Click to Play”,
vs 0.26% benchmark of “click to play” on expandable banner containing a video.
22% completed the full video, vs 69.5% benchmark for 30s display ad completion rate
1 in 5 of those who entered the full-video state clicked on the buttons linking to OEM’s website
Overall CTR on ad unit is 0.31% (calculated on click to OEM Site over Impression of Marquee Ad), significantly higher vs NC Model 0.23% (+38%)
*NC Model LB CTR calculated on NC Path & Hub pages only for consistency
Completion of desktop videos (length not specified) is ~79%. North America, 68%.
Classified Model Leaderboard ATS: 75.62%
CTS %
Benchmark: Video Advertising Benchmark, Paul Verna. April 2015 print

12. Marquee on New Car path - Ad Impact
Viewability change (pp) MREC: pp
STM: pp
Cadillac – Challenger vs Control
Scion – Challenger vs Control
Might be impacted by technical issues
Reference line to show the end traditional LB
CTR Changes:
NC Path placements:
Positive impact on most placements
All placements on KBB.com:
CTR increased by +8% for Cadillac and 11% for Scion among the products that had significant changes.
Viewability comparison range. Pre 4/1-415 vs Post 4/ Cadillac ATS & Scion FR-S

13. Marquee on NC - Traffic Impact
After Reaching NC Hub
Changes seen on both Cadillac and Scion:
Increased % visits to NC Section by 0.5% and 3.3%
Increased Time spent on site by 2%
Repeat Visitation decreased by ~1-2%
Cadillac Specific –
-6.7% decreased visit to Owner and -1.6% to UCS
Scion Specific –
during subsequent visit, we see increased PV/V on Owners and UCS section, by +18% and +19%, driving PV/V up by ~5%
Cadillac:
+8% Increased Gal Modal browsing
Decrease in all other sections, such as editorial, compare cars
No significant change in conversion down the path, from different entry point
Year Page
Cat/Style/Option
NC Hub
Scion:
Increased flow into additional NC Hub Page and Top10 articles by 27% and 68%.
Decreased Exit Rate by 19% and decreased action to go back to previous pages
Page load time has increased by ~1s per page, increasing total time to go from Year page to Hub from 48s to 52s

14. Overall Traffic and Ad Impacts - Owners Path
4/17/ /29/2015

15. Owners Path Summary The Good The Bad
“Video completion rate is high from expanded state
“Click to play” rate is higher than industry benchmark, although lower than NC Path
Increased time on site vs. control
Slight Increase in % of visits returning to Owners on subsequent visit
Increased Ad CTR on overall site level
The Bad
Overall CTR to OEM site is very 0.01%
Decreased repeat visitation, overall PV/Visit, and PV/Visitor
Decreased % of visitors to Home, New Car and UC Shopper
Decreased PV/Visit on Homepage and New Car
Difference & Uncertainty
While the creative is exactly the same, the Marquee video con the Owners path, CTR is significantly lower than its NC counterpart

16. Scion Owners Marquee Video Performance
Viewability of Marquee is 68.3%, high vs 59.2% on Scions Owner Model Leaderboard, and below industry benchmark for 1280x %
38% completed the auto-play video
0.9% engaged “Click to Play”,
vs 0.26% benchmark of “click to play” on expandable banner containing a video.
75% completed the video, vs 69.5% benchmark for 30s display ad completion rate
Completion of desktop videos (length not specified) is ~79%. North America, 68% .
Only 1% of those who entered the full-video state clicked on the buttons linking to OEM’s website
Overall CTR on ad unit is 0.01% (calculated on click to OEM Site over Impression of Marquee Ad) significantly lower vs Owners Model 0.05% (-79%)
Benchmark: Video Advertising Benchmark, Paul Verna. April 2015 print

17. Marquee on Owners path - Ad Performance Impact
For comparison:
Scion NC Path vs Control
Scion Owners Path vs Control
Viewability Pre-Post change (pp)
MREC: %
Decision Driver:
Ad Products on Owner’s Path:
+29% on MREC
-21% on STM
Looking at all products on the entire site, CTR increased by +8% among the products that had significant changes.
STM was not run during 4/1-4/15, so no fair comparison
Viewability comparison range. Pre 4/1-415 vs Post 4/ Scion FR-S

18. Marquee on Owners - Traffic Impact
Directional changes:
2.2% Increased time on site vs. control
0.2% increase in % of visitors to Owners on a subsequent visit
2.9% decreased repeat visitation, 0.7% on overall PV/Visit, and 3.5% PV/Visitor
Decreased of visitors to Home (–1.2%) New Car (-5.8%)and UC Shopper (-4.7%)
Decreased PV/Visit on Homepage (-0.6%)and New Car (-1.2%)
-1.2% change in conversion down the path, from different entry point
After Reaching Owners Hub
Style/Option
Option/Condition
Owners Hub
More people went to a previous page in the path
less browsing on additional Hub pages
Page load time has increased by ~0.2s per page, increasing total passage time from 51s to 52s

19. Key Takeaway & Recommendations

20. Executive Summary
Marquee video ads promotes high click through to OEM site on the New Car Path, even though viewer completion rate is low
Significant positive impact on click through rate on New Car path products, as well as all products on the site level
Although most metrics did not reach statistical significance, directionally speaking we see small decreases in repeat visitation, and increased page load time, but overall revenue is directionally positive
Impact on traffic metric and revenue impact differ by creative; Scion creative performed better than Cadillac
Performance on Owners Path is less favorable

21. Analytics Recommendations
Reconsider whether to utilize Marquee for the Owners path, as it garners lower engagement on Owners path while having bigger impact on traffic metrics
Design less disruptive form of video to further promote expansion of full-video state
Closely monitor the relationship between ad impression and auto-play to identify and resolve possible technical difficulties

22. Thank you

23. Appendix

24. Metric Summary Sheet NC - Cadillac NC - Scion Owners - Scion
NC - Cadillac
NC - Scion
Owners - Scion
High Level Traffic
Repeat Visitation
-1%
-2%
-3%
PVs/Visit 5%
PVs/Visitor 2%
-4%
Time Spent on Site
Conversion from Model to YMMT -
% Share of Visits to Section
Homepage 0%
NC Buyers 1% 3%
-6%
Used
-5%
UC Buyers
Owners
Classifieds
-7%
Engagement in Section (pvs/visit)
18%
17% 8%
12% 7%
Ad performance
Marquee vs NC Model LB
38%
-33%
-79%
New Car Model (MREC)
34%
New Car Model (STM)
15%
55%
29%
New Car Decision Driver Model
35%
-21%
Showcase - New Car
104%
Overall Significant Ad CTR Change
11%
Possible Revenue Impact
0.3%
4.7%
-0.8%
Marquee Viewability 9%
Page Load Time
+1sec per page
+0.2sec per page

25. Marque Video Comparison
Scion video had higher engagement on NC than Cadillac
Overall low engagement on Owners Path
VS.
Cadillac NC
Scion NC
Scion Owners
Industry Benchmark
Viewability
86.4%
80.6%
68.3%
85.5%
n/a for autoplay videos,
78% for 15s videos
Autoplay Completion
40%
60%
38%
% “Click to Play”
1.5%
2.8%
0.9%
0.3%
Full video Completion
22%
36%
75%
44%
%Click from expanded state
20%
12% 1%
n/a
Overall CTR
0.31%
0.20%
0.01%
*CTR on controls
0.23%
0.29%
0.05%