1. A First Look at Performance of TV Streaming Sticks
Ayon Chakraborty, Arani Bhattacharya, Santosh Ghosh, Samir R. Das
Good afternoon everyone. I am Arani Bhattacharya of Stony Brook University, and I would like to present out work “A First Look at Performance of TV Streaming Sticks”.

2. Smart TVs Increasingly Popular
… and so are streaming sticks!
Internet
With the arrival of smart TVs, streaming sticks are used to stream movies and other videos over the Internet. These sticks have the form factor of a dongle and they are plugged into the HDMI port of TVs. There are three TV streaming sticks that are commonly available in the US market – Amazon Fire, Chromecast and Roku. A key question of active interest for users is the difference among these devices.

3. Lots of User Concerns Internet
To figure out what sort of concerns user have, we ran a data analysis on three websites – Reddit, Chromecast Google Groups and Roku forum. We checked for keywords related to streaming and network problems in over 50K posts over three years. For example, …

4. Quality of Experience Affected
Internet
Startup Delay
We now look at the reasons behind the user complaints. Users care about the startup delay, video bitrate, buffering of video, amount of data usage and the device’s impact on other devices such as laptops or smartphones. They want to understand how different devices affect these Quality of Experience parameters.

5. Quality of Experience Affected
Internet
Startup Delay
Video Bitrate

6. Quality of Experience Affected
Internet
Startup Delay
Video Bitrate
Buffering

7. Quality of Experience Affected
$$$
Internet
Startup Delay
Video Bitrate
Buffering
Data Usage

8. Quality of Experience Affected
$$$
Internet
Startup Delay
Video Bitrate
Buffering
Data Usage
Impact Other Devices

9. Measuring the Quality of Experience Parameters
Internet
Startup Delay
Video Bitrate
We focus on the problem of measuring these QoE parameters.
Buffering
Data Usage
How can we measure parameters of different devices?
Impact Other Devices

10. Getting Bitrate and Resolution Challenging
Internet
Works only with Smart TVs
-- Cannot log parameters
Encrypted traffic
-- Intercepting traffic not useful
Measuring these QoE parameters is challenging because of two reasons. First, these devices work only with smart TVs and not with any other devices. Thus, their parameters cannot be easily logged. Secondly, they fetch data over the network in encrypted form. Thus, intercepting network data also does not directly help.
How can we measure parameters of different devices?

11. Adaptation Technique in Streaming Sticks
Internet
Works only with Smart TVs
-- Cannot log parameters
Encrypted traffic
-- Intercepting traffic not useful
Measuring these QoE parameters is challenging because of two reasons. First, these devices work only with smart TVs and not with any other devices. Thus, their parameters cannot be easily logged. Secondly, they fetch data over the network in encrypted form. Thus, intercepting network data also does not directly help.
Adaptation ONLY on client side

12. Replacing Internet channel by our server does not affect our results
Works only with Smart TVs
-- Cannot log parameters
Encrypted traffic
-- Intercepting traffic not useful
Measuring these QoE parameters is challenging because of two reasons. First, these devices work only with smart TVs and not with any other devices. Thus, their parameters cannot be easily logged. Secondly, they fetch data over the network in encrypted form. Thus, intercepting network data also does not directly help.
Adaptation ONLY on client side

13. Viewer can see the resolution and video bitrate
800× Kbps
Watermark
800x600
720x480
375 Kbps
Video frames watermarked with resolution and bitrate
Measuring these QoE parameters is challenging because of two reasons. First, these devices work only with smart TVs and not with any other devices. Thus, their parameters cannot be easily logged. Secondly, they fetch data over the network in encrypted form. Thus, intercepting network data also does not directly help.

14. 5 watermarked video files of 10 minutes each
Measurement Setup
800× Kbps
Watermark
Control Bandwidth, Latency
Wowza Video Server
5 watermarked video files of 10 minutes each
Tc-netem
800×600
235 Kbps
We now describe our measurement setup. First, we install our own server to control the videos. We encode the video at different bit rates and resolutions, and watermark these values on the video frames. We then store the video files using a publicly available streaming engine software Wowza. This brings a watermark on the playing video indicating the bitrate and the resolution. We also control the latency and the bandwidth from the router. Now, to avoid manual involvement, we record the playing of the video by a video camera. We now take the video frames, and use OCR to store its watermark values in a separate log file. We repeat this experiment with 5 video files of 10 minutes each.
1s:
720x480
375 Kbps
800×600, 235K
2s: 720×480,300K
………
Bitrate,
Resolution
Video camera
OCR on video frames
Log file

15. Stick Performance Metrics
Streaming Performance
Average Bitrate
Video Startup Delay
Video Stalls
Network Load
Data Wastage on Abandonment
Effect of Background Traffic
We first list out the performance metrics. We focus on two broad aspects – streaming performance and network load.

16. Roku provides the highest average bitrate
Better Quality
Average Network Bandwidth
Roku most aggressive
Chromecast targets an average bitrate

17. Increase in bitrate delays startup
Better Quality
Better Quality
Average Network Bandwidth
Average Network Bandwidth

18. Chromecast has the lowest amount of buffering
Better Quality
Better Quality
Average Network Bandwidth
Average Network Bandwidth

19. Roku prefetches data most aggressively
Better Quality
Roku prefetches most, followed by Chromecast and Fire

20. Fire is most affected by background traffic
Better Quality
Fire reduces its utilization the most.
Chromecast and Roku gets much less affected.

21. User Concerns Internet
To figure out what sort of concerns user have, we ran a data analysis on three websites – Reddit, Chromecast Google Groups and Roku forum. We checked for keywords related to streaming and network problems in over 50K posts over three years. For example, …

22. Analysis of User Complaints Confirm Our Measurement Study
Amazon has fewest complaints related to network, Roku has most.
Overall complaints about streaming are roughly similar.

23. Takeaways
A systematic way of understanding the complaints of TV stick users
We showed how streaming sticks prioritize different aspects of Quality of Experience:
Roku tries to fetch higher bitrates, loses on startup
Fire minimizes usage of network bandwidth
THANK YOU

24. A First Look at Performance of TV Streaming Sticks
Ayon Chakraborty, Arani Bhattacharya Santosh Ghosh, Samir R. Das