1. Modeling Web Quality-of-Experience on Cellular Networks
Athula Balachandran, Vaneet Aggarwal,
Emir Halepovic, Jeff Pang, Srinivasan Seshan,
Shobha Venkataraman, He Yan
AT&T Labs- Research CMU

2. Rise of Mobile
1 EB = 10006bytes = 1018bytes = B = 1000 petabytes = 1millionterabytes = 1billiongigabytes.
Compound annual growth rate
Cisco Visual Networking Index 2013: Global Mobile Traffic Data Update

3. Familiar?

4. User (dis)satisfaction
Why is QoE important?
Cellular
Network Factors
Quality of Experience
User (dis)satisfaction
Revenue

5. Why do we need a QoE model?
Service Quality Monitoring
Trending
Alarming
Better system designs and resource allocations schemes

6. Quality-of-Experience Cellular Network Factors
Outline
Quality-of-Experience
3. What QoE metrics to use?
4. How to extract QoE metrics?
5. Model the
relationship.
Cellular Network Factors
What are the network factors?
2. How to extract network factors?

7. Outline Quality-of-Experience 3. What QoE metrics to use?
4. How to extract QoE metrics?
5. Model the
relationship.
Cellular Network Factors
What are the network factors?
2. How to extract network factors?

8. Cellular Network Factors
Signal Strength
Cell load
Handovers
Failures
Throughput

9. Outline Quality-of-Experience 3. What QoE metrics to use?
4. How to extract QoE metrics?
5. Model the
relationship.
Cellular Network Factors
What are the network factors?
2. How to extract network factors?

10. Collecting Network Characteristics
Logs collected at Radio Network Controller
Intermittent: Signal strength, Throughput
Event based: Handovers, Failures

11. Quality-of-Experience Cellular Network Factors
Outline
Quality-of-Experience
3. What QoE metrics to use?
4. How to extract QoE metrics?
5. Model the
relationship.
Cellular Network Factors
What are the network factors?
2. How to extract network factors?

12. QoE Metrics Session Length Abandonment Partial Download Ratio (PDR)
session length (i.e., the number of pages a user clicks through)
abandonment or bounce rate (i.e., if a user leaves the website after only visiting the landing page)
ndeed, we find that many web sessions are only one click (and thus, by definition, abandoned). These metrics do little to distinguish satisfied and dissatisfied users of these single-click sessions. In this section, we show that partial download ratio, i.e., the fraction of HTTP objects that are not com- pletely downloaded in a session, is strongly correlated with session length and abandonment rate, so we can use it as a proxy to esti- mate user experience, even for sessions lasting a single click.

13. Outline Quality-of-Experience 3. What metrics to use?
4. How to extract these metrics?
5. Model the
relationship.
Cellular Network Factors
What are the network characteristics?
2. How to extract these metrics?

14. Detecting Clicks
Challenge: Classify embedded objects vs. click from network traces.
Current Approaches:
Idle-time based
Stream Structure
Our Approach: Text classification
To improve on this approach, StreamStructure [24] exploits the structure of “desktop” web pages to detect requests for new web- pages. However, we show in the next section that it is not as adept at identifying clicks in mobile web pages. Moreover, it is a page detection algorithm that is used to identify clicks resulting in new pages. Other client-side interaction (e.g., clicking to play a video within a page) are not identified by this algorithm.

15. Performance News, Social, Wiki
After testing with multiple machine learning algorithms (such as decision trees, logistic regression, Support Vector Machines [29]), we found that Naive Bayes performs the best compared to other approaches. This is not surprising given that Naive Bayes has been found to perform the best in other text classification problems as well [7].
Naive Bayes has been studied extensively since the 1950s. It was introduced under a different name into the text retrieval community in the early 1960s,[1]:488 and remains a popular (baseline) method for text categorization, the problem of judging documents as belonging to one category or the other (such as spam or legitimate, sports or politics, etc.) with word frequencies as the features. With appropriate preprocessing, it is competitive in this domain with more advanced methods including support vector machines.[2] It also finds application in automatic medical diagnosis.[3]
Naive Bayes is a simple technique for constructing classifiers: models that assign class labels to problem instances, represented as vectors of feature values, where the class labels are drawn from some finite set. It is not a single algorithm for training such classifiers, but a family of algorithms based on a common principle: all naive Bayes classifiers assume that the value of a particular feature is independent of the value of any other feature, given the class variable. For example, a fruit may be considered to be an apple if it is red, round, and about 3" in diameter. A naive Bayes classifier considers each of these features to contribute independently to the probability that this fruit is an apple, regardless of any possible correlations between the color, roundness and diameter features.
For some types of probability models, naive Bayes classifiers can be trained very efficiently in a supervised learning setting. In many practical applications, parameter estimation for naive Bayes models uses the method of maximum likelihood; in other words, one can work with the naive Bayes model without accepting Bayesian probability or using any Bayesian methods.
Precision is defined as the number of correct clicks identified divided by the total number of clicks identified
Recall is de fined as the number of correct clicks identified divided by
the total number of clicks.

16. Quality-of-Experience Cellular Network Factors
Outline
Quality-of-Experience
3. What metrics to use?
4. How to extract these metrics?
5. Model the
relationship.
Cellular Network Factors
What are the network characteristics?
2. How to extract these metrics?

17. Correlation Analysis Session Length Abandonment Cell load
Partial Download
Ratio (PDR)
Cell load
Signal Strength
Handovers
Failures
Throughput

18. Increasing Cell load leads to worse web QoE
Correlation Analysis
Cell load
Increasing Cell load leads to worse web QoE

19. Web QoE is interference limited and not power limited.
Correlation Analysis
RSSI: Received Signal Strenth
Signal Strength
ECNO： How well a signal can be
distinguished from the noise.
Similar to SINR in WiFi
to signal to noise ratio
AverageReceivedSignalCodePower(RSCP):Thisisthedown- link power received by the UE receiver on the pilot channel. It is measured in dBm.
Average received energy per chip of the pilot channel over the noise power density (ECNO): It is expressed in dB and it mea- sures how well a signal can be distinguished from the noise in a cell. It is measured in dB. Note that ECNO is measured on the pilot channel and thus may be different from the SINR of the traffic channel.
Average received Signal Strength Indicator (RSSI): Expressed in dBm, it is the wide-band received power within the relevant channel bandwidth. It is related to RSCP and ECNO as follows: RSSI = RSCP - ECNO. Note that RSSI is measured on the pilot channel and thus may be different from the received power of the signal on the traffic channel.
Signal-to-interference-plus-noise ratio
Web QoE is interference limited and not power limited.

20. IRAT handovers lead to worse QoE
Correlation Analysis
Soft
Handovers
Inter-frequency
IRAT
IRAT handovers lead to worse QoE

21. Web QoE is more latency-limited than throughput-limited
Correlation Analysis
Downlink
Throughput
Failures
Uplink
Higher radio data rate does not necessarily lead to better web QoE. Figure 11 shows the impact of radio data rate on partial download ratio. As web objects are primarily downloaded onto the mobile device, we start by looking at the downlink direction and find that higher data rates do not improve partial download ratio (Figure 11a). As expected, uplink data rate shows no impact (Fig- ure 11b). We find similar relationship between data link rates and other web QoE metrics (not shown). While it may not be intuitive that data rate and web QoE metrics have weak relationship, it has been shown that web browsing traffic is more latency-limited than throughput-limited [5, 9]
This is essentially available bandwidth for the UE
Web QoE is more latency-limited than throughput-limited

22. Correlation Analysis: Summary
Cell load, IRAT handovers lead to worse QoE.
Improving ECNO leads to better QoE.
Higher RSSI  worse QoE.
All other handovers, throughput, failures do not have much impact on QoE.

23. Complex Inter-Dependencies
Cell load
Failures
Signal Strength
Throughput
Handovers

24. Unified Model Machine Learning Web QoE metrics Network Characteristics
QoE Model

25. Predictive Models Model ML Algorithm Model RMSE Estimate PDR
Linear Regression
0.1709
Estimate Session length
Linear Regression
1.703
Model
ML Algorithm
Model Accuracy
Predict Abandonment
Decision Tree
69.12
Predict Partial Download
73.02

26. External Factors: Time of day
users are less likely to engage in long browsing sessions during working hours
We observe that average signal strength to in- terference (ECNO) is lower during peak hours compared to non- peak hours. On the other hand, average signal strength (RSSI) is higher during peak hours compared to non-peak hours.

27. External Factors: Website
users tend to have different browsing behavior on these websites: Shopping, Marketplace, and Social News sites understandably tend to have higher session lengths, while Wiki and Blog tend to have low ses- sion lengths.

28. Unified Model Machine Learning External Factors Web QoE metrics
Network Characteristics
Machine Learning
QoE Model

29. Predictive Models Model ML Algorithm Old RMSE Updated RMSE
Estimate PDR
Linear Regression
0.1709
0.087
Estimate Session length
1.703
1.401
Model
ML Algorithm
Old Accuracy
Updated Accuracy
Predict Abandonment
Decision Tree
69.12
74.30
Predict Partial Download
73.02
83.95
Baseline:
Estimate PDR:
Estimate Session length: 1.501
Predict abandonment: 69.16
Predict partial download: 67.5
To measure the “goodness-of-fit” of linear regression, we use the standard measure of root mean squared error (RMSE), where lower RMSE values indicate better prediction.

30. Updated Model
Fortunately, both linear regression and decision tree algorithms that gave the highest accuracy also generate very intuitive models.
We observe that the features that the models learnt (number of users, ECNO, RSSI etc.) are the same as those that we found to be impactful in Section 5. Moreover, the model also ignores factors such as downlink and uplink throughput that we found to not have an impact on web QoE metrics.
Network operators can hence use this model to understand the true impact of a parameter. For example, comparing the co-efficients, decreasing IRAT han- dovers and improving ECNO has the highest impact on improving partial download ratio. We also found similar conclusions from an- alyzing the regression co-efficients for session length (not shown due to space constraints).
Figure 16 shows the pruned decision tree that we learnt for pre- dicting partial download for the Wiki website. Again, consistent with our analysis in Section 5, the model picks parameters such as Number of users, ECNO, IRAT etc. to branch on, reconfirming the impact of these factors. Further, the decision tree rules separate the data based on time of day into a similar classification that we made for peak vs. non-peak (e.g., Time of day <= 9, Time of day > 9 and Time of day <= 19, Time of day > 19). We also observe that the feature splits conform with several of our observations. For ex- ample, during non-peak hours the partial downloads are lower (if Time of day > 21, predict full). Also if load is higher partial down- loads are higher (if Normalized Num User <= 1.05, predict full otherwise part).

31. Conclusions Web QoE  $$ QoE metrics and network parameters
Session Length, Abandonment, PDR  text classification
Network parameters  RNC logs
Network parameters impact web QoE
ECNO, Cell load, IRAT handovers
Build accurate and intuitive models
Complex relationships  ML algorithms
Incorporate external factors.

32. Extra SLIDES

33. Other models not discussed
Other models are not so accurate because they are more governed by user interest.

34. Updated Model: Estimate PDR

35. Unified Model Web QoE Model Web QoE Cell load Signal Strength
Handovers
Throughput
Failures
Web QoE Model
Web QoE
Estimate partial download ratio – Linear Regression
Estimate session length
Predict partial download – Decision Tree
Predict user abandonment

36. Why is QoE important?
How to measure QoE?
How to improve QoE?
Why measure QoE?