University of Michigan Electrical Engineering and Computer Science Anatomizing Application Performance Differences on Smartphones Junxian Huang, Qiang Xu, Birjodh Tiwana, Z. Morley Mao University of Michigan Ming Zhang, Victor Bahl Microsoft Research

University of Michigan Electrical Engineering and Computer Science Smartphones are popular  Various smartphone platforms  Many applications  High speed network access  3G (UMTS & EVDO)  4G (LTE) 1

University of Michigan Electrical Engineering and Computer Science Understanding application performance is important 2 AT&T T-Mobile Verizon

University of Michigan Electrical Engineering and Computer Science  User coverage vs. collected information Design tradeoff in smartphone measurement 3 User coverage More info Fewer users More info Fewer users Less info More users Less info More users Collected information Speedtest.net FCC’s broadband test Speedtest.net FCC’s broadband test Liu, MobiCom 2008 Zhong, HotMetrics 2010 Willkomm, DySPAN 2008 Our study

University of Michigan Electrical Engineering and Computer Science Our contributions  3GTest  Widely deployed  Provide new insights into cellular network performance  A systematic methodology to understand application performance  Isolate individual factors  Enable fair comparisons  Identify performance bottlenecks  Offer concrete recommendations for improvements 4

University of Michigan Electrical Engineering and Computer Science Outline  Cellular network performance study  3GTest design  3GTest results  Web performance analysis  Experimental methodology & setup  Web performance results  Streaming video & VoIP performance study  Conclusion 5

University of Michigan Electrical Engineering and Computer Science Key challenges of 3GTest  Adoptability  User privacy  Running time  Ease of use  Energy consumption  Data usefulness  Metrics relevant to network applications 6

University of Michigan Electrical Engineering and Computer Science How to make 3GTest easily adoptable?  Avoid personal identifiable information 7

University of Michigan Electrical Engineering and Computer Science What information is most relevant for 3GTest? 8 Smartphone Local DNS server Web server example.com? Get index.html Ping to 1 st hop End-end ping

University of Michigan Electrical Engineering and Computer Science Representativeness of 3GTest data 66K+ users 148K+ runs 84 carriers 147 countries over 10 months 9

University of Michigan Electrical Engineering and Computer Science Throughput comparison 10 CDF T- Mobile:600 kbps AT&T 1Mbps AT&T 1Mbps

University of Michigan Electrical Engineering and Computer Science RTT comparison 11 CDF T-Mobile 450ms T-Mobile 450ms AT&T 350ms AT&T 350ms

University of Michigan Electrical Engineering and Computer Science Retransmission rate comparison 12 CDF T-Mobile AT&T

University of Michigan Electrical Engineering and Computer Science Insights from 3GTest results  Local DNS server performance is good  200 ms comparable to ping to 1 st hop  Carrier network delay dominates end-to-end RTT  End-end ping close to ping to 1 st hop  Time of day effect  AT&T’s downlink throughput  Signal strength correlates with TCP performance 13

University of Michigan Electrical Engineering and Computer Science Smartphone web browsing is complicated 14 Smartphone Local DNS server Web server DNS lookup Content customization TCP handshake TCP data transfer Server configuration Client configuration HTML rendering & JavaScript execution 5 5 Network condition Server processing delay 3 3

University of Michigan Electrical Engineering and Computer Science Key challenges of web performance analysis  Multiple factors have impact  Network condition  Client settings  Server configurations  Content differences  Lack of low-level information and access  No visibility into link layer  Difficult to instrument client OS and application  No access to public web servers 15

University of Michigan Electrical Engineering and Computer Science How to isolate impact of individual factor?  Vary one factor at a time  Network condition  Simulate cellular network with WiFi  Client settings  Control TCP concurrency  Server configurations  Change compression mode and server parallelism  Content difference  Host static website snapshots locally 16

University of Michigan Electrical Engineering and Computer Science How to identify performance bottlenecks?  TCP flow analysis technique to break down page load time  DNS lookup time  TCP handshake time  TCP idle time  TCP transfer time 17 index.html logo.png Time

University of Michigan Electrical Engineering and Computer Science Controlled experiment setup  Platform  Web browsing  Other applications 18

University of Michigan Electrical Engineering and Computer Science Simple website page load time break down 19 TCP idle time TCP transfer time

University of Michigan Electrical Engineering and Computer Science Content-rich website page load time break down 20 TCP idle time TCP transfer time

University of Michigan Electrical Engineering and Computer Science JavaScript execution comparison 21 PC 80X (CPU <10X)

University of Michigan Electrical Engineering and Computer Science Lessons from application performance study  Performance bottlenecks of web browsing  Client execution  Data download  Higher concurrency significantly improves performance  Compression is usually helpful  Content customization is beneficial  Data URL scheme  Minification  Minimal redirections  Video & VoIP  Platform specific application behaviors 22

University of Michigan Electrical Engineering and Computer Science Conclusion  3GTest for measuring cellular network performance  Deployed widely  Gain insights into cellular network performance  A systematic methodology to understand application performance  Isolate impact of individual factor  Identify performance bottlenecks  Offer recommendations on performance improvement 23

University of Michigan Electrical Engineering and Computer Science Visit 3GTest website 24 AndroidiPhoneWindows Mobile