1. Improving and Controlling User-Perceived Delays in Mobile Apps Lenin Ravindranath By the time it loads, the church service is over. Too slow!!! Slow responsiveness, long load time. So slow. Did an intern write this app?? Slower than a snail. horrifically slow. Uninstalled. MIT So slow that my screen already dims before it's loaded

4. ~ Two Million Apps > 500,000 Developers

5. Too slow - killing the usefulness when you really need to go

6. “So slow. Did an intern write this app??” “Slower than a snail even on 3G.” “Slow and unresponsive like mud” “Sluggish and freezes my HTC phone.” “Very very slow compared to even browsing web.” “Consistently 3 seconds behind where I touch.” “Loading GPS data is *** slow”

7. Diverse environmental conditions – Network connectivity, GPS signal quality, – Location, Sensor conditions etc. Variety of hardware and OS versions Wide range of user interactions and inputs Hard for developers to test Performance problems are inevitable in the wild Significant diversity in the wild

8. App store is brutally competitive Need for improving and controlling performance Response time matters – Users are impatient – 100ms delay can cost substantial drop in revenue at Amazon – Similar observations from Google and Bing Help app developers

9. Find a bar

10. User InteractionRendering User-perceived delay Monitor in the hands of user Improve and control the user-perceived delay

11. User InteractionRendering AppInsight OSDI ‘12 Monitor performance in the wild What is the user-perceived delay? Where is the bottleneck? Developer Feedback Developer Feedback Users with slow network – Poor Performance Users with fast network – Not producing the best quality result

12. User InteractionRendering AppInsight OSDI ‘12 Monitor performance in the wild What is the user-perceived delay? Where is the bottleneck? Developer Feedback Developer Feedback Deal with uncontrolled variability

13. User InteractionRendering AppInsight OSDI ‘12 Monitor performance in the wild What is the user-perceived delay? Where is the bottleneck? Developer Feedback Developer Feedback Timecard SOSP ‘13 Adapt processing to conditions in the wild How much time has already been spent? How much time will be spent? Runtime Estimation Runtime Estimation Adapt High delays Reduce Tightly control user-perceived delay

14. User InteractionRendering AppInsight OSDI ‘12 Monitor performance in the wild What is the user-perceived delay? Where is the bottleneck? Developer Feedback Developer Feedback Timecard SOSP ‘13 Adapt processing to conditions in the wild How much time has already been spent? How much time will be spent? Runtime Estimation Runtime Estimation Adapt Tightly control user-perceived delay Low delays Improve result

15. User InteractionRendering AppInsight OSDI ‘12 Monitor performance in the wild What is the user-perceived delay? Where is the bottleneck? Developer Feedback Developer Feedback Timecard SOSP ‘13 Adapt processing to conditions in the wild How much time will be spent? Runtime Estimation Runtime Estimation How much time has already been spent?

16. AppInsight Timecard Minimal developer effort Readily deployable - No changes to the OS or runtime Negligible overhead Product Impact at Microsoft, Facebook  Used by dozens of developers

17. AppInsight Timecard App Instrumented Automatic Binary Instrumentation User Transaction Tracking Monitor app execution across async boundaries in a light-weight manner

18. AppInsight Monitor performance in the wild What is the user-perceived delay? Where is the bottleneck? Developer Feedback Developer Feedback Significant barrier for most app developers No platform support Only option is to instrument your app – Manage your own logging infrastructure

19. Users App Instrumenter App Instrumented Analysis Developer Feedback Developer Feedback Server Traces Performance Improved Developer AppInsight App Store

20. Highly interactive, UI centric – Single UI thread that should not be blocked Most tasks are performed asynchronously – Asynchronous APIs for Network, Sensor, IO etc. – Computation performed on background threads Highly Asynchronous Programming Pattern Monitoring App Performance is Challenging Tracing async code is challenging

21. Internet Location Mood Around Me Nearby Tweets GPS sad “Just got engaged :)” “My life sucks” “At last its sunny. But I have to sit and prepare my talk :(” “I am sad that there is no more how I met your mother”

22. ClickHandler() { location = GPS.Sample(); url = GetUrl(location) tweets = Http.Get(url); result = Process(tweets); UI.Display(result); } GetUrl(location) {... } Process(tweets) {... } ClickHandler Start ClickHandler End LogStart(); LogEnd(); Thread User-perceived Delay Hypothetical Synchronous Code User Interaction Mood Around Me sad

23. ClickHandler() { GPS.AsyncSample(GPSCallback); } GPSCallback(location) { url = GetUrl(location); Http.AsyncGet(url, DownloadCallback); } DownloadCallback(tweets) { result = Process(tweets); UI.Dispatch(Render, result); } Render(result) { UI.Display(result); } GetUrl(location){... } Process(tweets){... } Render UI Thread Background Thread UI Dispatch ClickHandler Start System Process Download Callback User Interaction Asynchronous Code Background Thread GPS Network ClickHandler End Async Http Call Async GPS Call GPS Callback

24. Render UI Thread UI Dispatch ClickHandler Start System Async GPS Call Download Callback User Interaction Background Thread Network Async Http Call GPS Callback GPS Background Thread User Transaction Transaction timeUser-perceived Delay

25. UI Thread Background Thread User Transaction

26. Apps are highly asynchronous Where is the bottleneck? Focus development efforts 30 popular apps 200,000 transactions – On average, 10 asynchronous calls per user transaction Up to 7000 edges – There are apps with 8 parallel threads per transaction

27. UI Thread Background Thread Twitter Thread WakeupThread Blocked Fire Render Callback Nearby Tweets Nearby Posts Twitter Facebook Process Posts Process Tweets Facebook User Interaction Aggregate GPS GPS Callback Mood Around Me

28. UI Thread Background Thread Thread WakeupThread Blocked Fire Render User Transaction User Interaction Aggregate GPS GPS Callback Critical Path Optimizing the critical path reduces the user perceived delay Callback Facebook Callback Twitter Facebook Process Posts Process Tweets

29. Users App Instrumenter App Instrumented Analysis Developer Feedback Developer Feedback Server Traces Developer AppInsight App Store

30. Low Overhead Capturing User Transactions UI Thread Background Thread User Transaction Instrumentation impacts app performance – They are already slow enough!!! Scarce Resources – Compute, Memory, Network, Battery

31. User Interactions Capture UI Thread Background Thread User Transaction User Interaction Event Handler

32. User Interactions Thread Execution Capture UI Thread Background Thread User Transaction ClickHandler Start ClickHandler End Render Start DownloadCallback Start End GPSCallback Start GPSCallback End Render End

33. User Interactions Thread Execution Async Calls and Callbacks Capture UI Thread Background Thread User Transaction UI Dispatch Call Download Callback Async GPS Call GPS Callback Async Http Call UI Dispatcher Callback

34. User Interactions Thread Execution Async Calls and Callbacks UI Updates Capture UI Thread Background Thread User Transaction UI Update

35. UI Thread Background Thread User Interactions Thread Execution Async Calls and Callbacks UI Updates Thread Synchronization Fire Thread Blocked Thread Wakeup Capture User Transaction

36. User Interactions Thread Execution Async Calls and Callbacks UI Updates Thread Synchronization Capture UI Thread Background Thread User Transaction

37. Trace every method – Prohibitive overhead Enough to log thread boundaries Log entry and exit of Upcalls Capturing Thread Execution Upcalls System App UI Thread Background Thread ClickHandler Start ClickHandler End Render Start End GPSCallback Start GPSCallback End Render End DownloadCallback Start

38. ClickHandler() { GPS.AsyncSample(GPSCallback); } GPSCallback(location) { url = GetUrl(location); Http.AsyncGet(url, DownloadCallback); } DownloadCallback(tweets) { result = Process(tweets); UI.Dispatch(Render, result); } Render(result) { UI.Display(result); } Event Handlers are Upcalls Function pointers point to potential Upcalls – Callbacks are passed as function pointers Identify Upcalls

39. Instrument Upcalls Rewrite app – Trace Upcalls ClickHandler() { Logger.UpcallStart(1); GPS.AsyncSample(GPSCallback); Logger.UpcallEnd(1); } GPSCallback(location) { Logger.UpcallStart(2); url = GetUrl(location); Http.AsyncGet(url, DownloadCallback); Logger.UpcallEnd(2); } DownloadCallback(tweets) { Logger.UpcallStart(3); result = Process(tweets); UI.Dispatch(Render, result); Logger.UpcallEnd(3); } Render(result) { Logger.UpcallStart(4); UI.Display(result); Logger.UpcallEnd(4); } GetUrl(location){... } Process(tweets){... } UI Thread Background Thread ClickHandler Start ClickHandler End Render Start DownloadCallback Start End GPSCallback Start GPSCallback End Render End Low Overhead 7000 times less overhead

40. UI Thread Background Thread UI Dispatch Call Download Callback Async GPS Call GPS Callback Async Http Call UI Dispatcher Callback Async Calls and Callbacks

41. Log Callback Start – We capture start of the thread Log Async Call Match Async Call to its Callback GPSCallback(location) { Http.AsyncGet(url, DownloadCallback); Logger.AsyncCall(5); } DownloadCallback(tweets) { Logger.UpcallStart(3);.... } Download Callback Async Http Call Detour Callbacks

42. System App Http.AsyncGet Async Call DownloadCallback DownloadCallback(tweets) { } Callbacks

43. Detour Callbacks DetourCallback(tweets) { DownloadCallback(tweets); } class DetourObject { } MatchId = 3 obj.DetourCallback MatchId = 3 Http.AsyncGet(url, DownloadCallback); Http.AsyncGet(url, obj.DetourCallback); obj = new DetourObject(DownloadCallback, MatchId++); System App Http.AsyncGet Async Call DownloadCallback DownloadCallback(tweets) { } while(...) { }

44. Detour Callbacks DetourCallback(tweets) { DownloadCallback(tweets); } class DetourObject { } MatchId = 4 obj.DetourCallback MatchId = 4 obj = new DetourObject(DownloadCallback, MatchId++); System App Http.AsyncGet Async Call DownloadCallback(tweets) { } while(...) { } Http.AsyncGet(url, obj.DetourCallback);

45. Detour Callbacks DetourCallback(tweets) { DownloadCallback(tweets); } class DetourObject { } MatchId = 5 obj.DetourCallback MatchId = 5 obj = new DetourObject(DownloadCallback, MatchId++); System App Http.AsyncGet Async Call DownloadCallback(tweets) { } while(...) { } Http.AsyncGet(url, obj.DetourCallback);

46. Detour Callbacks Delayed Callbacks – Track Object Ids Event Subscriptions Async Calls and Callbacks Low Overhead void foo() { } Thread t = new Thread(foo);... t.Start();

47. User Interactions Thread Execution Async Calls and Callbacks UI Updates Thread Synchronization Capture UI Thread Background Thread User Transaction

48. Keeping the overhead low Use app idle time – Garbage collect detour objects during idle time – Log to memory, write to disk during idle time Compress log before sending to server – Delta encoding to achieve high compression Background transfer after app quits

49. Users App Instrumenter App Instrumented Analysis Developer Feedback Developer Feedback Server Traces Developer AppInsight App Store

50. Users App Instrumenter App Instrumented Analysis Developer Feedback Developer Feedback Server Traces Developer AppInsight App Store

51. Aggregate Analysis Group similar transactions – Same transaction graph Outliers – Points to corner cases Highlight common critical paths – Focus development effort Root causes of performance variability – Highlight what really matters in the wild Async GPS CallAsync Http Call Network GPS Device Compute

52. Users App Instrumenter App Instrumented Analysis Developer Feedback Developer Feedback Server Traces Developer AppInsight App Store

53. Developer Feedback Web based Interface Long User Transactions Critical Path Aggregate Analysis – Outliers – Common Case – Factors affecting

54. Users App Instrumenter App Instrumented Analysis Developer Feedback Developer Feedback Server Traces Developer Deployment AppInsight App Store

55. Deployment 40 apps in the Windows Phone store 30 to few thousand users 6 months to 1 year of data > Million user transactions

56. AppInsight Overhead Battery <1% Binary Size 1.2% Memory 2% Network 4% Compute 0.02% Low Overhead Impact on app performance is minimal Low resource consumption Based on 6 months of data from 30 apps in the wild

57. User Transactions and Critical Path 15% of the user transactions took more than 5 seconds! – 30% took more than 2 seconds Top 2 edges responsible for 82% of transaction time Critical path analysis – Focus on less than half the edges for 40% of transactions Aggregate analysis – Pointed to factors creating performance variability – GPS, Device, Network, Sensors Focus development efforts

58. App Instrumenter App Instrumented Analysis Developer Feedback Developer Feedback Server Traces AppInsight Developer Improved Users App Store

59. App: TwitReviews Problem: UI hog AppInsight: High performance variability in UI thread Abnormal latencies only at the start of the session System loading DLLs in the critical path – Up to 1 second Fix: Preload the DLL asynchronously when the app starts – Dummy call to DLL Developer Case Study

60. App: TwitReviews Problem: Radio wakeup delay AppInsight: High performance variability before network call Radio wakeup delay in critical path Dominating when users are on cellular networks – More than 2 seconds Fix: Wakeup radio as the user starts typing before invoking the transactions Developer Case Study

61. App: Bing App Problem: Slow transactions AppInsight: Custom instrumentation in the critical path – Affecting user perceived delay

62. App: OneBusAway Problem: Slow transactions AppInsight: Aggregate analysis showed cellular latencies significantly affecting critical paths – Majority of transactions happens when user is in cellular – Known issue but no quantitative data – Current caching policy insufficient Fix: Change caching policy? Developer Case Study

63. Users App Instrumenter App Instrumented Analysis Developer Feedback Developer Feedback Server Traces AppInsight Developer App Store

64. Server-based User Transactions Core processing of the user transaction is at the server Core processing Server Input and sensors User Interaction App Send request Network Parse and render Receive response App Network

65. GPS Start Http Request GPS Callback Click Handler UI Dispatcher Download Callback Server Network + Server Joint app-server instrumentation – Azure services Server-based User Transactions Request Handler Start Request Handler End Network

66. Server Threads Spawn Workers Send Response Request Handler GPS Start Http Request Click Handler Download Callback GPS Callback UI Dispatcher Server-based User Transactions Identify and fix bottlenecks at the app and the server Network Server Network HTTP[“x-AppInsight-Id”]

67. Server User Interaction Rendering App Network Uncontrollable and Variable Delays Reading sensors Radio state Device type (3G, 4G, WiFi, LTE,..) RTT, tput, TCP, DNS Device type

68. Server User Interaction Rendering App Network Uncontrollable and Variable Delays Reading sensors Radio state Device type (3G, 4G, WiFi, LTE,..) RTT, tput, TCP, DNS Device type Significant variability in user-perceived delays

69. Server User Interaction Rendering App Network Uncontrollable and Variable Delays Reading sensors Radio state Device type (3G, 4G, WiFi, LTE,..) RTT, tput, TCP, DNS Device type Significant variability in user-perceived delays Users with high uncontrollable delays – Poor user-perceived delays Users with low uncontrollable delays – Not producing the best quality result

70. Server User Interaction Rendering App Network Uncontrollable and Variable Delays Reading sensors Radio state Device type (3G, 4G, WiFi, LTE,..) RTT, tput, TCP, DNS Device type Tightly control user-perceived delays Adapt to conditions in the wild

71. Server User Interaction Rendering App Network Adapt to conditions in the wild Adapt Do less or more processing Send less or more data Meet end-to-end deadline Trade-off on quality or quantity of the result Tightly control user-perceived delays

72. Request Response Deadline Server processing Server Controlling the Server Delay

73. Request Response Deadline Worker Server Controlling the Server Delay

74. Request Response Deadline Worker Better result Server Controlling the Server Delay

75. Deadline Server Server processing Controlling the Server Delay Servers keep fixed deadlines – No visibility into external delays

76. Server User Interaction Rendering App Network Adapt to conditions in the wild Tightly control user-perceived delays Adapt

77. Timecard App Server GetElapsedTime(); PredictRemainingTime (responseSize); Time elapsed since user interaction Predicted downlink & app processing delay

78. Timecard App Server GetElapsedTime(); PredictRemainingTime (responseSize); Adapt Processing Time Desired user-perceived delay

79. Timecard App Server GetElapsedTime(); PredictRemainingTime (responseSize); Adapt Processing Time Trade-off on quality of the result Desired user-perceived delay

80. Timecard App Server GetElapsedTime(); PredictRemainingTime (responseSize); Adapt Response PredictRemainingTime(10KB); PredictRemainingTime(5KB) PredictRemainingTime(15KB); Desired user-perceived delay

81. Timecard App Server GetElapsedTime(); PredictRemainingTime (responseSize); Adapt Response Desired user-perceived delay

82. Timecard App Server GetElapsedTime(); PredictRemainingTime (responseSize); Better quality result Desired user-perceived delay

83. Timecard App Server GetElapsedTime(); PredictRemainingTime (responseSize); Analysis of 4000 apps – 80% of them are single request-response transactions

84. Challenges Server App

85. Challenges UI Thread Background Thread GPS Start Http Request GPS Callback Click Handler Server App UI Dispatcher Download Callback App Highly asynchronous

86. Server Threads Challenges Spawn Workers Send Response Server Request Handler UI Thread Background Thread GPS Start Http Request GPS Callback Click Handler App UI Dispatcher Download Callback App Highly asynchronous

87. Challenges Server App GetElapsedTime(); No single reference clock Variable network delays and processing delays PredictRemainingTime (responseSize); Transaction Highly asynchronous

88. Timecard Server App GetElapsedTime(); PredictRemainingTime (responseSize); Transaction Transaction Tracking TimeSync Delay Predictors

89. Timecard App Instrumenter Service Developer Timecard.dll GetElapsedTime(); PredictRemainingTime (responseSize); Desired user-perceived delay App Instrumented App Store

90. Timecard Server App GetElapsedTime(); PredictRemainingTime (responseSize); Transaction Transaction Tracking TimeSync Delay Predictors

91. Server Threads Spawn Workers Send Response Server Request Handler Transaction UI Thread Background Thread GPS Start Http Request GPS Callback Click Handler App UI Dispatcher Download Callback App Transaction Tracking

92. Server Threads Spawn Workers Send Response Server Request Handler UI Thread Background Thread GPS Start Http Request GPS Callback Click Handler App UI Dispatcher Download Callback App Transaction Tracking TC Transaction context (TC) attached to every thread – Carry along timestamps, transaction information TC GetElapsedTime();

93. Timecard Server App GetElapsedTime(); PredictRemainingTime (responseSize); Transaction Transaction Tracking TimeSync Delay Predictors

94. TimeSync GPS Sync with Cell Tower – Off my several seconds to minutes GPS – Does not work indoors Probing Probe

95. Typical probing techniques don’t work well – Radio wakeup delays and queuing delays – Errors more than 100ms – Energy inefficient (network radio tail energy) TimeSync Radio-aware, network-aware probing – Probes when radio is active and network is idle

96. TimeSync in Timecard Server App

97. TimeSync in Timecard Probe UI Thread Background Thread GPS Start Http Request GPS Callback Click Handler App ~5ms error Energy efficient

98. Timecard App Server GetElapsedTime(); Time elapsed since user interaction

99. Predicting Remaining Time App Server PredictRemainingTime (responseSize);  Downlink delay  App processing delay

100. Predicting Downlink Delay Latency Throughput Loss rate TCP window state Most transfers are short – Median – 3KB – 90% percentile – 37KB Analysis of 4000 apps Cellular networks – High-bandwidth, rare-loss, high-latency, 99% transfers over HTTP – TCP window state matters -> multiple RTTs Response size

101. Predicting Downlink Delay TCP Window 1 extra RTT

102. Predicting Downlink Delay Response size Latency TCP parameters Determined by  RTT  Number of round trips

103. Predicting Downlink Delay Use recent estimate of RTT – Use time sync probes Read TCP window state Read TCP window state at server Middlebox Do not work with middleboxes – Split-TCP connection TCP window state at the middlebox matters – No easy way to estimate TCP window state Build an empirical data-driven model RTT Number of round trips

104. Predicting Downlink Delay PredictRemainingTime (responseSize); Data-driven model – Response size – Recent RTT – Network provider – Bytes already transferred in the TCP connection o Proxy for TCP window state at the middlebox Middlebox Downlink Delay Predictor Cellular Median error - 17 ms, 90 th percentile - 86 ms Wi-Fi Median error - 12 ms, 90 th percentile - 31ms Learn

105. Predicting App Processing Delay App Server PredictRemainingTime (responseSize); Parsing and rendering delay – Correlated with response size – Correlated with device type

106. Timecard App Server GetElapsedTime(); PredictRemainingTime (responseSize); Time elapsed since user interaction Predicted downlink & app processing delay

107. Mobile Ads Service Contextual ads to mobile apps

108. Mobile Ads Service Fetch and process ads for keywords Extract keywordsRender ad Send keywords Ad

109. Mobile Ads Service Extract keywordsRender ad Send keywords Ad Ad provider A Keywords

110. Mobile Ads Service Extract keywordsRender ad Send keywords Ad

111. Mobile Ads Service Extract keywordsRender ad Send keywords Ad GetElapsedTime(); PredictRemainingTime (responseSize); Adapt Processing Time

112. Mobile Ads Service Within 50ms of the target delay 90% of the time With Timecard Without Timecard Target delay User-perceived delay (ms) ~200,000 transactions

113. Twitter Analysis App and Service Adapt server processing time in steps of 150ms Adapt response from 10KB to 60KB With Timecard Without Timecard Target delay User-perceived delay (ms)

114. Timecard App Server GetElapsedTime(); PredictRemainingTime (responseSize); Adapt Resources Used Desired user-perceived delay

115. Timecard App Server GetElapsedTime(); PredictRemainingTime (responseSize); Request Prioritization

116. AppInsight Timecard  Identify performance bottlenecks o Critical path analysis  Control user-perceived delays o Elapsed time estimation o Remaining time prediction  User transaction tracking o Asynchrony, Time Sync

117. By the time it loads, the church service is over. Too slow!!! Slow responsiveness, long load time, and crashity crash crash. horrifically slow. Uninstalled. Crashes every time. DO NOT DOWNLOAD. RIP-OFF!! Slow and Crashes often. Fix. I will give 5 stars instead of 1. So slow that my screen already dims before it's loaded Too slow - killing the usefulness when you really need to go Crashes all the time. Should be called Crashing News. Crashed 3 times before I gave up Slow or no network crashes the app immediately.

118. UI Thread Background Thread Exception at GetSentiment() DownloadCallback() Stack trace GetSentiment() … DownloadCallback() Http.AsyncGet(url) GPSCallback() ClickHandler() GPS.AsyncSample() User Interaction Exception Path App Crashes App Store Slow and Crashes often. DO NOT DOWNLOAD. RIP-OFF!! Need to uncover crashes before the app is released

119. VanarSena MobiSys ‘13 App testing as a service Upload app to our service After a short time, obtain a detailed report of potential crashes Fix issues before submitting to app store Key techniques: Binary instrumentation and user transaction tracking

120. 1.Top 10% crash buckets cover 90% of the crashes – Crash bucket: Unique exception type and system call Built on insights from real-world crash reports 3.Dominant root causes affect many different execution paths in the app 2.Most crashes due to externally-inducible root causes such as network failures Analysis of 25 million Windows Phone crash reports – from over 100,000 Windows Phone apps reported in 2012 VanarSena tests apps for common faults… … that are externally inducible… … as thoroughly as possible…

121. Dynamic testing using an army of monkeys Greybox testing – Binary instrumentation of app binaries To obtain detailed insight into app runtime behavior – Significantly reduces testing time VanarSena Fast UI automation – 4 – 40x reduction in testing time Efficient fault injection 38 sec vs. ~1 hour

122. ~3,000 unique crashes in 1,100 apps – Unique exception type + stack trace Note that, these are apps in the App Store – Presumably well tested Compared to Windows Phone crash database 19 out of 20 top exceptions covered Key Results Ongoing Product Impact at Microsoft

123. Apps OS Sensors Network VanarSena MobiSys ‘14 VanarSena MobiSys ‘14 AppInsight OSDI ‘12 AppInsight OSDI ‘12 Timecard SOSP ‘13 Timecard SOSP ‘13 Procrastinator MobiSys ‘14 Procrastinator MobiSys ‘14 Wireless ESP NSDI ‘11 Wireless ESP NSDI ‘11 VTrack SenSys ‘09 VTrack SenSys ‘09 CTrack NSDI ‘11 CTrack NSDI ‘11 SmartAds MobiSys ‘13 SmartAds MobiSys ‘13 CodeInTheAir HotMobile ‘12 CodeInTheAir HotMobile ‘12 App—aware Operating Systems Nectar OSDI ‘10 Nectar OSDI ‘10 Wearables Mobile Support Best Paper Best Paper Nominee

124. Backup

125. 0.1% runtime overhead Less than 1% memory overhead Less than 1% network overhead Negligible battery overhead Timecard Overhead

127. VanarSena MobiSys ‘13 Fast UI automation – Based on transaction tracking – 4x - 40x reduction in testing time App testing as a service Automatic dynamic testing – Using an army of monkeys Uncovered ~3000 crash bugs in 1100 apps in the app store