End-to-End Performance Analytics For Mobile Apps Lenin Ravindranath, Jitu Padhye, Ratul Mahajan Microsoft Research 1

“Slower than a snail.” “Slow and unresponsive like mud” “Consistently 3 seconds behind where I touch.” “So slow. Did an intern write this app??” “Very very slow compared to even browsing web.” “Sluggish and freezes my HTC phone.” Apps are Slow 2

Cloud Services Mobile Apps Most Mobile Apps talk to the Cloud 3

Cloud Service Location Restaurants 4

AppInsight App Instrumented Instrumenter ClickHandler() { [Instrument] LocationStart(LocationCallback); } LocationCallback(location) { [Instrument] url = GetRequestUrl(location); [Instrument] WebRequest(url, WebResponseCallback); } WebResponseCallback(data) { [Instrument] list = ParseData(data); [Instrument] UIDispatch(Display, list); } Display(list) { [Instrument] UIListBox.DataContext = list; } Automatic Instrumentation 5

AppInsight User click Web RequestWeb Response Web Request delay User Perceived Delay GPS delay Parsing and Rendering delay 6

AppInsight User click Web RequestWeb Response User Perceived Delay Where is the bottleneck? Network? Server? DB? Uplink? Downlink? 7

End-to-End Insight User click Web RequestWeb Response User Perceived Delay Server Uplink delay Downlink delay Server delay Query Processing delay DB GPS delay Parsing and Rendering delay 8

EndInsight Joint App and Server Instrumentation – Windows Phone Apps, ASP.NET Services – Automatic Instrumentation – Zero developer effort – Binary Instrumentation Readily deployable – No changes to the OS, runtime or platform Low overhead 9

App App Instrumenter Instrumented App Service Instrumenter Service Instrumented Service Developer Cloud App Store Analysis Server Developer Feedback Traces Downloads 10

App Instrumentation User click Web RequestWeb Response Web Request delay GPS delay Parsing and Rendering delay Using AppInsight 11

AppInsight Recap Track User Transactions – From user click to UI rendering Across asynchronous thread boundaries Automatically track – UI manipulation – Threads, asynchronous calls – Web calls, Sensor APIs – UI update (dispatcher events) User request UI Update UI Thread Background thread GPS fix Callback UI dispatch Background thread Web Request Response Callback App GPS Start 12

Server Instrumentation Automatic Instrumentation of.NET binaries Azure services – Web APIs, ASPX pages, MVC, WCF Automatically track – Request-Response entry points – DB queries – Network calls Across asynchronous boundaries – Similar to app instrumentation class AroundMeController: ApiController { public Result Get(type, location) { [Instrument] return GetList(type, location); [Instrument] } public Result GetList(type, location) { [Instrument] result = DBQuery(type, location); [Instrument] return result; } 13

Server Instrumentation Server delay Query Processing delay DB Request Response Request Processing Delay 14

Putting them together? Server delay Query Processing delay DB Request Response Web RequestWeb Response App Performance Server Performance 15

Matching App and Server Traces Pass a Unique Id from App to Server Add a new HTTP header in web requests – x-EndInsight-Id App Instrumentation – Instrument HTTP request calls – Generate unique Id when app makes a web request – Add x-EndInsight-Id header in web requests Server Instrumentation – Log the unique id and remove the extra header at the request entry point 16

Matching App and Server Traces User click Web RequestWeb Response Server DB Add header [x-EndInsight-Id] = “59bc73e a7-41ac86c6c039” Log and Remove header [x-EndInsight-Id] 17

DB Request Response Web RequestWeb Response Finding Network Delays Server Request Processing Delay App Web Request Delay Network Delay = [App Web Request Delay] – [Server Request Processing Delay] 18

Separating Uplink and Downlink Delays Server Uplink delay Downlink delay DB Requires knowing the clock offset and drift between Client and Server Different Clocks Timestamp 19

Finding Clock Offset and Drift NTP-like Technique Probes from client timestamped by the server Timestamp t s Timestamp t e Timestamp s Clock Offset = s – ((t s + t e ) / 2) Clock Drift = Slope of linear regression of many such probes More probes, better estimate 20

Automatic Instrumentation of App – To send periodic probes to server Automatic Instrumentation of Server – Open up API to timestamp probes Probing Efficient technique for probing – Aware of the radio state and traffic – Minimal extra delays – Energy efficient 21

Network Delays Uplink – DNS – Connection setup Downlink Analysis App Delays App processing Sensor processing Parsing/Rendering Radio wakeup Disk access Server Delays Server Processing Query Processing Automatic analysis of end-to-end transaction delays 22

Critical Path Analysis 23

Group similar transactions Outliers – Points to corner cases Highlight common critical paths – Focus development effort Root causes of Performance Variability – Highlight factors affecting performance Aggregate Analysis 24

Web based Interface Developer Feedback 25

Compute Network Memory Battery low 2% 4% <0.1% Impact on app and server performance is minimal Low resource consumption EndInsight Overhead 26

Timecard: Controlling End-to-End Delays 27

Servers Control Variability in Response Times Request Response Fixed deadlines Trade-off quality for response time – More time to compute, better quality results – Flexible Services Deadline Server processing 28

The elephant is outside the room Request Response Server processing 29

The elephant is outside the room User click Server processing App Server Request Response UplinkDownlink 30

Unaware of end-to-end delays Clients with non-trivial external delays – Poor end-to-end response times Client with minimal external delays – Do not produce the best quality result Could adapt differently for different users Servers have no visibility into delays outside 31

Timecard GetElapsedTime(); PredictDownstreamTime ( responseSize ); Time elapsed since user request Predicted downlink + app processing delay App Server 32

Timecard App Server Desired end-to-end delay App GetElapsedTime(); PredictDownstreamTime ( responseSize ); Adapt Processing Time 33

Timecard App Server Desired end-to-end delay App GetElapsedTime(); PredictDownstreamTime ( responseSize ); Adapt Response 34

Timecard App Server Desired end-to-end delay App GetElapsedTime(); PredictDownstreamTime ( responseSize ); 35

Online Transaction Tracking Time Synchronization Downstream Delay Predictor – Downlink delay – App processing delay Timecard 36

Timecard Implementation App Server Transaction Tracking Transaction Tracking Transaction Tracking Transaction Tracking Logging Predictor Time Sync Data Collection Automatic Instrumentation WP Apps ASP.NET services APIs 37