Performance Debugging for Distributed Systems of Black Boxes Yinghua Wu, Haiyong Xie Yinghua Wu, Haiyong Xie

Outline Problem statement & goals Problem statement & goals Overview of our approach Overview of our approach Algorithms Algorithms The nesting algorithm (RPC) The nesting algorithm (RPC) The convolution algorithm (RPC or free-form) The convolution algorithm (RPC or free-form) Experimental results Experimental results Visualization GUI Visualization GUI Related work Related work Conclusions Conclusions

Motivation Complex distributed systems Complex distributed systems Built from black box components Built from black box components Heavy communications traffic Heavy communications traffic Bottlenecks at some specific nodes Bottlenecks at some specific nodes These systems may have performance problems These systems may have performance problems High or erratic latency High or erratic latency Caused by complex system interactions Caused by complex system interactions Isolating performance bottlenecks is hard Isolating performance bottlenecks is hard We cannot always examine or modify system components We cannot always examine or modify system components We need tools to infer where bottlenecks are We need tools to infer where bottlenecks are Choose which black boxes to open Choose which black boxes to open

Example multi-tier system client web server client web server database server database server application server application server authentication server 100ms

Goals Isolating performance bottlenecks Isolating performance bottlenecks Find high-impact causal path patterns Find high-impact causal path patterns Causal path: series of nodes that sent/received messages. Each message is caused by receipt of previous message, and Some causal paths occur many times Causal path: series of nodes that sent/received messages. Each message is caused by receipt of previous message, and Some causal paths occur many times High-impact: occurs frequently, and contributes significantly to overall latency High-impact: occurs frequently, and contributes significantly to overall latency Identify high-latency nodes on high-impact patterns Identify high-latency nodes on high-impact patterns Add significant latency to these patterns Add significant latency to these patterns Then What should We do? Messages Trace is enough Messages Trace is enough

The Black Box Desired properties Desired properties Zero-knowledge, zero-instrumentation, zero-perturbation Zero-knowledge, zero-instrumentation, zero-perturbation Scalability Scalability Accuracy Accuracy Efficiency (time and space) Efficiency (time and space) Complex distributed system built from “black boxes” Performance bottlenecks

Outline Problem statement & goals Problem statement & goals Overview of our approach Overview of our approach Algorithms Algorithms The nesting algorithm (RPC) The nesting algorithm (RPC) The convolution algorithm (RPC or free-form) The convolution algorithm (RPC or free-form) Experimental results Experimental results Visualization GUI Visualization GUI Related work Related work Conclusions Conclusions

Overview of Approach Obtain traces of messages between components Obtain traces of messages between components Ethernet packets, middleware messages, etc. Ethernet packets, middleware messages, etc. Collect traces as non-invasively as possible Collect traces as non-invasively as possible Require very little information: Require very little information: [timestamp, source, destination, call/return, call-id] Analyze traces using our algorithms Analyze traces using our algorithms Nesting: faster, more accurate, limited to RPC-style systems Nesting: faster, more accurate, limited to RPC-style systems Convolution: works for all message-based systems Convolution: works for all message-based systems Visualize results and highlight high-impact paths Visualize results and highlight high-impact paths

Recap. causal path client web server client web server database server database server application server application server authentication server 100ms

Challenges Trace contain interleaved messages from many causal paths Trace contain interleaved messages from many causal paths How to identify causal paths? How to identify causal paths? Causality trace by Timestamp Causality trace by Timestamp Want only statistically significant causal paths Want only statistically significant causal paths How to differentiate significance? How to differentiate significance? It is easy! They appear repeatedly It is easy! They appear repeatedly

Outline Problem statement & goals Problem statement & goals Overview of our approach Overview of our approach Algorithms Algorithms The nesting algorithm (RPC) The nesting algorithm (RPC) The convolution algorithm (RPC or free-form) The convolution algorithm (RPC or free-form) Experimental results Experimental results Visualization GUI Visualization GUI Related work Related work Conclusions Conclusions

The nesting algorithm Depends on RPC-style communication Depends on RPC-style communication Infers causality from “nesting” relationships by message timestamps Infers causality from “nesting” relationships by message timestamps Suppose A calls B and B calls C before returning to A Suppose A calls B and B calls C before returning to A Then the B  C call is “nested” in the A  B call Then the B  C call is “nested” in the A  B call Uses statistical correlation Uses statistical correlation time node A node B node C call return

Nesting: an example causal path AB C D Consider this system of 4 nodes time node A node B node C call return call return node D return Looking for internal delays at each node

Steps of the nesting algorithm 1. Pair call and return messages (A  B, B  A), (B  D, D  B), (B  C, C  B) (A  B, B  A), (B  D, D  B), (B  C, C  B) 2. Find and score all nesting relationships B  C nested in A  B B  C nested in A  B B  D also nested in A  B B  D also nested in A  B 3. Pick best parents Here: unambiguous Here: unambiguous 4. Reconstruct call paths A  B  [C ; D] A  B  [C ; D] O(m) run time O(m) run time m = number of messages m = number of messages time node A node B node C call return call return node D return

Pseudo-code for the nesting algorithm Detects calls pairs and find all possible nestings of one call pair in another Detects calls pairs and find all possible nestings of one call pair in another Pick the most likely candidate for the causing call for each call pair Pick the most likely candidate for the causing call for each call pair Derive call paths from the causal relationships Derive call paths from the causal relationships procedure FindCallPairs for each trace entry (t1, CALL/RET, sender A, receiver B, callid id) case CALL: store (t1,CALL,A,B,id) in Topencalls case RETURN: find matching entry (t2, CALL, B, A, id) in Topencalls if match is found then remove entry from Topencalls update entry with return message timestamp t2 add entry to Tcallpairs entry.parents := {all callpairs (t3, CALL, X, A, id2) in Topencalls with t3 < t2} procedure ScoreNestings for each child (B, C, t2, t3) in Tcallpairs for each parent (A, B, t1, t4) in child.parents scoreboard[A, B, C, t2-t1] += (1/|child.parents|) procedure FindNestedPairs for each child (B; C; t2; t3) in call pairs maxscore := 0 for each p (A, B, t1, t4) in child.parents score[p] := scoreboard[A, B, C, t2-t1]*penalty if (score[p] > maxscore) then maxscore := score[p] parent := p parent.children := parent.children U {child} procedure FindCallPaths initialize hash table Tpaths for each callpair (A, B, t1, t2) if callpair.parents = null then root := { CreatePathNode(callpair, t1) } if root is in Tpaths then update its latencies else add root to Tpaths function CreatePathNode(callpair (A, B, t1, t4), tp) node := new node with name B node.latency := t4 - t1 node.call_delay := t1 - tp for each child in callpair.children node.edges := node.edges U { CreatePathNode(child, t1)} return node

Inferring nesting − Local info not enough − Use aggregate info − Histograms keep track of possible latencies − Medium-length delay will be selected − Assign nesting − Heuristic methods time node A node B node C An example of Parallel calls An example of Parallel calls t1 t2 t3 t4

Outline Problem statement & goals Problem statement & goals Overview of our approach Overview of our approach Algorithms Algorithms The nesting algorithm The nesting algorithm The convolution algorithm The convolution algorithm Experimental results Experimental results Visualization GUI Visualization GUI Related work Related work Conclusions Conclusions

The convolution algorithm “Time signal” of messages for each “Time signal” of messages for each A sent message to B at times 1,2,5,6,7 A sent message to B at times 1,2,5,6,7 S1(t)= A  B messages time

The convolution algorithm Look for time-shifted similarities Look for time-shifted similarities Compute convolution X(t) = S 2 (t)  S 1 (t) Compute convolution X(t) = S 2 (t)  S 1 (t) Use Fast Fourier Transforms Use Fast Fourier Transforms S 1 (t) (A  B) S 2 (t) (B  C) X(t) A  BB  C Peaks in X(t) suggest causality between A  B and B  C Time shift of a peak indicates delay

Convolution details Time complexity: O(em+eVlogV) Time complexity: O(em+eVlogV) m = messages m = messages e = output edges e = output edges V = number of time steps in trace V = number of time steps in trace Need to choose time step size Need to choose time step size Must be shorter than delays of interest Must be shorter than delays of interest Too coarse: poor accuracy Too coarse: poor accuracy Too fine: long running time Too fine: long running time Robust to noise in trace Robust to noise in trace

Algorithm comparison Nesting Nesting Looks at individual paths and then aggregates Looks at individual paths and then aggregates Finds rare paths Finds rare paths Requires call/return style communication Requires call/return style communication Fast enough for real-time analysis Fast enough for real-time analysis Convolution Convolution Applicable to a broader class of systems Applicable to a broader class of systems Slower: more work with less information Slower: more work with less information May need to try different time steps to get good results May need to try different time steps to get good results Reasonable for off-line analysis Reasonable for off-line analysis

Communication style Rare events Level of Trace detail Time and space complexity Visualization Nesting AlgorithmConvolution Algorithm RPC onlyRPC or free-form messages Yes, but hardNo + call/return tag Linear space Linear time Linear space Polynomial time RPC call and return combined  More compact Less compact Summarize

Outline Problem statement & goals Problem statement & goals Overview of our approach Overview of our approach Algorithms Algorithms Experimental results Experimental results Maketrace: a trace generator Maketrace: a trace generator Maketrace web server simulation Maketrace web server simulation Pet Store EJB traces Pet Store EJB traces Execution costs Execution costs Visualization GUI Visualization GUI Related work Related work Conclusions Conclusions

Maketrace Synthetic trace generator Synthetic trace generator Needed for testing Needed for testing Validate output for known input Validate output for known input Check corner cases Check corner cases Uses set of causal path templates Uses set of causal path templates All call and return messages, with latencies All call and return messages, with latencies Delays are x ± y seconds, Gaussian normal distribution Delays are x ± y seconds, Gaussian normal distribution Recipe to combine paths Recipe to combine paths Parallelism, start/stop times for each path Parallelism, start/stop times for each path Duration of trace Duration of trace

Desired results for one trace Causal paths Causal paths How often How often How much time spent How much time spent Nodes Nodes Host/component name Host/component name Time spent in node and all of the nodes it calls Time spent in node and all of the nodes it calls Edges Edges Time parent waits before calling child Time parent waits before calling child

Measuring Added Delay Added 200msec delay in WS2 Added 200msec delay in WS2 The nesting algorithm detects the added delay, and so does the convolution algorithm The nesting algorithm detects the added delay, and so does the convolution algorithm

Results: Petstore Sample EJB application Sample EJB application J2EE middleware for Java J2EE middleware for Java Instrumentation from Stanford’s PinPoint project Instrumentation from Stanford’s PinPoint project 50msec delay added in mylist.jsp 50msec delay added in mylist.jsp

Results: running time TraceLength (messages) Duration (sec) Memory (MB) CPU time (sec) Nesting Multi-tier (short)20, Multi-tier202, Multi-tier (long)2,026,6585, PetStore234,0362, Convolution (20 ms time step) PetStore234,0362, , More details and results in paper

Accuracy vs. parallelism Increased parallelism degrades accuracy slightly Increased parallelism degrades accuracy slightly Parallelism is number of paths active at same time Parallelism is number of paths active at same time

Other results for nesting algorithm Clock skew Clock skew Little effect on accuracy with skew ≤ delays of interest Little effect on accuracy with skew ≤ delays of interest Drop rate Drop rate Little effect on accuracy with drop rates ≤ 5% Little effect on accuracy with drop rates ≤ 5% Delay variance Delay variance Robust to ≤ 30% variance Robust to ≤ 30% variance Noise in the trace Noise in the trace Only matters if same nodes send noise Only matters if same nodes send noise Little effect on accuracy with ≤ 15% noise Little effect on accuracy with ≤ 15% noise

Visualization GUI Goal: highlight dominant paths Goal: highlight dominant paths Paths sorted Paths sorted By frequency By frequency By total time By total time Red highlights Red highlights High-cost nodes High-cost nodes Timeline Timeline Nested calls Nested calls Dominant subcalls Dominant subcalls Time plots Time plots Node time Node time Call delay Call delay

Related work Systems that trace end-to-end causality via modified middleware using modified JVM or J2EE layers Systems that trace end-to-end causality via modified middleware using modified JVM or J2EE layers Magpie (Microsoft Research), aimed at performance debugging Magpie (Microsoft Research), aimed at performance debugging Pinpoint (Stanford/Berkeley), aimed at locating faults Pinpoint (Stanford/Berkeley), aimed at locating faults Products such as AppAssure, PerformaSure, OptiBench Products such as AppAssure, PerformaSure, OptiBench Systems that make inferences from traces Systems that make inferences from traces Intrusion detection (Zhang & Paxson, LBL) uses traces + statistics to find compromised systems Intrusion detection (Zhang & Paxson, LBL) uses traces + statistics to find compromised systems

Future work Automate trace gathering and conversion Automate trace gathering and conversion Sliding-window versions of algorithms Sliding-window versions of algorithms Find phased behavior Find phased behavior Reduce memory usage of nesting algorithm Reduce memory usage of nesting algorithm Improve speed of convolution algorithm Improve speed of convolution algorithm Validate usefulness on more complicated systems Validate usefulness on more complicated systems

Conclusions Looking for bottlenecks in black box systems Looking for bottlenecks in black box systems Finding causal paths is enough to find bottlenecks Finding causal paths is enough to find bottlenecks Algorithms to find paths in traces really work Algorithms to find paths in traces really work We find correct latency distributions We find correct latency distributions Two very different algorithms get similar results Two very different algorithms get similar results Passively collected traces have sufficient information Passively collected traces have sufficient information