1. 1 Internet Networking and Application Troubleshooting Yao Zhao EECS Department Northwestern University

2. 2 Outline Motivation Dissertation Overview Application Layer Troubleshooting –Rake Network Layer Troubleshooting –VScope, Lend, FAD and SPA Conclusions and Future Work

3. 3 Motivation “When something breaks in the Internet, the Internet's very decentralized structure makes it hard to figure out what went wrong and even harder to assign responsibility.” - “Looking Over the Fence at Networks: A Neighbor's View of Networking Research”, by Committees on Research Horizons in networking, National Research Council, 2001.

4. 4 Troubleshooting Philosophy Entity Oriented Troubleshooting –Monitor entity separately E.g. Router packet drop rates, queue size and other SNMP counters E.g. Machine CPU load, I/O intensity, network utility and other performance counters –Potential problems Not all entities can be monitored Inferring entity performance from the counters may be challenging

5. 5 Troubleshooting Philosophy Entity Oriented Troubleshooting Task Based Troubleshooting –Use task performance to infer entity performance E.g. From Internet path loss rate to infer link-level loss rates –Advantage Work with limited monitor points (e.g. end hosts) Focus on target performance directly

6. 6 Thesis Statements We design troubleshooting systems that monitor and diagnosis the Internet distribute systems in both network layer and application layer using the task based troubleshooting philosophy.

7. 7 Publications Conference Papers –Y. Zhao, Y. Chen, S. Ratnasamy, Load balanced and Efficient Hierarchical Data-Centric Storage in Sensor Networks, in the Proc. of SECON 2008 –Y. Gao, Y. Zhao, R. Schweller, S. Venkataraman, Y. Chen, D. Song, and M. Kao, Detecting Stealthy Spreaders Using Online Outdegree Histograms, in the Proc. of IWQos, 2007Detecting Stealthy Spreaders Using Online Outdegree Histograms –Y. Zhao and Y. Chen, A Suite of Schemes for User-level Network Diagnosis without Infrastructure, in the Proc. of IEEE INFOCOM, 2007A Suite of Schemes for User-level Network Diagnosis without Infrastructure –P. Narayana, R. Chen, Y. Zhao, Y. Chen, Z. Fu, and H. Zhou, Automatic Vulnerability Checking of IEEE 802.16 WiMAX Protocols through TLA+, in Proc. of NPSec, 2006 Automatic Vulnerability Checking of IEEE 802.16 WiMAX Protocols through TLA+ –Y. Zhao, Y. Chen, and D. Bindel, Towards Unbiased End-to-End Network Diagnosis, in Proc. of ACM SIGCOMM 2006Towards Unbiased End-to-End Network Diagnosis –Y. Zhao, Q. Zhang, B. Li, Y. Chen and W. Zhu, Hop ID based Routing in Mobile Ad Hoc Networks, in Proceedings of ICNP, 2005Hop ID based Routing in Mobile Ad Hoc Networks

8. 8 Publications Journal Papers –Y. Zhao and Y. Chen, FAD and SPA: A Suite of Schemes for User-level Network Diagnosis without Infrastructure, submitted to Computer Networks Journal and the last review feedback is minor revision.FAD and SPA: A Suite of Schemes for User-level Network Diagnosis without Infrastructure –Y. Zhao, Y. Chen, and D. Bindel, Towards Unbiased End-to-End Network Diagnosis, submitted to IEEE/ACM Transactions on Networking and the last review feedback is minor revisionTowards Unbiased End-to-End Network Diagnosis –Y. Zhao, Y. Chen, B. Li, Q. Zhang, Hop ID: A Virtual Coordinate based Routing for Sparse Mobile Ad Hoc Networks, in IEEE Transaction on Mobile Computing, Vol. 6, No. 9, pp. 1075-1089, September, 2007 Patents –E. C. Gillum, Q. Ke, Y. Xie, F. Yu and Y. Zhao, Graph Based Bot-User Detection, being filed through Microsoft Corporation, MS docket number 324953.01. –J. Wang, Y. Chen, D. Pei, Y. Zhao, and Z. Zhu, Towards Efficient Large- Scale Network Monitoring and Diagnosis Under Operational Constraints, being filed through AT&T, docket number 1209-144.

9. 9 Outline Motivation Dissertation Overview Application Layer Troubleshooting –Rake Network Layer Troubleshooting –VScope, Lend, FAD and SPA Conclusions and Future Work

10. 10 Internet Troubleshooting Diagnosis Model Data Link Network Transport Application Monitoring

11. 11 Components in Network Troubleshooting Model –Defines the extrinsic observations and intrinsic faulty problems as well as the relationship between them Define and instantiate the model Monitoring –Collect the observations Diagnosis –Identify the faulty location and find out the root cause

12. 12 Thesis Research Topics Diagnosis Model Data Link Network Transport Application Monitoring Lend, FAD and SPA VScop e Rake

13. 13 Outline Motivation Dissertation Overview Application Layer Troubleshooting –Rake Network Layer Troubleshooting –VScope, Lend, FAD and SPA Conclusions and Future Work

14. 14 Rake: Semantic Assisted Large Distributed System Diagnosis Motivation Related Work Rake Evaluation Conclusions

15. 15 Motivation Large distributed systems involve hundreds or thousands of nodes –E.g. search system, CDN Host-based monitoring cannot infer the performance or detect bugs –Hard to translate OS-level info (such as CPU load) into application performance –Application log may not be enough Task-based approach is adopted in many diagnosis systems –WAP5, Magpie, Sherlock

16. 16 Task-based Approaches The Critical Problem – Message Linking –Link the messages in a task together into a path or tree

17. 17 Example of Message Linking in Search System URL Search keyword URL Search keyword Doc ID

18. 18 Task-based Approaches The Critical Problem – Message Linking –Link the messages in a task together into a path or tree Black-box approaches –Do not need to instrument the application or to understand its internal structure or semantics –Time correlation to link messages Project 5, WAP5, Sherlock White-box approaches –Extracts application-level data and requires instrumenting the application and possibly understanding the application's source codes –Insert a unique ID into messages in a task X-Trace, Pinpoint

19. 19 Problems of White-box and Black-Box White-box –Invasive due to source code modification Black-box –Rely on time Correlation –Accuracy affected by cross traffic

20. 20 Related Work Non-InvasiveInvasive Network Sniffing Interpo- sition App or OS Logs Source code modification Black-box Project 5, Sherlock WAP5Footprint Grey-boxRakeMagpie White-box X-Trace, Pinpoint Invasiveness Application Knowledge

21. 21 Rake Key Observations –Generally no unique ID linking the messages associated with the same request –Exist polymorphic IDs in different stages of the request Semantic Assisted –Use the semantics of the system to identify polymorphic IDs and link messages

22. 22 Message Linking Example URL Search keyword URL Search keyword Doc ID

23. 23 Questions on Semantics What Are the Necessary Semantics? –In worst case, re-implement the application How Does Rake Use the Semantics? –Naïve design is to implement Rake for each application with specific application semantics How Efficient Is the Rake with Semantics –Can message linking to accurate? –What’s the computational complexity of Rake?

24. 24 Necessary Semantics Intra-node linking –The system semantics Inter-node link –The protocol semantics Node P Q R S

25. 25 Utilize Semantics in Rake Implement Different Rakes for Different Application is time consuming –Lesson learnt for implementing two versions of Rake for CoralCDN and IRC Design Rake to take general semantics –A unified infrastructure –Provide simple language for user to supply semantics

26. 26 Example of Rake Language (IRC) TCP 6667 Regular expression PRIVMSG\s+(.*) Same as Link ID No Return ID P Q R S Link_IDFollow_ID = Query_ID = Response_ID

27. 27 Signature Signature to Classify Messages – TCP 6667 – Formats of Signatures –Socket information Protocol, port –Expression for TCP/IP header udp [10]&128==0 –Regular expression –User defined function

28. 28 Link_ID and Follow_ID Follow_IDs –The IDs will be in the triggered messages by this message –One message may have multiple Follow_IDs for triggering multiple messages Link_ID –The ID of the current message –Match with Follow_ID previously seen Linking of Link_ID and Follow_ID –Mainly for intra-node message linking

29. 29 Query_ID and Response_ID Query_IDs –The communication is in Query/Response style, e.g. RPC call and DNS query/response. –The IDs will be in the response messages to this message Response_ID –The ID of the current message to match Query_ID previously seen –By default requires the query and response to use the same socket Linking of Query_ID and Response_ID –Mainly for inter-node message linking

30. 30 Complicated Semantics The process of generating IDs may be complicated –XML or regular expression is not good at complex computations –So let user provide own functions User provide share/dynamic libraries Specify the functions for IDs in XML Implementation using Libtool to load user defined function in runtime

31. 31 Example for DNS UDP 53 udp[10] & 128 == 0 User Function dns.so Link_ID Link_ID Link_ID …………………………….. Extract the queried host

32. 32 Accuracy Analysis One-to-one ID Transforming –Examples In search, URL -> Keywords -> Canonical format In CoralCDN, URL -> Sha1 hash value –Ideally no error if requests are distinct Request ambiguousness –Search keywords Microsoft search data Less than 1% messages with duplication in 1s –Web URL Two real http traces Less than 1% messages with duplication in 1s –Chat messages No duplication with timestamps

33. 33 Potential Applications Search –Verified by a Microsoft guy CDN –CoralCDN is studied and evaluated Chat System –IRC is tested Distributed File System –Hadoop DFS is tested

34. 34 Evaluation Application –CoralCDN –Deployed on PlanetLab Experiment –Employ PlanetLab hosts as web clients –Retrieve URLs from real traces with different frequency Metrics –Linking accuracy (false positive, false negative) –Diagnosis ability Compared Approach –WAP5

35. 35 CoralCDN Task Tree

36. 36 Message Linking Accuracy Rake Linking Accuracy is 100% for CoralCDN –Sha1 hash provides almost one-to-one URL to HashID mapping –The cache mechanism If the same URL is received twice, the 2 nd one will be blocked until the first one retrieves back the webpage Use Rake Linking as Ground Truth to Evaluate WAP5

37. 37 Message Linking Accuracy (1) The higher request rate, the less accuracy in WAP5.

38. 38 Message Linking Accuracy (1) The higher request rate, the less accuracy in WAP5.

39. 39 Diagnosis Ability Controlled Experiments –Inject junk CPU-intensive processes –Calculated the packet processing time using WAP5 and Rake Obviously Rake can identify the slow machine, while WAP5 fails.

40. 40 Discussion Implementation Experience –How hard for user to provide semantics CoralCDN – 1 week source code study DNS – a couple of hours Hadoop DFS – 1 week source code study Inter-process Communication Encryption –Dynamic library interposition

41. 41 Conclusions of Rake Feasibility –Rake works for many popular applications in different categories Easiness –Rake allows user to write semantics via XML –Necessary semantics are easy to obtained given our experience Accuracy –Much more accurate than black-box approaches and probably matches white-box approaches

42. 42 Outline Motivation Dissertation Overview Application Layer Troubleshooting –Rake Network Layer Troubleshooting –VScope, Lend, FAD and SPA Conclusions and Future Work

43. 43 Network Layer Troubleshooting LEND [Sigcomm06] –Tomography Diagnosis with least statistic assumptions FAD & SPA [Infocom05] –On-demand loss rate diagnosis without infrastructure VScope [Patent] –Experimental design for ISP VPN network monitoring and diagnosis

44. 44 LEND Motivation –Use end-to-end measurement to infer link level properties with the measurement infrastructure Problem Formulation –Given end-to-end measurements, what is the finest granularity of link properties can we achieve under the basic assumptions? Basic assumptions More and stronger statistic assumptions Virtual link Diagnosis granularity? Better accuracy

45. 45 LEND Contributions –Define the minimal identifiable unit under basic assumptions (MILS) –Prove that only E2E paths are MILS with a directed graph topology (e.g., the Internet) –Propose good path algorithm (incorporating measurement path properties) for finer MILS Basic assumptions More and stronger statistic assumptions Virtual link Diagnosis granularity? Better accuracy

46. 46 FAD & SPA Motivation –How do end users, with no special privileges, identify packet loss inside the network with one or two computers? Conclusions –We proposed three user-level loss rate diagnosis approaches –The combo of our approaches and Tulip [SOSP03] is much better than any single approach

47. 47 VScope Challenges in ISP Network Monitoring and Diagnosis –Operational constraints on monitors and links A monitor can measure a certain number of paths at a time Measurement traffic through a link cannot exceed a threshold Path and monitor selection constraints –Real-time diagnosis Conclusions –Propose multi-round monitoring scheme for the monitor selection problem under operational constraints. –Propose continuous monitoring and diagnosis algorithm to quickly diagnose faulty links

48. 48 Outline Motivation Dissertation Overview Application Layer Troubleshooting –Rake Network Layer Troubleshooting –VScope, Lend, FAD and SPA Conclusions and Future Work

49. 49 Conclusions and Future Work Demonstrate Task-based Troubleshooting Is Promising –Network layer troubleshooting VScope, LEND, FAD and SPA –Application layer troubleshooting Rake Future Work –Make Rake a tool ready to publish –Extend Rake in diagnosis Timeline for Thesis Writing –From present to Feb. 1

50. 50 Q & A? Thanks!

51. 51

52. 52 Backup

53. 53 Monitor Setup Phase Single-round Monitoring –Measure all the target paths simultaneously –Basic and is adopted by most monitoring experimental design papers Multi-round Monitoring –Measure all the target paths in different time period (round) Tradeoff between time and link/node constraints –Multi-round Monitoring is necessary and efficient for two reasons Existing of operational constraints Star-like topology

54. 54 Single-Round Monitor Selection Pure Greedy Algorithm –Select monitors one by one and every time select the monitor that can measure most uncovered links under the constraints To calculate the gain of adding a new monitor is a variant of Maximum k-Coverage problem –Simple and local optimized Greedy Assisted Linear Programming based algorithm

55. 55 Greedy Assisted Linear Programming based algorithm Formulate Integer Linear Programming First –ILP is NP-hard problem Relaxation to Linear Programming –Change all {0,1}-variable to continuous variable between 0 and 1 Random Rounding –Solve the linear programming in polinomial time –Round the solutions within [0, 1] back to {0,1}-integers with certain probabilities

56. 56 Multi-round Monitor Selection Star-like Topology and Operation Constraints Make Single-round Monitor Selection Inefficient –Multi-round monitoring vs Reducing measurement frequency Algorithms for Multi-round Monitor Selection –Multiple the constraints with the round number and run single-round monitor selection –Schedule the paths to measure in different rounds Greedy scheduling Random scheduling Linear programming based scheduling

57. 57 Path Measurement Scheduling Greedy algorithm –Minimize link utilization in every step Random algorithm –Randomly schedule paths independently –Run random algorithm multiple times to get the best one Linear Programming based algorithm with random rounding

58. 58 Monitoring and Diagnosis Path Monitoring and Faulty Path Discovery Faulty Link Diagnosis –Select and measure some paths which favor of the diagnosis of the potential faulty links

59. 59 Background and Related Work Network Layer Diagnosis –Linear algebraic model –Monitoring experimental design –Diagnosis algorithms Application Layer Diagnosis –Sherlock: enterprise network service diagnosis

60. 60 Linear Algebraic Model Path loss rate p i, link loss rate l j : A D C B 1 2 3 p1p1 p2p2 Usually an underconstrained system G

61. 61 Monitoring Experimental Design Monitor Placement Problem –Select least monitors that can measure some paths covering all the links [Infocom03] Path Selection Problem –Selection of the basis of the path matrix [Sigcomm04] –SVD based path selection [Infocom05] –Bayesian experimental design [Sigmetrics06] Network Layer Diagnosis

62. 62 Network Layer Diagnosis Internet Tomography –Temporal correlations based algorithms Unbiased if multicast is supported –Statistic algorithms Introducing additional statistic assumption or optimization goal 0.1 0

63. 63 Publications Papers –Y. Zhao, Y. Chen, S. Ratnasamy, Load balanced and Efficient Hierarchical Data- Centric Storage in Sensor Networks, in the Proc. of SECON 2008 –Y. Gao, Y. Zhao, R. Schweller, S. Venkataraman, Y. Chen, D. Song, and M. Kao, Detecting Stealthy Spreaders Using Online Outdegree Histograms, in the Proc. of IWQoS, 2007 Detecting Stealthy Spreaders Using Online Outdegree HistogramsIWQoS, 2007 –Y. Zhao and Y. Chen, A Suite of Schemes for User-level Network Diagnosis without Infrastructure, in the Proc. of IEEE INFOCOM, 2007A Suite of Schemes for User-level Network Diagnosis without InfrastructureIEEE INFOCOM, 2007 –P. Narayana, R. Chen, Y. Zhao, Y. Chen, Z. Fu, and H. Zhou, Automatic Vulnerability Checking of IEEE 802.16 WiMAX Protocols through TLA+, in Proc. of NPSec, 2006Automatic Vulnerability Checking of IEEE 802.16 WiMAX Protocols through TLA+NPSec, 2006 –Y. Zhao, Y. Chen, and D. Bindel, Towards Unbiased End-to-End Network Diagnosis, in Proc. of ACM SIGCOMM 2006Towards Unbiased End-to-End Network DiagnosisACM SIGCOMM 2006 –Y. Zhao, Q. Zhang, B. Li, Y. Chen and W. Zhu, Hop ID based Routing in Mobile Ad Hoc Networks, in Proceedings of ICNP, 2005Hop ID based Routing in Mobile Ad Hoc NetworksICNP, 2005 Patents –E. C. Gillum, Q. Ke, Y. Xie, F. Yu and Y. Zhao, Graph Based Bot-User Detection, being filed through Microsoft Corporation, MS docket number 324953.01. –J. Wang, Y. Chen, D. Pei, Y. Zhao, and Z. Zhu, Towards Efficient Large-Scale Network Monitoring and Diagnosis Under Operational Constraints, being filed through AT&T, docket number 1209-144.

64. 64 Problem Definition (2) Monitor Setup Phase –From certain monitor candidates select minimal number of monitors, which in the measurement phase can measure a certain path set that covers all links in the network under the given measurement constraints –NP-hard even without considering constraints Monitoring and Fault Diagnosis Phase –When faulty paths are discovered in the path monitoring phase, how to quickly select some paths under the operational constraints to be further measured so that the faulty link(s) can be accurately identified?

65. 65 VScope Motivation Two Important Services Provided by ISP –Internet access service –VPN service Monitoring and Diagnosis on ISP Networks –Ensure Service Level Agreement (SLA) –Help Network Operations