Differential Provenance: Better Network Diagnostics with Reference Events Ang Chen Yang Wu Andreas Haeberlen Wenchao Zhou+ Boon Thau Loo University of Pennsylvania Georgetown University+

Motivation: Finding the root cause of a symptom Traffic arriving at the wrong server !?! Overly specific flow entry 4.3.2.0/24 4.3.3.0/24 Internet Bob Web server 2 Web server 1 DPI Networks can (and frequently do!) have bugs Example: Software-defined networks We need a good debugger!

Debugging networks with provenance C received packet Packet P Packet P B sent packet A B C B received packet Rule match on B Rule installed by controller A sent packet A received packet Rule match on A Incoming packet at controller Typical debuggers tell us what happened: NetSight: Packet histories Y!: Network provenance Key benefit: Rich explanation of what, when, and why.

Problem: Explanation can be too big! Rule 7: Next-hop=port2 root Root cause: faulty rule Packet arrives at wrong server The problem: Finding the root cause in a large provenance tree.

Key insight: Use reference events! Bob Web server 2 Web server 1 DPI Remember that some packets were routed correctly. The same things should have happened to all packets! Key insight: If we have both a (bad) symptom and a (good) reference, we only need to reason about the differences between them!

A new debugger Bob collects both a bad symptom and a good reference fault Field 3 of config entry 4 is wrong! Bob reference Debugger Bob collects both a bad symptom and a good reference Bob sends both events to the debugger Debugger generates provenance, outputs difference Ideally, there is only one diff—the root cause!

Outline Motivation: Network diagnostics Background Key insight A new debugger Differential provenance Are references typically available? Strawman approach Our approach Initial results Conclusion

Are references typically available? Survey: Posts on the ‘Outages’ mailing list in Sept-Dec 2014. 64 posts related to diagnostics. 42/64 (66%) posts involve both a fault and some reference. Examples: Some DNS servers have stale records, but others are good Probes sometimes fail, sometimes succeed More examples in the paper

Strawman solution - = ? Bad provenance Reference provenance A strawman solution: Pick out different nodes in trees. Bad provenance: 201 nodes Reference provenance: 156 nodes Naïve diff: 278 nodes!

Why does the strawman not work? Faulty rule Observation: The diff can be larger than the individual trees. Reason #1: Differences that “do not matter” E.g., timestamps, packet payloads, etc. Reason #2: “Butterfly effect” A small difference can change later events drastically!

Differential provenance Output: - Rule 7: change port - Rule 9: change range Bad provenance Reference provenance Approach: Change past events, and think about what could have happened. (1) Find some early ‘differences’ in the trees. (2) Change the faulty node to a correct equivalent. (3) Use replay to determine what would have happened. (4) Output the set of changes that align the trees.

Technical challenges Challenge #1: Where do we start? Heuristics: Change early events, minimum changes… E.g., prefer changing 1 event than 1000 events. Challenge #2: How should we make the change? Approach: Think about what should have happened. E.g., packet should go to switch 2, not 1. Challenge #3: Irrelevant differences? Approach: Equivalence relations between events. E.g., IPs 4.3.2.1 and 4.3.3.1 See paper for more details.

Setup Setup Overly specific flow entry 4.3.2.0/24 4.3.3.0/24 Internet Web server 1 DPI Setup Platform: RapidNet SDN: 6 switches, 2 servers The symptom: misrouted packets from 4.3.2.0/24 The reference: packets from 4.3.3.0/24

Differential provenance Initial results = Fault: 201 nodes Naïve diff Reference: 156 nodes = Rule 7: next hop should be port 1, not 2! Differential provenance Differential provenance finds a single node (the faulty rule) to be the root cause!

Conclusion Thanks! Debugging networks is hard Need good debuggers! Provenance can find the causes of an event Problem: Explanation can be too detailed. Idea: Use reference events Sufficient to find the (few) differences to the observed symptom New debugger based on differential provenance Result: Very precise diagnostics Ideally, can identify a single root cause! Thanks!