Secure Network Provenance Wenchao Zhou *, Qiong Fei*, Arjun Narayan*, Andreas Haeberlen*, Boon Thau Loo*, Micah Sherr + * University of Pennsylvania + Georgetown University

Motivation An example scenario: network routing  System administrator observes strange behavior  Example: the route to foo.com has suddenly changed  What exactly happened (innocent reason or malicious attack)? 2 Why did my route to foo.com change?! Alice foo.com Route r 1 Route r 2 Innocent Reason? Malicious Attack? A D E B C

We Need Secure Forensics For network routing …  Example: incident in March 2010 Traffic from Capital Hill got redirected … but also for other application scenarios  Distributed hash table: Eclipse attack  Cloud computing: misbehaving machines  Online multi-player gaming: cheating Goal: secure forensics in adversarial scenarios 3

Ideal Solution 4 The Network A: Because someone accessed Router D and changed the configuration from X to Y. Alice foo.com Route r 2 A D E B C Not realistic: adversary can tell lies Why did my route to foo.com change?! Q: Explain why the route to foo.com changed to r2.

Challenge: Adversaries Can Lie 5 The Network Q: Explain why the route to foo.com changed to r2. Alice foo.com Route r 2 A D E B C Problem: adversary can …... fabricate plausible (yet incorrect) response … point accusation towards innocent nodes Everything is fine. Router E advertised a new route. I should cover up the intrusion.

Existing Solutions Existing systems assume trusted components  Trusted OS kernel, monitor, or hardware E.g. Backtracker [OSDI 06], PASS [USENIX ATC 06], ReVirt [OSDI 02], A2M [SOSP 07]  These components may have bugs or be compromised  Are there alternatives that do not require such trust? Our solution:  We assume no trusted components;  Adversary has full control over an arbitrary subset of the network (Byzantine faults). 6

Ideal Guarantees Ideally: explanation is always complete and accurate Fundamental limitations  E.g. Faulty nodes secretly exchange messages  E.g. Faulty nodes communicate outside the system What guarantees can we provide? 7 Fundamentally impossible

Realistic Guarantees No faults: Explanation is complete and accurate Byzantine fault: Explanation identifies at least one faulty node Formal definitions and proofs in the paper 8 The Network Q: Why did my route to foo.com change to r2? A: Because someone accessed Router D and changed its router configuration from X to Y. Alice foo.com Route r 2 A D E B C Aha, at least I know which node is compromised.

Outline Goal: A secure forensics system that works in an adversarial environment  Explains unexpected behavior  No faults: explanation is complete and accurate  Byzantine fault: exposes at least one faulty node with evidence Model: Secure Network Provenance Tamper-evident Maintenance and Processing Evaluation Conclusion 9

Provenance as Explanations Origin: data provenance in databases  Explains the derivation of tuples (ExSPAN [SIGMOD 10])  Captures the dependencies between tuples as a graph  “Explanation” of a tuple is a tree rooted at the tuple 10 Alice foo.com route(C, foo.com) link(C, foo.com) route(A, B) A BC D E …… route(B, foo.com) link(B, C) route(A, foo.com) link(A, B) route(D, foo.com) link(D, E) route(E, foo.com) link(E, B) route(A, D)

Secure Network Provenance Challenge #1. Handle past and transient behavior  Traditional data provenance targets current, stable state  What if the system never converges?  What if the state no longer exists? 11 route(C, foo.com) link(C, foo.com) route(B, foo.com) link(B, C) route(A, foo.com) link(A, B)

Secure Network Provenance Challenge #1. Handle past and transient behavior  Traditional data provenance targets current, stable state  What if the system never converges?  What if the state no longer exists?  Solution: Add a temporal dimension 12 route(C, foo.com) link(C, foo.com) route(B, foo.com) link(B, C) route(A, foo.com) link(A, B) route(C, foo.com) link(C, foo.com) route(B, foo.com) link(B, C) route(C, foo.com) link(C, foo.com) Time = t1Time = t2Time = t3 Timeline

Secure Network Provenance Challenge #2. Explain changes, not just state  Traditional data provenance targets system state  Often more useful to ask why a tuple (dis)appeared  Solution: Include “deltas” in provenance 13 route(C, foo.com) link(C, foo.com) route(B, foo.com) link(B, C) route(A, foo.com) link(A, B) route(C, foo.com) link(C, foo.com) route(B, foo.com) link(B, C) route(C, foo.com) link(C, foo.com) Time = t1Time = t2Time = t3 Timeline +route(A, foo.com) +route(C, foo.com) +route(B, foo.com) +link(C, foo.com)

Secure Network Provenance Challenge #3. Partition and secure provenance  A trusted node would be ideal, but we don’t have one  Need to partition the graph among the nodes themselves  Prevent nodes from altering the graph 14 route(C, foo.com) link(C, foo.com) route(B, foo.com) link(B, C) route(A, foo.com) link(A, B) route(D, foo.com) link(D, E) route(E, foo.com) link(E, B)

Partitioning the Provenance Graph Step 1: Each node keeps vertices about local actions  Split cross-node communications Step 2: Make the graph tamper-evident 15 route(C, foo.com) link(C, foo.com) route(B, foo.com) link(B, C) route(A, foo.com) link(A, B) route(D, foo.com) link(D, E) route(E, foo.com) link(E, B) RECVSEND

Securing Cross-Node Edges Step 1: Each node keeps vertices about local actions  Split cross-node communications Step 2: Make the graph tamper-evident  Secure cross-node edges (evidence of omissions) 16 RECVSEND RECEIVE Signed commitment from B Signed ACK from A Router A Router B

Outline Goal: A secure forensics system that works in an adversarial environment  Explains unexpected behavior  No faults: explanation is complete and accurate  Byzantine fault: exposes at least one faulty node with evidence Model: Secure Network Provenance Tamper-evident Maintenance and Processing Evaluation Conclusion 17

System Overview Stand-alone provenance system On-demand provenance reconstruction  Provenance graph can be huge (with temporal dimension)  Rebuild only the parts needed to answer a query 18 Application Primary systemProvenance system Network UsersOperator Query engine Maintenance engine Extract provenance Maintain provenance Query provenance Maintenance engine Query engine

Extracting Dependencies Option 1: Inferred provenance  Declarative specifications explicitly capture provenance  E.g. Declarative networking, SQL queries, etc. Option 2: Reported provenance  Modified source code reports provenance Option 3: External specification  Defined on observed I/Os of a black-box system 19

Secure Provenance Maintenance Maintain sufficient information for reconstruction  I/O and non-deterministic events are sufficient  Logs are maintained using tamper-evident logging Based on ideas from PeerReview [SOSP 07] 20 Alice foo.com A BC D E …… SEND RCV-ACK …… RECV ACK

Secure Provenance Querying Recursively construct the provenance graph  Retrieve secure logs from remote nodes  Check for tampering, omission, and equivocation  Replay the log to regenerate the provenance graph 21 Alice foo.com A BC D E route(A, foo.com) link(A, B) Explain the route from A to foo.com. RECV (from B)

Secure Provenance Querying Recursively construct the provenance graph  Retrieve secure logs from remote nodes  Check for tampering, omission, and equivocation  Replay the log to regenerate the provenance graph 22 Alice foo.com A BC D E route(B, foo.com) link(B, C) route(A, foo.com) link(A, B) RECV (from C)

Secure Provenance Querying Recursively construct the provenance graph  Retrieve secure logs from remote nodes  Check for tampering, omission, and equivocation  Replay the log to regenerate the provenance graph 23 Alice foo.com route(C, foo.com) link(C, foo.com) A BC D E link(B, C) route(A, foo.com) link(A, B) route(B, foo.com) OK. Now I know how the route was derived.

Outline Goal: A secure forensics system that works in an adversarial environment  Explains unexpected behavior  No faults: explanation is complete and accurate  Byzantine fault: exposes at least one faulty node with evidence Model: Secure Network Provenance Tamper-evident Maintenance and Processing Evaluation Conclusion 24

Evaluation Results Prototype implementation (SNooPy)  How useful is SNP? Is it applicable to different systems?  How expensive is SNP at runtime? Traffic overhead, storage cost, additional CPU overhead? Does SNP affect scalability?  What is the querying performance? Per-query traffic overhead? Turnaround time for each query? 25

Usability: Applications We evaluated SNooPy with  Quagga BGP: RouteView (external specification) Explains oscillation caused by router misconfiguration  Hadoop MapReduce: (reported provenance) Detects a tampered Mapper that returns inaccurate results  Declarative Chord DHT: (inferred provenance) Detects an Eclipse attacker that always returns its own ID SNooPy solves problems reported in existing work 26

Runtime Overhead: Storage Manageable storage overhead  One week of data: E.g. Quagga – 7.3GB; Chord – 665MB 27 Over 50% of the overhead was due to signatures and acks. Batching messages would help.

Query Latency Query latency varies from application to application Reasonable overhead 28 dominated by verifying logs and snapshots Verification Replay Download largely due to replaying logs

Summary Secure network provenance in untrusted environments  Requires no trusted components  Strong guarantees even in the presence of Byzantine faults Formal proof in a technical report  Significantly extends traditional provenance model Past and transient state, provenance of change, …  Efficient storage: reconstructs provenance graph on demand  Application-independent (Quagga, Hadoop, and Chord) 29 Project website: Questions?

Provenance with an Example Forensics as provenance  Origins: data provenance in database  Capture the dependencies as a graph Our challenge: secure provenance information 30 ab c (a,c,5) (a,b,3) (b,c,2) Dependency relationship between network state Execution of a part of a protocol (derivation rule)

System Model and Provenance Representation of primary systems  System state as tuples: E.g.  System’s execution logic as derivation rules: E.g. ← Λ Forensics as provenance  Origins: Data provenance from database community  Capture the dependencies as a graph 31 ab c (a,c,5) (a,b,3) (b,c,2)

Reliable Provenance Treat model: Byzantine adversaries, no trust  An arbitrary subset of the network is fully controlled by a Byzantine adversary  Existing work that relies on trusted components OS kernel, virtual machine, monitor, hardware, etc  Are there alternatives that do not require such trust? 32 Alice foo.com

Strawman Solution Strawman solution: provenance + fault detection  Not sure if a query result is correct (detection delay) Alice A C B Route r D S OK, I need to take action What should I do now… 33

Strawman Solution Strawman solution: provenance + fault detection  Not sure if a query result is correct (detection delay)  Information on other (benign) nodes may be corrupted  System becomes useless when it is most needed! Alice A C B Route r D S 34

Integration with Legacy Application Proxies  Intercept network I/Os (and system state updates)  Specify dependency logic in NDlog “Maybe” rules  Treat legacy application as “black-box”  Possible causal relationships between I/Os and state updates  E.g. incoming and outgoing route advertisement: outputRoute(AS,Prefix,Route2) ?- inputRoute(AS,Prefix,Route1), f_isExtend(Route2,Route1,AS)=1. Multiple incoming route advertisement to the prefix Decision is made based on (secret) policies

Optimizations Periodic snapshots of system state  Log-replay for long-running executions is expensive  Optimization: retrieve only part of the snapshot  Use Merkle Hash Tree for integrity check Batching messages  Mitigate costs of signatures and ACKs  Trade latency for communication/storage/computation cost 36