SOSP 2007 © 2007 Andreas Haeberlen, MPI-SWS 1 Practical accountability for distributed systems Andreas Haeberlen MPI-SWS / Rice University Petr Kuznetsov MPI-SWS Peter Druschel MPI-SWS

SOSP © 2007 Andreas Haeberlen, MPI-SWS Motivation Distributed state, incomplete information General case: Multiple admins with different interests Admin

SOSP © 2007 Andreas Haeberlen, MPI-SWS General faults occur in practice Many faults are not 'fail-stop' Node is still running, but its behavior changes Examples: Hardware malfunctions Misconfigurations Software modifications by users Hacker attacks...

SOSP © 2007 Andreas Haeberlen, MPI-SWS Dealing with general faults is difficult How to detect faults? How to identify the faulty nodes? How to convince others that a node is (not) faulty? Incorrect message Responsible admin

SOSP © 2007 Andreas Haeberlen, MPI-SWS Learning from the 'offline' world Relies on accountability Example: Banks Can be used to detect, identify and convince But: Existing fault-tolerance work mostly focused on prevention Goal: A general+practical system for accountability RequirementSolution CommitmentSigned receipts Tamper-evident recordDouble-entry bookkeeping InspectionsAudits

SOSP © 2007 Andreas Haeberlen, MPI-SWS Outline Introduction What is accountability? How can we implement it? How well does it work?

SOSP © 2007 Andreas Haeberlen, MPI-SWS Ideal accountability Whenever a node is faulty in any way, the system generates a proof of misbehavior against that node Fault := Node deviates from expected behavior Recall that our goal is to detect faults identify the faulty nodes convince others that a node is (or is not) faulty Can we build a system that provides the following guarantee?

SOSP © 2007 Andreas Haeberlen, MPI-SWS Can we detect all faults? Problem: Faults that affect only a node's internal state Requires online trusted probes at each node Focus on observable faults: Faults that causally affect a correct node This allows us to detect faults without introducing any trusted components A A X C C

SOSP © 2007 Andreas Haeberlen, MPI-SWS Can we always get a proof? Problem: He-said-she-said situation Three possible causes: A never sent X B refuses to accept X X was lost by the network Cannot get proof of misbehavior! Generalize to verifiable evidence: a proof of misbehavior, or a challenge that the node cannot answer What if, after a long time, no response has arrived? Does not prove the fault, but we can suspect the node A A X B B C C ? I sent X! I never received X! ?!

SOSP © 2007 Andreas Haeberlen, MPI-SWS Practical accountability We propose the following definition of a distributed system with accountability: This is useful Any (!) fault that affects a correct node is eventually detected and linked to a faulty node It can be implemented in practice Whenever a fault is observed by a correct node, the system eventually generates verifiable evidence against a faulty node

SOSP © 2007 Andreas Haeberlen, MPI-SWS Outline Introduction What is accountability? How can we implement it? How well does it work?

SOSP © 2007 Andreas Haeberlen, MPI-SWS Adds accountability to a given system Implemented as a library Provides secure record, commitment, auditing, etc. Assumptions: Implementation: PeerReview 1. System can be modeled as collection of deterministic state machines 2. Nodes have reference implementations of the state machines 3. Correct nodes can eventually communicate 4. Nodes can sign messages

SOSP © 2007 Andreas Haeberlen, MPI-SWS M PeerReview from 10,000 feet All nodes keep a log of their inputs & outputs Including all messages Each node has a set of witnesses, who audit its log periodically If the witnesses detect misbehavior, they generate evidence make the evidence avai- lable to other nodes Other nodes check evi- dence, report fault A's log B's log A A B B M C C D D E E A's witnesses M

SOSP © 2007 Andreas Haeberlen, MPI-SWS PeerReview detects tampering A B Message Hash chain Send(X) Recv(Y) Send(Z) Recv(M) H0H0 H1H1 H2H2 H3H3 H4H4 B's log ACK What if a node modifies its log entries? Log entries form a hash chain Inspired by secure histories [Maniatis02] Signed hash is included with every message  Node commits to its current state  Changes are evident Hash(log)

SOSP © 2007 Andreas Haeberlen, MPI-SWS PeerReview detects inconsistencies What if a node keeps multiple logs? forks its log? Check whether the signed hashes form a single hash chain H3'H3' Read X H4'H4' Not found Read Z OK Create X H0H0 H1H1 H2H2 H3H3 H4H4 OK "View #1" "View #2"

SOSP © 2007 Andreas Haeberlen, MPI-SWS Module B PeerReview detects faults How to recognize faults in a log? Assumption: Node can be modeled as a deterministic state machine To audit a node: Replay inputs to a trusted copy of the state machine Check outputs against the log Module A Module B =? Log Network Input Output State machine if ≠ Module A

SOSP © 2007 Andreas Haeberlen, MPI-SWS PeerReview offers provable guarantees PeerReview guarantees that: 1) Faults will be detected 2) Good nodes cannot be accused Formal definitions and proof in a TR If node commits a fault + has a correct witness, then witness obtains a proof of misbehavior (PoM), or a challenge that the faulty node cannot answer If node is correct there can never be a PoM, and it can answer any challenge

SOSP © 2007 Andreas Haeberlen, MPI-SWS Outline Introduction What is accountability? How can we implement it? How well does it work? Is it widely applicable? How much does it cost? Does it scale?

SOSP © 2007 Andreas Haeberlen, MPI-SWS PeerReview is widely applicable App #1: NFS server in the Linux kernel Many small, latency-sensitive requests Tampering with files Lost updates App #2: Overlay multicast Transfers large volume of data Freeloading Tampering with content App #3: P2P Complex, large, decentralized Denial of service Attacks on DHT routing More information in the paper Metadata corruption Incorrect access control Censorship

SOSP © 2007 Andreas Haeberlen, MPI-SWS How much does PeerReview cost? Dominant cost depends on number of witnesses W O(W 2 ) component Baseline Avg traffic (Kbps/node) Number of witnesses Baseline traffic Signatures and ACKs Checking logs W dedicated witnesses

SOSP © 2007 Andreas Haeberlen, MPI-SWS Mutual auditing Small probability of error is inevitable Example: Replication Can use this to optimize PeerReview Accept that an instance of a fault is found only with high probability Asymptotic complexity: O(N 2 )  O(log N) Small random sample of peers chosen as witnesses Node

SOSP © 2007 Andreas Haeberlen, MPI-SWS PeerReview is scalable Assumption: Up to 10% of nodes can be faulty Probabilistic guarantees enable scalability Example: system scales to over 10,000 nodes with P= DSL/cable upstream system w/o accountability O((log N) 2 ) O(log N) system + PeerReview (P= ) system + PeerReview (P=1.0) System size (nodes) Avg traffic (Kbps/node)

SOSP © 2007 Andreas Haeberlen, MPI-SWS Summary Accountability is a new approach to handling faults in distributed systems detects faults identifies the faulty nodes produces evidence Our practical definition of accountability: Whenever a fault is observed by a correct node, the system eventually generates verifiable evidence against a faulty node PeerReview: A system that enforces accountability Offers provable guarantees and is widely applicable Thank you! Andreas Haeberlen received a SOSP travel scholarship, which was supported by Infosys