Marcos K. Aguilera Microsoft Research Silicon Valley No Time for Asynchrony Michael Walfish UCL, Stanford, UT Austin

Problem: Nodes in Distributed Systems Fail Pragmatic response: end-to-end timeouts  Getting them right: hard. primarybackup apps OS VM protocols drivers Getting them wrong: bad.

Problem: Nodes in Distributed Systems Fail Pragmatic response: end-to-end timeouts  Getting them right: hard. primarybackup apps OS VM protocols drivers Getting them wrong: bad.

Problem: Nodes in Distributed Systems Fail Pragmatic response: end-to-end timeouts  Getting them right: hard. primarybackup apps OS VM protocols drivers Getting them wrong: bad.

Problem: Nodes in Distributed Systems Fail Pragmatic response: end-to-end timeouts  Getting them right: hard. Getting them wrong: bad. Current view/lore/wisdom: design for asynchrony  Very general  guarantee of safety primarybackup ? Paxos apps OS VM protocols drivers

Different Points of View 1. Keep it simple: 2. Keep it safe: 3. Our view:  We want simplicity, safety, and high availability  Our mantra: “no end-to-end timeouts” “rely on time and timeouts” “design for asynchrony” there is good in both

A Proposal That Meets Our Goals app OS VM/hypervisor network driver network card Spies indicate crashed-or-not authoritatively Why do we want device drivers killing OSes? backupprimary spy failure detector

Scope Enterprises, data centers Not Byzantine failures This Talk Will Argue: 1. Asynchrony is problematic  (And often disregarded in practice) 2. Spy-based failure detection meets our goals

Asynchrony Detracts From Safety 1. “Safety under asynchrony” downplays liveness  But highest layers in a system have deadlines  Lower layer loses liveness  at deadline, higher layer may be bereft  lose “whole system” safety

Asynchrony Detracts from Safety (Cont’d.) 2. Under asynchrony, components hide useful info.  Unresponsiveness  higher layers guess  Wrong guesses  loss of safety 3.  complex designs (example: Paxos)  Complexity  mistakes  safety violations asynchronous component ?

Empirical Observations Against Asynchrony Paxos-using systems rely on synchrony for safety World fundamentally synchronous  Electrons, CPUs, human beings, organizations  “Safety under asynchrony” hard to meet  Generality of asynchrony maybe not needed in reality Leases, … Paxos Chubby [Burrows OSDI06], Petal [Lee ASPLOS96], WheelFS [Stribling et al. NSDI09], …

Recap Argument Against Asynchrony Appeal of asynchrony: generality  safety Argument against asynchrony: Async. components can lead to unsafe systems Hard to meet “safety under asynchrony” Asynchrony doesn’t represent reality People forced to depart from asynchrony anyway

Our Argument, Continued 1. Asynchrony is problematic  (And often disregarded in practice) 2. Spy-based failure detection meets our goals

A Powerful Abstraction: Perfect Failure Detectors “ ” “up” ? Want a model where:  CRASHED?( ) [Chandra & Toueg, JACM 96] processes Perfect failure detector (PFD) A perfect failure detector is an oracle Asynchronous model:

PFDs  Safe, Simple Distributed Algorithms Replication by primary-backup instead of Paxos Other examples in the paper (not our contribution) PFD backupprimary “ ”

How to Build a Perfect Failure Detector? FD ? Failure detection (not PFD) uses status messages Hard to make this FD a PFD  Variable timing, system a black box

Current proposals coarse [Fetzer IEEE Trans. 2003, Ricciardi & Birman PODC91]  Focus on E2E behavior  Use E2E timeouts  Kill/exclude any suspect Realizing Perfect Failure Detectors PFD Recall our third goal: high availability. Approach is “surgical”:  Operate inside layers  Use only local timing  Kill as a last resort app OS VM/hypervisor network driver network card ? ?

Spies Orchestrated to Form Surgical PFD network switch app OS VM/hypervisor network driver network card Example: spy in VM tracks OS state Lower-level spies also monitor higher-level ones  Allows localization of smallest failed component PFD

Limitations and Discussion 1. Under network partition, PFD module blocks 2. To realize spies, must modify system infrastructure We think this is okay in data centers  Partitions often cause block anyway  One administrative domain Harder to address in wide area  Requires spies in Internet switches and routers  Network to host feedback not totally implausible

Summary and Conclusion End-to-end timing assumptions problematic. So:  Avoid timing with inside info., assassination  Avoid end-to-end by infiltrating many layers The gain: simple, safe, and live distributed systems But: PFDs, spies not a good fit for all environments Next step: get it implemented and deployed This is a call to arms