Concurrent, Distributed Systems Stock ExchangesTelecoms Commuter Rail

Concurrent, Distributed Systems System Nodes exchanging Messages Execution 1.Node gets message event 2.Executes event handler - Updates node state - Sends new messages 3. Repeat…

Distributed Systems: Challenges System Nodes exchanging Messages Challenges Nodes: enter, leave, fail Messages: reordered, lost System must stay available - Eventually, all nodes regroup - Eventually, all packets delivered - Eventually, some good happens Liveness Properties

The Space of System Executions Initial State At each state, scheduler picks: 1.Node n 3.Executes code

An Execution = Sequence of Choices

An Execution = Sequence of Choices

An Execution = Sequence of Choices

An Execution = Sequence of Choices

Bad States Safety Bugs: Execution that drives system to bad state Safety Bugs Bad States Null Dereferences Buffer overflows Assertion Failures Low-level crash

How to find Safety Bugs? Find path from Initial to Bad By systematically exploring executions (Iterating over sequences of choices) Initial State Bad States

Model Checking for Safety Bugs Bad States Find path from Initial to Bad By systematically exploring executions [Verisoft 97, Cmc 04, Chess 07]

Safety Properties are too Low Level Find path from Initial to Bad By systematically exploring executions [Verisoft 97, Cmc 04, Chess 07]

Safety Properties are too Low Level Distributed Systems: Designed for crashes & failures Challenge: End-to-end Problems Liveness bugs

Live States Bad States Initial State Good States: All nodes regroup All packets delivered Live States: Eventually Good Happens

Live Executions Initial State Live States

Liveness Violations Initial State Live States Execution never reaches live state

How to Find Liveness Violations? Live States Explore all executions ? Infinitely many...

How to Find Liveness Violations? Live States Explore all executions upto bound ? Combinatorial explosion (depth < 50) Liveness at depth >> 50 [Verisoft 97, Cmc 04, Chess 07]

How to Find Liveness Violations? Live States Looks pretty hopeless...

Live States Idea 1: Dead States Dead States No execution can reach live states Recovery is impossible

Idea 1: Dead States To find Liveness bugs, Look for Dead executions. How to tell if a state is Dead ?

Idea 2: Random Walks Live States Dead States Execute long random walks from state Pr[reaching live] = 0 Pr[reaching live] = 1 How to tell if a state is Dead ?

Executions and Random Walks At each execution step, 1.Scheduler picks node n 2.Schedular picks 3.Executes event code Random Walk: Scheduler picks randomly (from some Prob. Dist. over nodes, events)

Liveness Bugs = Search + Random Walks 1. Systematic Search: find candidates 2. Random Walk: test if candidate dead Live States Iterate

Liveness Bugs = Search + Random Walks Live States If walk length >> avg. steps to liveness Then non-live walk is likely liveness bug! 100k Events 1k Events 100,000 Step Execution (2 Gb Log file) How to pinpoint bug ?

Live States Idea 3: The Critical Transition Dead States System transitions from a recoverable to a dead state How to find Critical Transition without knowing Dead States?

Live States Idea 3: The Critical Transition Binary Search using Random Walks!

Live States Idea 3: The Critical Transition Binary Search using Random Walks! Binary Search

Live States Idea 3: The Critical Transition Critical Transition Dead States System transitions from a recoverable to a dead state Pinpoints bug

Recap Liveness Bugs Found System has shot itself (but doesnt know it) Systematic Search Finds candidate dead states Random Walks Determine if candidate is dead Critical Transition The event that makes recovery impossible

Bells and Whistles (1/2) Random Walk Bias Assign “likely” events higher weight e.g. application > network > timer > fail Bugs not missed Random walk only tests deadness Live state reached sooner Error traces shorter, simpler

Bells and Whistles (2/2) Prefix-Based Search Restart search after reaching liveness Analyzes effect of failures in “steady-state”

Evaluation Liveness Bugs, Critical Transition Mace (C++) System MaceMC Liveness Properties

Systems RandTree Random Overlay Tree with max degree. MaceTransport User-level, reliable messaging service. Pastry Key-based routing, using an overlay ring. Chord Key-based routing, using an overlay ring.

Liveness Properties RandTree Random Overlay Tree with max degree. MaceTransport User-level, reliable transport service. Pastry Key-based routing, using an overlay ring. Chord Key-based routing, using an overlay ring. Eventually, all messages acknowledged. Eventually, all nodes form single tree. Eventually, all nodes form a ring.

Sample Bug: RandTree Nodes With Child, Parent pointers Property Eventually nodes form tree

Sample Bug: RandTree C C A A C requests to join under A A sends ack C fails and restarts C ignores ack from A C joins under B Bug: System stuck as a DAG! C’s failure not propagated to A B B

Liveness Bugs Yield Safety Assertions Dead States Violations of a priori unknown safety properties Critical Transition Helps identify dead states Yields new safety properties and bugs

New Safety Property: Chord Nodes with Fwd, Back pointers Property Eventually nodes form a ring Critical Transition To Dead State Where: n.back=n, n.fwd = m New Safety Property IF n.back=n THEN n.fwd=n

Scorecard SystemBugsLivenessSafety MaceTransport1156 RandTree17125 Pastry550 Chord19910 Totals Several “protocol level” bugs Routinely used by Mace programmers

Programming Challenges How to handle unexpected events ? How to propagate effects of failures ? How to limit impact on performance?

Take Away Message Liveness Bugs Are Very Important Randomness Helps.