Parallel and Distributed Systems Laboratory Paradise: A Toolkit for Building Reliable Concurrent Systems On Building Reliable Concurrent Systems Vijay K. Garg Professor, Department of ECE and CS Director, PDSL The University of Texas at Austin Austin, TX 78712

2 Motivation: Reliable Software  Multithreaded Distributed programs are prone to errors. – Concurrency, nondeterminism, process and channel failures Techniques to ensure program correctness  Before program development: Model Checking  During: Testing and Debugging  After: Software Fault-Tolerance

3 Paradise Approach Key Abstraction: Global Properties  Model Checking and Verification: Check global properties against model of the program (Promela)  Testing and Debugging: Global breakpoints, Trace analysis  Software Fault-Tolerance: Monitoring for global properties, Controlled Reexecution

4 Talk Outline  Motivation  Monitoring Distributed Systems – Clock : Tracking Dependency – Camera : Global Snapshot (Checkpoint) – Sensor : Detecting Global Properties – Slicer : Computation Slicing – Supervisor: Controlling Execution  Other Projects at PDSL

5 Paradise Environment ProgramMonitorSlicerPredicate Observe Control

6 Trace Model: Total Order vs Partial Order  Total order: interleaving of events in a trace  Partial order: Lamport’s happened-before model f2f2 e1e1 CS 2 CS 1 f1f1 e2e2 P1P1 P2P2 Partial Order Trace CS 2 CS 1 e1e1 e2e2 f1f1 f2f2 e2e2 e1e1 CS 2 f1f1 f2f2 Successful Trace Specification: CS 1 Λ CS 2 ¬CS 2 ¬CS 1 ¬CS 2 ¬CS 1 ¬ CS 2 Faulty Trace 

7 Tracking Dependency computation: a set of events ordered by “happened before” relation  Problem: Timestamp events to answer – e happened before f ? – e concurrent with f ?

8 Clocks in a Distributed System Result: s happened before t i the vector at s is less than the vector at t. Vector Clocks [Fidge 89, Mattern 89] P1P1 (1,0,0)(2,1,0)(3,1,0) P2P2 (0,1,0)(0,2,0) P3P3 (0,0,1)(0,0,2)(2,1,3)

9 Dynamic Chain Clocks  Problem with vector clocks: scalability, dynamic process structure  Idea: Computing the “chains” in an online fashion [Aggarwal and Garg PODC 05] for relevant events a f e b d c h g abcd e fgh A computation with 4 processes The relevant subcomputation P1P2P3P4P1P2P3P4

10 Experimental Results Simulation of a computation with 1% relevant events Measured – number of components vs number of threads – total time overhead vs number of threads

11 Talk Outline  Motivation and Overview  Instrumentation – Clock : Tracking Dependency  Property Checking – Camera : Global Snapshot (Checkpoint) – Sensor : Detecting Global Properties – Slicer : Computation Slicing

12 Global Snapshot Problem: Compute a global snapshot/checkpoint of the system (state of the processes and the channels) Motivation – Checkpointing for fault tolerance – Distributed debugging – Detecting stable predicates

13 Key Difficulties: Taking care of messages Two sites A and B with $400 each. Site A sends a message with $ 100 to site B.  Problem 1: Inconsistent State checkpoint (A) before message sent - $400 message received before checkpoint (B) - $500  Problem 2: Messages in transit message sent before checkpoint (A) - $300 checkpoint (B) before message received - $400 P1P1 P2P2 P3P3 G1G1 G2G2

14 Current Algorithms Key idea: white/red processes and messages  A process must be red to act on a red message  Record white messages received by red processes  [Chandy and Lamport 85] A "marker" message on every channel  [Mattern 89, SBF+ 04] Number of white messages sent on every channel O(N 2 ) messages for completely connected topology

15 Our Algorithms joint work with Rahul Garg, and Yogish Sabharwal, IBM IRL. (ACM International Conference on Supercomputing 2006) – Lower Bound: Ω(N log w) messages – Implementation: Blue Gene/L with MPI. – w: average number of messages in transit AlgorithmNumber of messagesMessage size Grid BasedO(N 3/2 )O(N 1/2 logw) Tree BasedO(N logN logw)O(logw + logN) CentralizedO(N logw)O(logw + logN)

16 Global Property Detection Predicate: A global condition expressed using variables on processes – e.g., more than one process is in critical section, there is no token in the system Problem: find a global state that satisfies the given predicate P1P1 P2P2 G1G1 G2G2 Critical section

17 The Main Difficulty in Partial Order Algorithm for general predicate [Cooper and Marzullo 91] Too many global states : A computation may contain as many as O(k n ) global states k: maximum number of events on a process n: number of processes e1e1 e2e2 f1f1 f2f2 T ┴ P1P1 P2P2 {e 1, ┴ } {f 1, ┴ } {e 1, f 1, ┴ } {e 2, e 1, f 1, ┴ } {e 2, e 1, f 2, f 1, ┴ {e 1, f 2, f 1, ┴ } {e 2, e 1, ┴ } {┴}{┴}

18 Efficient Predicate Detection for Special Cases  stable predicate: [Chandy and Lamport 85] once the predicate becomes true, it stays true e.g., deadlock  unstable predicate: observer independent predicate [Charron-Bost et al 95] occurs in one interleaving  occurs in all interleavings e.g., any disjunction of local predicate linear predicate [Chase and Garg 95] e.g., conjunctive predicates such as there is no leader in the system relational predicate: x1 + x2 +…+ xn ≥ k [Chase and Garg 95] e.g., violation of k-mutual exclusion

19 Linear Predicates The set of consistent cuts that satisfy a linear predicate is closed under intersection [ Chase and Garg 95]. Examples:  conjunctive predicates: critical 1 and critical 2  channel predicates: “all channels are empty”, “there are exactly k messages in the channel from process P to Q”  some relational predicates: x 1 + x 2 · k, when x i is mon. non- decreasing WXWX W Å X

20 Conjunctive Predicates A predicate that can be expressed as l 1 Λ l 2 Λ … l n, where l i is local to P i. Detect errors that may be hidden in some run due to race conditions. Examples: – mutual exclusion problem: (P1 in CS) and (P2 in CS) – missing primary: (P1 is secondary) and (P2 is secondary) and (P3 is secondary) Importance: Sufficient for detection of any boolean expression of local predicates

21 Conjunctive Predicates: Centralized Algorithm (l 1 Λ l 2 Λ … l n ) is true iff there exist s i in P i such that l i is true in state s i, and s i and s j are incomparable for distinct i,j.

22 Algorithms for Conjunctive Predicates Centralized Algorithm [Garg and Waldecker 92] Each non-checker process maintains its local vector and sends to the checker process the chain clock whenever – local predicate is true – at most once in each message interval. Time complexity: Checker requires at most O(n 2 m) comparisons. – token based algorithm [Garg and Chase 95] – completely distributed algorithm [Garg and Chase 95] – keeping queues shorter [Chiou and Korfhage 95] – avoiding control messages [Hurfin, Mizuno, Raynal, Singhal 96]

23 Other Special Classes of Predicates  Relational Predicates – Let x i : number of token at P i – Σ x i < k: loss of tokens – Algorithms: max-flow techniques [Groselj 93, Chase and Garg 95, Wu and Chen 98] – Dilworth's partition [Tomlinson and Garg 96]

24 Relational Predicates Let x i  0 be a variable at P i. Predicates of the form [Chase and Garg 95]  x i  k Algorithm: Consistent cut with minimum value = min cut in the flow graph

25 Predicate Detection in General Explore the state-space (need to examine all global states) without constructing the graph  breadth first manner [Cooper and Marzullo 91]  depth first manner [Alagar and Venkatesan 94]  lexical order [Garg 03] {e 1, ┴ } {f 1, ┴ } {e 1, f 1, ┴ } {e 2, e 1, f 1, ┴ } {e 2, e 1, f 2, f 1, ┴ {e 1, f 2, f 1, ┴ } {e 2, e 1, ┴ } {┴}{┴}

26 Talk Outline  Motivation and Overview  Instrumentation – Clock : Tracking Dependency  Property Checking – Camera : Global Snapshot (Checkpoint) – Sensor : Detecting Global Properties – Slicer : Computation Slicing – Supervisor: Controlling Execution

27 The Main Idea of Computation Slicing Partial order trace slice state explosion keep all red global states slicing

28 How does Computation Slicing Help? Partial order trace slice retain all global states satisfying b 1 slicing for b 1 check b 1 Λ b 2 check b 2 satisfy b 1

29 Example  Detect predicate (x*y + z < 5) Λ (x ≥1) Λ (z ≤ 3) P1P1 P2P2 P3P3 x y z a 1 b 2 c d 0 e 0 f 2 g 1 h 3 u 4 v 1 w 2 x 4 {a,e,f,u,v} {b} {w}{g} Computation Slice with respect to (x ≥1) Λ (z ≤3)

30 Slice slice: a sub-trace such that:  it contains all consistent cuts of the trace satisfying the given predicate  it contains the least number of consistent cuts [Garg and Mittal 01, Mittal and Garg 01] predicate trace slice Slicer

31 Results Efficient polynomial-time algorithms for computing the slice for: – linear predicates: [Garg and Mittal 01] time-complexity: O(n 2 m) – general predicate: Theorem: Given a computation, if a predicate b can be detected efficiently then the slice for b can also be computed efficiently. [Mittal,Sen and Garg 03] – combining slices: Boolean operators – temporal logic operators: EF, AG, EG – approximate slice: For arbitrary boolean expression n: number of processes m: number of events

32 POTA Architecture [Sen and Garg 04] Instrumentor Specification Slicer Predicate Detector Trace Slice Predicate (Specification) Translator Execute Program Execute SPIN Program Instrumented Program Promela TraceSlice yes/ witness no/ counter example no/ counter example yes Analyzer

33 Experiments: Dining Philosophers Trace Verification  POTA: Partial Order Trace Analyzer (based on slicing) [Sen and Garg 03]  SPIN: A widely used model checking tool [Holzmann 97] – SPIN: 250 seconds for n = 6, runs out of memory for n > 6. – POTA: can handle n= 200. Used 400 seconds. Predicate: Two neighboring dining philosophers do not eat concurrently

34 Supervisor: Motivation for Control  maintain global invariants or proper order of events – Examples: Distributed Debugging ensure that busy1 V busy2 is always true ensure that m 1 is delivered before m 2  Fault tolerance On fault, rollback and execute under control

35 Rollback Recovery for Software Faults Re-execution Problem:  To re-execute in order to avoid a recurrence of a previously detected failure – Progressive Retry [Wang et al 97] – Controlled Re-execution [Tarafdar and Garg 98] P1P1 P2P2 P3P3 faulty state restored state

36 Controlled Re-execution Add the synchronization necessary to maintain safety property – e.g., mutual exclusion Critical section P1P1 P2P2 a G1G1 c db

37 Results Efficient algorithms for computing the synchronization for: – Locks [Tarafdar, Garg DISC98] O(nm) algorithm for various types of locks – disjunctive predicate [Mittal, Garg 00] e.g., (n-1)-mutual exclusion time-complexity:} O(m 2 ) minimizes the number of synchronization arrows – region predicate [Mittal, Garg PODC 00] e.g., virtual clocks of processes are “approximately” synchronized time-complexity:O(nm 2 ) maximizes the concurrency in the controlled computation n: number of processes, m: number of events

38 Conclusions  Efficient algorithms possible for monitoring global properties  Observation and Control: a powerful abstraction  Current execution engines are designed for performance rather than fault-tolerance

39 Other Research Projects  Distributed simulation: GVT algorithms, fault- tolerance  Recovery Schemes: Optimistic Message Logging, fault-tolerance without replication  Model Checking: Partial Order Methods  Formal Methods: Petri Nets, Lattice Theory, Max Plus Algebra

40 Questions ??