1. 1 Modeling and Analyzing Fault-Tolerant, Real-Time Communication Protocols Nancy Lynch Theory of Distributed Systems MIT Second MURI Workshop Berkeley, California June 4, 2001

2. 2 MIT Participants Leaders: Nancy Lynch, Idit Keidar Students: Carl Livadas, Roger Khazan, Ziv Bar-Joseph Collaborators: Paul Attie, Alex Shvartsman, Roberto Segala, Frits Vaandrager

3. 3 At Last Year’s Workshop…

4. 4 General Models and Proof Methods I/O automaton models [Lynch, Tuttle 87] –Nondeterministic, infinite-state machines –Input/output/internal actions, traces –Modularity: Composition, levels of abstraction Mathematical, language-independent Used to model distributed algorithms, communication protocols Validation, code generation, upper and lower bounds

5. 5 Timing, Hybrid Considerations Timing: TIOAs [Lynch, Vaandrager] –Timeout-based algorithms. –Local clocks, clock synchronization –Performance analysis Hybrid: HIOAs [L, Segala, V, Weinberg 96] –Real world + computer components –Continuous flows of data

6. 6 Other Embellishments Probabilities: PIOA, PTIOA [ Segala 95] –Probabilistic and nondeterministic behavior. –Randomized distributed algorithms –Systems with probabilistic assumptions Dynamic systems: DIOA [Attie, Lynch 99] –Run-time process creation and destruction, mobility. –Agent systems

7. 7 Communication Protocol Modeling and Analysis. At-most-once (AMO) Message Delivery TCP, T/TCP Reliable channels from unreliable channels Self-stabilizing communication protocols Network clock synchronization Group communication systems

8. 8 Group Communication Service Communication middleware Manages group membership, current view Handles joins, leaves, failures, partitions, merges Multicast communication among members –Multicasts respect views –Ordering, reliability constraints for message delivery, e.g., FIFO, causal within each view. Isis, Transis, Totem,…

9. 9 VStoTO VS VStoTO TO bcast brcv gpsnd gprcv newview

10. 10 Conditional Performance Analysis Assume VS satisfies: –If a network component C stabilizes, then soon thereafter, views become consistent within C, and messages sent in the final view are delivered everywhere in C, within bounded time. And VStoTO satisfies: –Simple timing, fault-tolerance assumptions. Then TO satisfies: –If C stabilizes, then soon thereafter, any message sent or delivered anywhere in C is delivered everywhere in C, within bounded time.

11. 11 Conditional Performance Analysis Give conditional claims about system performance under particular assumptions about behavior of environment and of network substrate, e.g.: –Stabilization of underlying network. –Limited rate of change. –Bounds on message delay. –Limited amount of failure (number, density). –Limited input arrivals (number, density). Assumptions => Guarantees. Get probabilistic statements as corollaries. Composable

12. 12 What we proposed: 1. Model, analyze communication protocols. 2. Develop conditional performance analysis techniques. 3. Extend I/O automata theory to accommodate performance, reliability, hybrid, probability, dynamic considerations. 4. Relate, integrate I/O automata with other frameworks.

13. 13 Progress this year 1. Communication protocol design/analysis –Scalable Group Communication –Totally Ordered Multicast with QoS –Scalable Reliable Multicast 2.Conditional performance analysis methods –Evolving… 3. I/O automaton models –Hybrid I/O Automata –Dynamic I/O Automata –IOA language support 4. Comparing, integrating with other models – A start…

14. 14 1.Protocol Modeling/Analysis

15. 15 Scalable Group Communication [Keidar, Khazan 00]

16. 16 Group Communication Service Manages group membership, current view. Multicast communication among group members, with ordering, reliability guarantees. Virtual Synchrony [Birman, Joseph 87] –Integrates group membership and group communication. –Processes that move together from one view to another deliver the same messages in the first view. –Useful for replicated data management. –Before announcing new view, processes must synchronize, exchange messages.

17. 17 Example: Virtual Synchrony 3: i,j,k 4: i, j mcast(m) rcv(m) VS algorithm supplies missing m ijk

18. 18 Group Communication in WANs Difficulties: –High message latency, message exchanges are expensive –Frequent connectivity changes New, scalable GC algorithm: –Uses scalable GM service of [Keidar, Sussman, et al. 00], implemented on special membership servers. –GC (with virtual synchrony) implemented on clients. VS Net GM VSGC

19. 19 Group Communication in WANs Try to minimize time from when network stabilizes until GC delivers new views to clients. After stabilization: GM forms view, VSGC algorithm synchronizes. Existing systems (LANs): –GM, VSGC uses several message exchange rounds –Continue in spite of new network events Inappropriate for WANs GM Algorithm VSGC Algorithm Net event view(v)

20. 20 New Algorithm VSGC uses one message exchange round, in parallel with GM’s agreement on views. GM usually delivers views in one message exchange. Responds to new network events during reconfiguration: –GM produces new membership sets –VSGC responds to membership changes Distributed implementation [Tarashchanskiy 00] GM Algorithm VSGC Algorithm Net event view(v)

21. 21 Correctness Proofs Models, proofs (safety and liveness) Developed new incremental modeling, proof methods [Keidar, Khazan, Lynch, Shvartsman 00] –Proof Extension Theorem: Used new methods for the safety proofs. SS’ AA’

22. 22 Performance Analysis Analyze time from when network stabilizes until GC delivers new views to clients. Compare with other strategies. System is a composition: –Network and GM services, plus –VSGC processes Use composition in the analysis.

23. 23 Performance Analysis 1.Analyze the VSGC algorithm alone, in terms of its inputs and timing assumptions. 2.State reasonable performance guarantees for GM and Network. 3.Combine to get conditional performance properties for the system as a whole.

24. 24 1. Analysis of VSGC algorithm Assume component C stabilizes: –GM delivers same views –Net provides reliable communication with latency . Let –T[start], T[view] be times of last GM events for C – be upper bound on local step time. Then VSGC outputs new views by time max (T[start] +  + x, T[view]) +

25. 25 Analysis of VSGC Algorithm GM algorithm VS Algorithm Net Event  + x T[start]T[view] view(v) start

26. 26 2. Assumed Bounds for GM Bounds for “Fast Path” of [Keidar, et al. 00], observed empirically in almost all cases. GM   T[start]T[view] start view(v)

27. 27 VSGC  + x GM   T[start]T[view] start view(v) 3. Combining VSGC and GM Bounds Bounds for system, conditional on GM bounds. view(v)

28. 28 Totally Ordered Multicast with QoS [Bar-Joseph, Keidar, Anker, Lynch 00]

29. 29 Totally Ordered Multicast with QoS Multicast to dynamic group, subject to joins, leaves, and failures. Global total ordering of messages QoS: Message delivery latency Built on reliable network with latency guarantees Add ordering guarantees, preserve latency bounds. Target applications –State machine replication –Military command and control –Distributed games –Shared editing

30. 30 Two Algorithms Algorithm 1: Basic Totally Ordered Multicast –Sends, receives consistent with total ordering of messages. –Non-failing processes agree on messages from non-failing processes. –Latency: Constant, even with joins, leaves, failures. Algorithm 2: Atomic Multicast –Non-failing processes agree on all messages. –Latency: Joins, leaves only: Constant With failures: Linear in f Net TOM fail_i fail_j

31. 31 FrontEnd_i Memb_i Sniffer_i Net Ord_i rcv(m) join leave mcast(m) join leave mcast(join) mcast(leave) rcv(join) rcv(leave) mcast(m) rcv(m) progress(s,j) joiners(s,J), leavers(s,J) end-slot(s) members(s,J) Local Node Process

32. 32 Local Algorithm Operation FrontEnd divides time into slots, tags messages with slots. Ord delivers messages by slot, in order of process indices. Memb determines slot membership. –Join, leave messages –Failures: Algorithm 1 uses local failure detector. Algorithm 2 uses consensus on failures. –Requires new dynamic version of consensus. Timing-dependent

33. 33 Architecture for Algorithm 2 Net GM TO-QoS

34. 34 Performance Analysis (Planned) 1. Latency of TO-QoS in terms of GM 2. GM latency bounds 3. Combine

35. 35 Using Caching to Improve Reliable Multicast Algorithms [Livadas]

36. 36 SRM [Floyd, et al.] Reliable multicast to dynamic group. Built over IP multicast Based on requests (NACKs) and retransmissions Limits duplicate requests/replies using: –Deterministic suppression: Ancestors suppress descendants, by scheduling requests/replies based on distance to source. –Probabilistic suppression: Siblings suppress each other, by spreading out requests/replies.

37. 37 SRM Architecture IPMcast SRM

38. 38 New Protocol Tries to improve SRM by using loss history information. –Useful if future losses occur on same link. Uses deterministic suppression for siblings also Determines, caches best requestor, best replier –Chooses requestor closest to source. –Chooses replier closest to requestor. –Break ties with processor ids. Defaults to SRM Requestor Replier

39. 39 Performance Metrics: –Loss recovery latency: Time from detection of packet loss to receipt of first retransmission –Loss recovery overhead: Number of messages multicast to recover from a message loss Protocol performance benefits: –Removes delays caused by probabilistic suppression –Following election of requestor and replier: Reduces latency by using best requestor and replier. Reduces overhead by using single requestor and replier. Latency analysis (Planned)

40. 40 3. I/O Automaton Models

41. 41 Hybrid I/O Automata [Lynch, Segala, Vaandrager, HSCC 01] New, simpler version of HIOA model of [LSVW96] Supports decomposing hybrid system descriptions: –External behavior: Discrete actions and continuous flows –Composition: Synchronizes external actions and flows, respects external behavior –Abstraction: Implementation and simulation relation notions, respect external behavior. Separate mechanisms: –External actions for discrete communication. –External variables for continuous flow.

42. 42 Example: Delay Buffer Del(d) Accepts discrete and continuous input, produces isomorphic output, with delay d. Compose in sequence, in cycle: Composition implements Del(d1 + d2): Del(d1)Del(d2) Del(d1)Del(d2)Del(d1)Del(d2) Del(d1 + d2)

43. 43 Example: Vehicle and Controller Keep vehicle speed in [v1, v2]. Sensor senses velocity, reports to Controller every time d. Controller suggests acceleration. Vehicle follows suggested acceleration, with uncertainty ε. Compose: Discrete, continuous interactions Prove invariant: velocity in [v1,v2]. Use auxiliary invariants, including timing. Vehicle SensorActuator Controller report(v) vel-out suggest(a) acc-in

44. 44 HOIA definition U, X, Y: Input, output, internal (state) variables Θ: Initial states I, O, H: Input, output, internal actions D, discrete transitions T, trajectories –Mappings from time intervals to valuations of variables Closure properties Input-enabling for actions, flows Execution: τ0, a1, τ1, a2, τ2, … Trace: Restrict to external variables and actions

45. 45 Composition and Abstraction Abstraction: –A implements B if comparable and traces(A) subset of traces(B). –Simulation relation: Start, step, trajectory conditions –Theorem: Simulation relation implies implementation Composition: –Synchronize external actions and variables –Theorems: Projection, pasting, substitutivity Receptiveness: –Doesn’t cooperative in producing Zeno behavior –Theorem: Closed under composition (with technical assumption).

46. 46 Dynamic I/O Automata [Attie, Lynch, Concur 01] Dynamic version of I/O automata, including: – Automaton creation and destruction – Signature change Two-level model: Automata, configurations. Mobility modeled using signature change

47. 47 IOA Language and Tools Language for describing I/O automata: [Garland, Lynch] Front end: [Garland] –Translates to Java objects –Completely rewritten this year. –Needs support for composition. Theorem-prover connection: [Garland, Bogdanov] –Connection with LP –Seeking connections: SAL, Isabelle, STeP, NuPRL I O A

48. 48 IOA Language and Tools Simulator: [Chefter, Ramirez, Dean] –Has support for paired simulation. –Needs additions. –Being instrumented for invariant discovery using Daikon [Ernst] Code generator: Tauber, Tsai –Local code-gen (translation to Java) running. –Needs composition, communication service calls, correctness proof. Challenge examples

49. 49 Plans

50. 50 Plans 1. Protocol modeling/verification –Finish analysis of Scalable GC, Totally Ordered Multicast with QoS, SRM –Other protocols from this project. 2. Conditional analysis methods –Develop general methods –Compare with other methods (Trivedi)

51. 51 Plans 3. I/O automaton models –Timed models: Composition theorems for timing properties Specially structured TIOAs for CP analysis –Hybrid models: Finish basic model Integrate control theory methods –Probabilistic models: Compositional analysis methods Combine hybrid and probabilistic models

52. 52 Plans I/O automaton models, cont’d –Dynamic models: External behavior notion, composition results –Combine extensions –Language constructs to support extensions –IOA tools: Finish, make portions available 4. Integration with other models/methods –Shared variable models

53. 53 Thank you!

54. 54 Best Requestor and Replier S Requestor Replier

55. 55 Latency Analysis (Planned) 1.Bound recovery latency assuming cache hits. 2.Bound latency for cache misses. 3.Combine 4.Compare with SRM

56. 56 1. Cache Hit Performance Recovery latency bounded by c t SA* + t AB + d t AB* + t BA, where: –S = source, –A = requestor, B = retransmitter –c,d : delay parameters of deterministic suppression –t SA*, t AB* : inter-host delay estimates –t AB,, t BA : actual inter-host delays

57. 57 2. Cache Miss Performance Depends on: –Loss location –Locations of requestor, retransmitter, … –Timing Needs analysis Hope: Similar to SRM