1. 1 Compositional Design and Analysis of Timing-Based Distributed Algorithms Nancy Lynch Theory of Distributed Systems MIT Third MURI Workshop Washington, D.C. December 10, 2002

2. 2 MIT Participants Leader –Nancy Lynch Postdoctoral associates –Idit Keidar, Dilsun Kirli Graduate students –Roger Khazan, Carl Livadas, Ziv Bar-Joseph, Rui Fan, Seth Gilbert, Sayan Mitra Collaborators –Alex Shvartsman and students, Frits Vaandrager, Roberto Segala

3. 3 I. Project Overview Design and analyze timing-based distributed algorithms that implement global services with strong guarantees: –Reliable communication –Strongly coherent data services –Consensus –… Many of the algorithms work in a dynamic environment, tolerating joins, leaves, and failures. Prove correctness. Analyze performance conditionally, under various assumptions about timing and failures. Develop underlying semantic modeling framework, based on interacting state machines (IOA, TIOA, HIOA,…) … Net …

4. 4 Conditional performance analysis Give conditional claims about system performance under particular assumptions about behavior of environment and of network substrate, e.g.: –Stabilization of underlying network. –Limited rate of change. –Bounds on message delay. –Limited amount of failure (number, density). –Limited input arrivals (number, density). Assumptions  guarantees. Get probabilistic statements as corollaries. Composable

5. 5 Algorithms studied Scalable group communication [Khazan, Keidar] Early-delivery dynamic atomic broadcast [Bar-Joseph, Keidar, Lynch] Scalable reliable multicast [Livadas, Keidar, Lynch] Reconfigurable atomic memory [Lynch, Shvartsman] In progress: –Rambo extensions [Gilbert, Musial, Lynch, Shvartsman] –Concurrency control using metadata [Fan] –Consensus [De Prisco, Lynch, Shvartsman] –Mobile: Clock synchronization, tracking –Peer-to-peer: Fault-tolerant location services, data services

6. 6 This Talk I.Introduction  II.Completed work: I.Scalable group communication II.Early-delivery dynamic atomic broadcast III.Scalable reliable multicast IV.Reconfigurable atomic memory V.Modeling framework VI.Plans for next two years

7. 7 II. Completed work

8. 8 Scalable Group Communication [Keidar, Khazan 00, 02], [Khazan 02], [Keidar, Khazan, Lynch, Shvartsman 02] [Taraschanskiy 00] GCS

9. 9 Group Communication Services Cope with changing participants using abstract groups of client processes with changing membership sets. Processes communicate with group members indirectly, by sending messages to the group as a whole. GC services support management of groups: –Maintain membership information. Form new views in response to changes. –Manage communication. Communication respects views. Provide guarantees about ordering, reliability of message delivery. Virtual synchrony Systems; Isis, Transis, Totem, Ensemble,… GCS

10. 10 Group Communication Services High-level programming abstraction Hides complexity of coping with changes Applications: –Managing replicated data –Distributed multiplayer interactive games –Multi-media conferencing, collaborative work Disadvantages: –Can be costly, especially when forming new views. –May have problems scaling to large networks.

11. 11 Scalable GC Algorithm Specification, including virtual synchrony. New algorithm: –Uses a scalable membership service, implemented on a small set of membership servers [Keidar, Sussman, Marzullo, Dolev]. –Multicast implemented on all the nodes. –View change uses only one round for state exchange, in parallel with membership service’s agreement on views. –Participants can join during view formation. GCS Net Memb GCS

12. 12 Models and analysis Safety proofs, using new incremental proof methods. Liveness proofs Performance analysis: –Analyze time from when network stabilizes until the GCS announces the final view. –Analyze message latency. –Conditional analysis, based on input, failure, and timing assumptions. –Compositional analysis, based on performance of Membership service and Net. Modeled and analyzed data-management application running on top of the new GCS. Distributed implementation. SS’ AA’

13. 13 Early-Delivery Dynamic Atomic Broadcast [Bar-Joseph, Keidar, Lynch 02] DAB

14. 14 Dynamic Atomic Broadcast Atomic bcast, in a setting where processes may join, leave, or fail. Participants receive consistent sequences of messages. Safety: Sending, receiving orders are consistent with a single global message ordering S. No gaps. Liveness: Eventual join-ack, leave-ack. Eventual delivery, including the first message the process itself sends. Strong latency guarantees: Fast delivery, even with joins, leaves. Application: Distributed multiplayer interactive games. join leave mcast(m) join-ack leave-ack rcv(m) join-ack join … DAB

15. 15

16. 16

17. 17

18. 18

19. 19

20. 20

21. 21

22. 22

23. 23

24. 24

25. 25

26. 26

27. 27

28. 28

29. 29

30. 30

31. 31

32. 32

33. 33

34. 34

35. 35

36. 36

37. 37

38. 38

39. 39

40. 40

41. 41

42. 42

43. 43

44. 44

45. 45

46. 46 General Models and Proof Methods I/O automaton models [Lynch, Tuttle 87] –Nondeterministic, infinite-state machines –Input/output/internal actions, traces –Modularity: Composition, levels of abstraction Mathematical, language-independent Used to model distributed algorithms, communication protocols Validation, code generation, upper and lower bounds

47. 47 Timing, Hybrid Considerations Timing: TIOAs [Lynch, Vaandrager] –Timeout-based algorithms. –Local clocks, clock synchronization –Performance analysis Hybrid: HIOAs [L, Segala, V, Weinberg 96] –Real world + computer components –Continuous flows of data

48. 48 Other Embellishments Probabilities: PIOA, PTIOA [ Segala 95] –Probabilistic and nondeterministic behavior. –Randomized distributed algorithms –Systems with probabilistic assumptions Dynamic systems: DIOA [Attie, Lynch 99] –Run-time process creation and destruction, mobility. –Agent systems

49. 49 Hybrid I/O Automata [Lynch, Segala, Vaandrager, HSCC 01] New, simpler version of HIOA model of [LSVW96] Supports decomposing hybrid system descriptions: –External behavior: Discrete actions and continuous flows –Composition: Synchronizes external actions and flows, respects external behavior –Abstraction: Implementation and simulation relation notions, respect external behavior. Separate mechanisms: –External actions for discrete communication. –External variables for continuous flow.

50. 50 Example: Delay Buffer Del(d) Accepts discrete and continuous input, produces isomorphic output, with delay d. Compose in sequence, in cycle: Composition implements Del(d1 + d2): Del(d1)Del(d2) Del(d1)Del(d2)Del(d1)Del(d2) Del(d1 + d2)

51. 51 Example: Vehicle and Controller Keep vehicle speed in [v1, v2]. Sensor senses velocity, reports to Controller every time d. Controller suggests acceleration. Vehicle follows suggested acceleration, with uncertainty ε. Compose: Discrete, continuous interactions Prove invariant: velocity in [v1,v2]. Use auxiliary invariants, including timing. Vehicle SensorActuator Controller report(v) vel-out suggest(a) acc-in

52. 52 HOIA definition U, X, Y: Input, output, internal (state) variables Θ: Initial states I, O, H: Input, output, internal actions D, discrete transitions T, trajectories –Mappings from time intervals to valuations of variables Closure properties Input-enabling for actions, flows Execution: τ0, a1, τ1, a2, τ2, … Trace: Restrict to external variables and actions

53. 53 Composition and Abstraction Abstraction: –A implements B if comparable and traces(A) subset of traces(B). –Simulation relation: Start, step, trajectory conditions –Theorem: Simulation relation implies implementation Composition: –Synchronize external actions and variables –Theorems: Projection, pasting, substitutivity Receptiveness: –Doesn’t cooperative in producing Zeno behavior –Theorem: Closed under composition (with technical assumption).

54. 54 VI. Plans for the next two years

55. 55 Plans: Distributed Algorithms Scalable Reliable Multicast –Analyze SRM in the presence of leaves and node failures. –Finish analysis of CESRM. –Study LMS reliable multicast protocol [Papadopoulos, Varghese 98], compare with SRM, CESRM. Reconfigurable Atomic Memory –Optimizations: concurrent gc, remove reads,… –Experiments, demo applications Paxos consensus Mobile: clock synchronization, tracking, layers Peer-to-peer: location service with provable fault-tolerance guarantees, under steady-state assumptions. Building strongly coherent data service over location service.

56. 56 Plans: Semantic framework Timed models: –Composition theorems for timing properties –Specially structured TIOAs for conditional performance analysis Hybrid models: –Integrate control theory methods. Probabilistic models: –Compositional analysis methods –Combine hybrid and probabilistic models