1. Failure Recovery of Overlay Tree-based Structures Ing. Vladimír Dynda Doc. RNDr. Ing. Petr Zemánek, CSc. (supervisor) Czech Technical University in Prague Faculty of Electrical Engineering Department of Computer Science and Engineering Doctoral Thesis

2. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Agenda Introduction Solution BR Platform Bypass Routing Leader Link Election Tree Reconnection Summary of Results Conclusion

3. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Agenda Introduction Solution BR Platform Bypass Routing Leader Link Election Tree Reconnection Summary of Results Conclusion

4. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures T = ( TM, CE ) Introduction Problem statement S = ( N, L ) TM CE FC T1T1 T0T0 T2T2 T3T3 T4T4 T5T5 T6T6 T R = ( TM\FC, CE’ ) 1

5. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Introduction Problem statement Failure recovery Reconnection of T 0, T 1,..., T N-1 into a restored network T R = ( TM \ FC, CE’ ) Correctness – T R is acyclic Completeness – T R contains all the fragments 2

6. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Introduction Problem statement Environment Asynchronous distributed system No central authority / no global knowledge Unlimited sizes of S and T Arbitrary traffic direction in T Failures Node failures only Fail stop failure model Failures must not split S 3

7. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Introduction Goals of the thesis Proposal of a generic recovery platform Illustration of the tree restoration methods Simulation & verification of the theoretical properties Survey of possible applications 4

8. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Introduction State of the art On-demand / preplanned recovery Preplanned methods Employ pre-computed backup structures Existing preplanned methods Complete graph (Narada) Ancestor list (Yang-Fei, EFTMRP, HMTP) Administrative hierarchy (Nice, Nemo) Secondary trees (Dual-tree, Coop-net) Link to random nodes (HMTP, Yoid) 5

9. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Introduction State of the art Weaknesses of the existing methods Poor scalability Restricted set of applicable trees Single points of failure Fixed level of fault tolerance Unrecoverable multiple failures Non-local restoration 6

10. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Agenda Introduction Solution BR Platform Bypass Routing Leader Link Election Tree Reconnection Summary of Results Conclusion

11. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures BR Platform Bypass ring platform Ensures correctness and completeness Forms a basis for a tree reconnection Fabric of redundant links in T: Bypass rings of optional diameter Alternative paths in the event of failure Location & routing among the fragments 7

12. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures BR Platform Failure recovery T = ( TM, CE ) FC T R = ( TM\FC, CE’ ) BC( FC ) Leader link election Tree reconnection Leader n1n1 n2n2 n1n1 BR T (n 1,2) BR T (n 1,3) BR T (n 1,4) n2n2 BR T (n 2,2) Bypass routing Bypass rings 8

13. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures BR Platform Elemental steps of the recovery 1.Initialization of the platform 2.Failure detection 3.Designated nodes discovery 4.Leader link election 5.Tree reconnection 6.Bypass rings reconfiguration Bypass routing Correctness Completeness & 9

14. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Agenda Introduction Solution BR Platform Bypass Routing Leader Link Election Tree Reconnection Summary of Results Conclusion

15. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Bypass Routing Partially ordered tree (POT) T = ( TM, CE ) A0 09 67 93 B9 CE E8 1D 5E 42 F7 11 3C 72 17 B2 0F 79 9F 24 4A Seq T ( A0 ) Seq T ( 3C ) R + ( A0, 3C ) R - ( A0, 3C ) B T ( A0, 3C ) R + ( A0, 3C ) Ordered neighbor sequence Ordered rays 10

16. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures BR T (n,d max ) Bypass Routing Bypass ring BR T (n, d) n R-(n,n0)R-(n,n0) n0n0 n1n1 n2n2 n3n3 R +(n,n1)R +(n,n1) BR T (n,2) BR T (n,3) BR T (n,4) BT(n,n1)BT(n,n1) BT(n,n2)BT(n,n2) BT(n,n3)BT(n,n3) BT(n,n0)BT(n,n0) d max = 4 Seq T (n) R -(n,n1)R -(n,n1) R +(n,n2)R +(n,n2) R -(n,n2)R -(n,n2) R +(n,n3)R +(n,n3) R -(n,n3)R -(n,n3) R +(n,n0)R +(n,n0) 11

17. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Bypass Routing Bypass rings BT(n,n1)BT(n,n1) n n1n1 n2n2 n3n3 n4n4 n5n5 n d max R +(n,n1)R +(n,n1) BR T (n 1,2) BR T (n 1,3) BR T (n 2,4) BR T (n 2,5) BR T (n m,d max ) T = ( TM, CE ) FC 12

18. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Bypass Routing Routing algorithm T = B T (n i, n j ), n j  A T (n i )  FC FC R +(ni1,nj1)R +(ni1,nj1) BC( FC ) BT(ni2,nj2)BT(ni2,nj2) BT(ni3,nj3)BT(ni3,nj3) BT(ni1,nj1)BT(ni1,nj1) T = ( TM, CE ) ni3ni3 nj3nj3 ni1ni1 nj1nj1 ni2ni2 nj2nj2 13

19. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Bypass routing Example T = ( TM, CE ) FC BC( FC ) A0 09 67 93 B9 CE E8 1D 5E 42 F7 11 3C 72 17 B2 0F 79 9F 24 4A R + ( 72, 3C ) BR T ( 3C,2) BR T ( 3C,3) BR T ( A0,4) 14

20. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Bypass Routing Properties Memory overhead at node n  T: O(deg T (n) * d max ) Routing is successful if len X (n i, n i+1 )  d max, X = R + (n i, n j ) for all neighbors n i and n i+1  BC( FC ) Lower bound of maximum size of FC : d max /2 nodes for arbitrary clusters 15

21. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Agenda Introduction Solution BR Platform Bypass Routing Leader Link Election Tree Reconnection Summary of Results Conclusion

22. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Leader Link Election Leader link election (LLE) Guarantees correctness Communication structure – BC( FC ) Node states Passive – initial state of the election Active – leader candidates Relay – election is lost 16

23. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Leader Link Election LLE on ordered rings FC n0n0 n1n1 n2n2 n3n3 n4n4 n5n5 n6n6 n N-1 ID(n 0 ) < ID(n 1 ) <... < ID(n N-1 ) ELECTION( n 0 ) ID(n 0 ) < ID(n 1 ) ID(n 1 ) < ID(n 2 ) ELECTION( n 1 ) ID(n N-1 ) < ID(n 0 ) BC( FC ) = BR T (n,2) n Seq T (n) Leader 17

24. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures HID T ( 4F, 16 ) Leader Link Election LLE in partially ordered trees nrnr FC Seq T (n r ) BC( FC ) Leader Hierarchical identifier HID T (n r,n i ) 4F 97 D8 16 4F.A1.BA.D8 4F.A1.BA.97 4F.A1.16 HID T ( 4F, D8 ) HID T ( 4F, 97 ) A1 BA R+R+ Sweep process ELECTION( A1.BA.97 ) Seq T ( A1 ) A1.BA < A1.16 ELECTION( 4F.* ) SWEEP( 4F.A1 ) 18

25. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Leader Link Election Example T = ( TM, CE ) FC A0 09 B9 CE E8 1D 3C 72 B2 0F 67 93 5E 42 F7 11 17 79 9F 24 4A ELECTION( A0.B9.CE ) nrnr A0.B9 < A0.1D ELECTION( 3C.A0.1D ) 3C.A0 < 3C.A0 nrnr SWEEP( 3C.A0 ) Leader 19

26. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Leader Link Election Properties Average message complexity: O(N log b N); b is the average branching factor of FC nodes in T Time complexity: O(N) 20

27. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Agenda Introduction Solution BR Platform Bypass Routing Leader Link Election Tree Reconnection Summary of Results Conclusion

28. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Tree Reconnection Reconnection methods Reconnect the fragments located by the routing algorithm Abide by the results of LLE Designed to meet the specific application requirements Influence properties of the restored tree 21

29. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Tree Reconnection LR method n1n1 n3n3 n2n2 BC( FC ) 22

30. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures q 0 = = q 0 Tree Reconnection HR-x method n1n1 n3n3 n2n2 BC( FC ) = q 0 q1q1 q2q2 = q 3 q1q1 q2q2 q 5 = q3q3 q4q4 q1q1 q2q2 q3q3 (q 0, q i ) if i  1 (mod x) (q i-1, q i ) otherwise HR-1 23

31. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Tree Reconnection HR-x method n1n1 n3n3 n2n2 BC( FC ) (q 0, q i ) if i  1 (mod x) (q i-1, q i ) otherwise HR-2 24

32. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Tree Reconnection Example FC A0 09 B9 CE E8 1D 3C 72 B2 0F 67 93 5E 42 F7 11 17 79 9F 24 4A ELECTION( A0.B9.CE ) ELECTION( 3C.A0.1D ) SWEEP( 3C.A0 ) T R = ( TM\FC, CE’ ) HR-2 25

33. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Tree Reconnection Properties 26

34. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Tree Reconnection Properties 27

35. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Agenda Introduction Solution BR Platform Bypass Routing Leader Link Election Tree Reconnection Summary of Results Conclusion

36. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Summary of Results Properties of the BR platform Node memory overhead: O(deg T (n) * d max ) Average message complexity: O(N log b N) for arbitrary failures N for single failures Lower bound of max. recoverable failure: d max /2 nodes for arbitrary failed clusters d max -1 nodes for internal failed clusters 28

37. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Summary of Results Simulation results Successfully recovered cluster Average diameter: d max -2 Average size: 1.5 d max Linear recovery time d max parameter Controls fault-tolerance vs. costs d max =4 provides ample tolerance for GFS 29

38. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Summary of Results Properties of the platform Locality Multiple failure recovery Scalability Application requirements consideration Optional level of fault tolerance Protection selectivity Designated nodes discovery Tree reconnection method Independence of the protected tree type 30

39. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Summary of Results Applications Overlay multicast Applicable in all types Network-layer multicast Extension with BR(n,1) needed Sample application – GFS multicast Designed for large-scale P2P systems Based on a layered administrative hierarchy Employs BR platform to achieve fault- tolerance 31

40. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Agenda Introduction Solution BR Platform Bypass Routing Leader Link Election Tree Reconnection Summary of Results Conclusion

41. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Conclusion Thesis summary Analysis of overlay trees environment and identification of recovery properties Proposal of BR platform Design of the specialized leader election Illustration of the tree reconnection Simulation of the platform Outline of the overlay multicast scheme 32

42. Vladimír Dynda: Failure Recovery of Overlay Tree-based Structures Conclusion Ideas for further research Autonomous management of fault- tolerance level and protection selectivity More sophisticated tree reconnection methods Extension of the platform for network-layer multicast 33

43. Thank You