FAULT-TOLERANT NETWORKS AND FAULT-TOLERANT ROUTING SONER DEDEOĞLU 10/12/2015 1

Outline Introduction Measures of Resilience ▫Graph-Theoretical Measures ▫Computer Networks Measures Common Network Topologies and Their Resilience ▫Multi-stage and Extra-Stage Networks ▫Crossbar Networks ▫Regular Mesh and Interstitial Mesh ▫Hypercube Network Fault Tolerant Routing ▫Hypercube Fault-Tolerant Routing ▫Origin-Based Routing in the Mesh 10/12/ SONER DEDEOĞLU

10/12/2015 3

Types of Interconnection Networks In shared-memory multiprocessor systems, connecting processors and memories ▫Processors read from or write into memories. In distributed systems, connecting processors ▫Each has its own local memory ▫Communicate through messages while executing parts of a common application Wide-area-networks, connecting large number of processors that operate independently ▫Information sharing through packets ▫Communication over switchboxes and routers 10/12/ SONER DEDEOĞLU

Network Topology Definition: Network organization One or more paths between source and the destination Uni- or Bi-directional links and switchboxes. If a single path exists between sender and receiver, one fault along the path will disconnect the communication terminals. Fault tolerance is achieved by having multiple paths and/or spare units. 10/12/2015SONER DEDEOĞLU 5

10/12/2015 6

Graph Theoretical Measures Node (link) connectivity ▫Minimum number of nodes (links) that must be removed to disconnect the graph Distance between nodes ▫Smallest number of links Diameter ▫Longest distance in the graph Diameter stability ▫Rate of increase in diameter due to faulty nodes. ▫Persistence: Smallest number of nodes that must fail in order to increase the diameter of the graph 10/12/2015SONER DEDEOĞLU 7

Computer Network Measures Reliability - R(t) ▫Probability that all nodes are operational and communicate during the time interval [0,t] ▫Path reliability: Same definition for a specific sender-receiver pair. Bandwidth ▫Maximum rate of flow of messages Connectability – Q(t) ▫Expected number of source-destination pairs still connected at time t in the presence of faults 10/12/2015SONER DEDEOĞLU 8

10/12/2015 9

Types of topologies TYPE-1 ▫Input and output nodes are connected through links and switchboxes.  Multi-stage networks  Crossbar ▫Resilience measures  Bandwidth  Connectivity TYPE-2 ▫Nodes are not connected through switchboxes, but links. Nodes are both computational units and also serve as switches.  Mesh  Hypercube ▫Resilience measure  (Path) Reliability 10/12/2015SONER DEDEOĞLU 10

Multi-stage Networks Butterfly network ▫Built out of 2x2 switches – Two input and two output ports  Straight  Cross  Upper broadcast  Lower broadcast 10/12/2015SONER DEDEOĞLU 11

Multi-stage Networks (cont’d) k-stage butterfly network ▫2 k inputs and 2 k outputs ▫2 k-1 switches in each stage ▫Connections follow a recursive pattern from input to output. ▫There is only one path from any given input to a specific output. (NOT FAULT TOLERANT) ▫If a fail occurs, the system still operates but in a degraded mode. 10/12/2015 SONER DEDEOĞLU 12

Extra-stage Networks Duplication of stage-0 at the input of network Bypass multiplexers around switchboxes at the input and output stages. A failed switch can be bypassed by routing around it. FAULT TOLERANT up to one faulty switchbox anywhere in the system. 10/12/2015 SONER DEDEOĞLU 13

Resilience Analysis of Butterfly Bandwidth Calculation (without failures) Bandwidth ▫Expected number of access requests from the processors that reach the memories Assumptions ▫Each processor generate in each cycle, with a probability p r, a request to a memory module, directed to any of the N memory modules with equal probability 1/N. ▫Requests in each cycle are independent from requests in previous cycles. 10/12/2015SONER DEDEOĞLU 14

Resilience Analysis of Butterfly Bandwidth Calculation (without failures) (cont’d) p r (i) : probability that a link at stage i carries a request – calculated recuresively from processors (i=k-1) to memories (i=0). Stage k-1 : Two processors feed each link: ▫Union of two independent events with the probability p r /2 for each. p r (k-1) = p r /2 + p r /2 - (p r /2) 2 = p r – p r 2 /4 ▫Recursively for a link in stage i-1 (i = k-1, …, 1) p r (i-1) = p r (i) – (p r (i-1) ) 2 /4 ▫and BW = Np r (0) 10/12/2015SONER DEDEOĞLU 15

Resilience Analysis of Butterfly Bandwidth Calculation (including failures) q l : probability of link failure p l – q l : probability of fault-free link p l p r (i) / 2: probability that a request at the input line to stage I will propagate to an output in stage i Resulting recursive equation is p r (i-1) = p l p r (i) – (p l p r (i) ) 2 /4 and BW = Np r (0) 10/12/2015SONER DEDEOĞLU 16

Resilience Analysis of Butterfly Connectability (including failures) Connectability ▫Expected number of connected sender-receiver pairs Senders and receivers are fault-free. Exactly one path exists between each pair. Each path has k+1 links k switchboxes. Probabilities of link and switchbox failures ▫ q l and q s, respectively. Probabilities of fault-free link and switchbox ▫p l = 1 – q l and p s = 1 – q s, respectively. Probability of a fault-free path ▫p l k+1 p s k There are 2 2k = N 2 sender-receiver pairs. Q = 2 2k p l k+1 p s k = N 2 p l k+1 p s k 10/12/2015SONER DEDEOĞLU 17

Resilience Analysis of Extra-Stage Connectability Each processor-memory pair connected by two disjoint paths ▫Probability of at least one fault-free path = Prob{1 st path is fault-free} + Prob{2 nd path is fault-free} – Prob{both paths are fault-free} This probability can assume one of the following two expressions A = (1-q l 2 ) p l k (1-q l 2 ) + p l k+2 – p l 2 B = 2(1-q l 2 ) p l k+1 – p l 2k+2 (1-q l 2 ) 2 (1-q l 2 ) is the probability that or a switchbox with a bypass multiplexer at least one link is operational There are 2 2k = N 2 sender-receiver pairs. Q = (A+B)2 2k /2 = (A+B)N 2 /2 10/12/2015SONER DEDEOĞLU 18

Crossbar Networks A switchbox for each sender- receiver pair. NxM crossbar: ▫N-senders, M-receivers, NM- switches The (i, j) switchbox connects row i input to column j output and is capable of ▫propagating a message along row from left link to right link ▫propagating a message along column from bottom to top link ▫turning a message from left link to top link Failure of any switchbox will disconnect certain pairs (NOT FAULT TOLERANT) 10/12/2015 SONER DEDEOĞLU 19

Redundant Crossbars A row and a column of switches are added. Input and output connections are augmented. ▫Each input can be send either of two rows and can be received on either two columns. If a switch becomes faulty, row and column to which it belongs are replace by the space row and column (FAULT TOLERANT) 10/12/2015 SONER DEDEOĞLU 20

Resilience Analysis of Crossbar Connectability Probability that a link is faulty: q l Probability that a link is fault-free: p l = 1 – q l Probability fo switchbox failures included in link failure probabilities. For input i to be connected to output j, we have to go through i+j links. 10/12/2015SONER DEDEOĞLU 21

Mesh Networks 2-Dimensional NxM rectangular mesh network ▫All nodes are computing ▫No separate switchboxes Message sending ▫A path from source to destination identified and message forwarded along path Conventional mesh reliability ▫R mesh (t) = [R(t)] NM ▫R(t)  reliability of single node NOT FAULT TOLERANT 10/12/2015 SONER DEDEOĞLU 22

Interstitial Mesh Networks (1, 4) Interstitial redundancy ▫A spare node can be switched in to replace any neighbor failed. ▫Each primary node has a single spare node while each spare node is a spare node for primary nodes ▫Redundancy overhead = 25% (4, 4) Interstitial redundancy ▫Primary node has four spare nodes. ▫Each spare node is a spare for four primary nodes. ▫Redundancy overhead = 100% 10/12/2015 SONER DEDEOĞLU 23

Resilience Analysis of Interstitial Mesh Reliability (4,1) Interstitial Mesh ▫Mesh is size of NxM where N and M are even numbers ▫Cluster: four primary nodes with one spare ▫Mesh has NM/4 clusters at all. ▫R(t) : Reliability of a single primary or spare node ▫Reliability of cluster R cluster (t)= R 5 (t) + 5R 4 (t)[1-R(t)] ▫Reliability of mesh R mesh (t) = [R cluster (t)] NM/4 (4,4) Interstitial Mesh ▫No simple algorithm to calculate reliability of (4,4) interstitial mesh 10/12/2015SONER DEDEOĞLU 24

Hypercube Networks H n : An n-dimensional hypercube network including 2 n nodes. A 0-dimensional hypercube H 0 is a single node. H n constructed by connecting the corresponding nodes of two H n-1 networks. The edges added to connect the corresponding nodes are called dimension-(n-1) edges. Each node in an n-dimensional hypercube has n edges incident upon it. 10/12/2015SONER DEDEOĞLU 25

Hypercube Networks (cont’d) 10/12/2015SONER DEDEOĞLU 26

Fault Tolerance to Hypercubes Hn (n ≥ 2) can tolerate link failures by multiple path between source and destination. Node failures can disrupt the operation. Adding fault tolerance to overcome node failures ▫Adding one or more spare nodes ▫Increasing number of communication ports of each original node from n to n+1 ▫Connecting the extra ports through the additional links to spare node. ▫Using crossbar switches with outputs connected to spare node reduces number of ports of spare node to n+1. 10/12/2015SONER DEDEOĞLU 27

Fault Tolerance to Hypercubes (cont’d) 10/12/2015SONER DEDEOĞLU 28

Resilience Analysis of Hypercube Reliability Links and nodes all fail independently Reliability of H n is the product of ▫Reliability of 2 n nodes, ▫Probability that every node can communicate with every other node Exact evaluation of the probability is difficult due to multiple-path connection between source-destination pairs. Lower bound on the reliability can be evaluated as addition of probabilities of three mutually exclusive cases for which the network is connected. 10/12/2015SONER DEDEOĞLU 29

Resilience Analysis of Hypercube Reliability (cont’d) H n is decomposed into two H n-1 hypercubes, A and B, and the dimesion- (n-1) links connecting them. CASE-1: ▫Both A and B are operational and at least one of dimension-(n-1) link is functional. CASE-2: ▫One of A, B is operational and the other is not, and all dimension-(n-1) links are functional. CASE-3: ▫Only one of A,B is operational, exactly one dimension-(n-1) link is faulty and is connected in the nonoperational H n-1 to a node that has at least one functional link to another node. 10/12/2015SONER DEDEOĞLU 30

Resilience Analysis of Hypercube Reliability (cont’d) Notations ▫q c : Probability of a node failure ▫q l : Probability of a link failure ▫NR(H n, q l, q c ) : Reliability of hypercube H n Assumption ▫Nodes are perfect reliable (q c = 0) CASE-1: Prob{Case-1} = [NR(H n-1, q l, 0)] 2 (1 - q l 2 n-1 ) CASE-2: Prob{Case-2} = 2NR(H n-1, q l, 0) [1-NR(H n-1, q l, 0)] (1 - q l ) 2 n-1 CASE-3: Prob{Case-3} = 2NR(H n-1, q l, 0) [1-NR(H n-1, q l, 0)] 2 n-1 q l (1-q l ) 2 n-1 -1 (1-q l n-1 ) NR(H n, q l, 0) = Prob{Case-1} + Prob{Case-2} + Prob{Case-3} 10/12/2015SONER DEDEOĞLU 31

10/12/2015 SONER DEDEOĞLU 32

Concepts Objective: ▫Get a message from source to destination despite a subset of the network being faulty. Basic Idea: ▫If no shortest or most convenient path is available because of failures, reroute message throught other paths to destination. Unicast Routing: ▫A message is sent from a source to just a one destination Multicast Routing: ▫Copies of a message sent to a number of nodes. 10/12/2015SONER DEDEOĞLU 33

Classification of Routing Algorithms Centralized routing ▫A central controller knows the network state – faulty links or nodes, congested links - and selects path for each message. Distributed routing ▫Each intermediate node decides to which node to send it next. Unique routing ▫One path for each source-destination pair Adaptive routing ▫Path selected according to network conditions (congestion) 10/12/2015SONER DEDEOĞLU 34

Hypercube Fault Tolerant Routing Basic Idea: ▫List the dimensions along which the packet must travel, and traverse them one by one. ▫As edges are traversed and are crossed off the list. ▫If, due to a link or node failure, the desired link is not available, another edge in the list, if any, is chosen for traversal ▫If packet arrives at some node to find all dimensions on its list down, it backtracks to the previous node and tries again. 10/12/2015SONER DEDEOĞLU 35

Hypercube Fault Tolerant Routing (cont’d) TD: list of dimensions that the message has traveled on - in order of traversal TD R : list of dimensions in reversed order k i=1 : Exculsive or operation carried out k times sequentialy. D: destination S: source d = D S SR(A): the set of relative addresses reachable by traversing each of the dimensions listed in A. e n i : n-bit vector consisting of a 1 in the i th bit position and 0 everywhere else. : append operation TRANSMIT(j): Send packet (d e j, message payload, TD j) along the j th dimensional link from the present node 10/12/2015SONER DEDEOĞLU 36

Hypercube Fault Tolerant Routing (cont’d) 10/12/2015SONER DEDEOĞLU 37

Hypercube Fault Tolerant Routing (cont’d) Case study: ▫We are given an H 3 with faulty node 011. ▫S = 000 wants to send a message to D = 111. ▫At 000, d = 111, so it sends the message out on dimension-0, to node 001 ▫At node 001, d = 110 and TD = (0). This node attempts to send it out on its dimension-1 edge. However, because node 011 is down, it cannot do so. ▫Since bit 2 of d is also 1, it checks and finds that the dimension-2 edge to 101 is available. ▫The message is now sent to 101, from which it makes its way to /12/2015 SONER DEDEOĞLU 38

Origin Based Routing in Mesh Assumptions: ▫Two-dimensional NxN mesh with at most N-1 failures ▫All faulty regions are square, if not, additional nodes are declared to have pseudo faults ▫Each node knows the distance along each direction to the nearest faulty region in that direction ▫One node defined as the origin ▫Origin chosen so that its row and column do not have any faulty nodes 10/12/2015 SONER DEDEOĞLU 39

Origin Based Routing in Mesh (cont’d) Sending a message from S to D IN-path: Edges that take the message closer to the origin OUT-path: takes the message farther away from the origin Outbox: the smallest rectangular region that contains both the origin and the destination Safe node: V is a safe node with respect to D and a set of faulty nodes F if ▫V is in the outbox for D ▫There exists a fault-free OUT-path from V to D Diagonal band: Diagonal band for D - all nodes V in the outbox such that xv – yv = xD – yD + e where e {-1,0,1} Once we get to a safe node, there exists an OUT-path from that node to D 10/12/2015 SONER DEDEOĞLU 40

Origin Based Routing in Mesh (cont’d) Three Phase Algorithm PHASE-1: ▫The message is routed on an IN path until it reaches the outbox, at node U. PHASE-2: ▫Compute the distance from U to the nearest safe node and compare to the distance to the nearest faulty region in that direction. If the safe node is closer than the fault, route to the safe node; otherwise, continue to route on the IN links. PHASE-3: ▫Once the message is at a safe node U, if there is a safe nonfaulty neighbor V that is closer to the destination, send it to V ; otherwise, U must be on the edge of a faulty region. In such a case, move the message along the edge of ▫the faulty region toward the destination D, and turn toward the diagonal band when it arrives at the corner of the faulty square. 10/12/2015SONER DEDEOĞLU 41

Origin Based Routing in Mesh (cont’d) Case Study: ▫Routing a message from node S at northwest end of the network to D. ▫The message first moves along the IN links, getting closer to the origin. ▫It enters the outbox at node A. ▫Since there is a failure directly east of A, it continues on the IN links until it reaches the origin. ▫Then it continues, skirting the edge of the faulty region until it reaches node B. ▫At this point, it recognizes the existence of a safe. ▫node immediately to the north and sends the message through this node to the destination. 10/12/2015 SONER DEDEOĞLU 42

I. Koren and C.M. Krishna, Fault Tolerance Systems, Morgan Kaufmann, D. M. Blough and N. Bagherzadeh, “Near-Optimal Message Routing and Broadcasting in Faulty Hypercubes,” International Journal of Parallel Programming, Vol. 19, pp. 405–423, October R. Libeskind-Hadas and E. Brandt, “Origin-Based Fault-Tolerant Routing in the Mesh,” IEEE Symposium on High Performance Computer Architecture, pp. 102–111, /12/2015 SONER DEDEOĞLU 43

10/12/2015 SONER DEDEOĞLU 44