1. Lecture 22 Interconnection Networks Computer Architecture COE 501

2. Network Media Copper, 1mm think, twisted to avoid attenna effect (telephone) Used by cable companies: high BW, good noise immunity Light: 3 parts are cable, light source, light detector. Multimode light disperse (LED), Single mode sinle wave (laser) Twisted Pair: Coaxial Cable: Copper core Insulator Braided outer conductor Plastic Covering Fiber Optics Transmitter – L.E.D – Laser Diode Receiver – Photodiode light source Silica Total internal reflection Air

3. Shared vs. Switched Media Shared media: nodes share a single interconnection medium (e.g., Ethernet) –Only one message sent at a time –Inexpensive : one medium used by all processors –Limited bandwidth : medium becomes the bottleneck –Needs arbitration Switched media : allow direct communication between source and destination nodes (e.g., ATM) –Multiple users at a time –More expensive : need to replicated medium –Higher total bandwidth –No arbitration –Added latency to go through the switch

4. Ethernet Protocol - Shared Ethernet is an example of a shared media - similar to a bus The ethernet nodes must coordinate so that only one message is sent at a time. –Nodes “listens” to the network to ensure it is not being used. –If the network is not being used, the node tries to send the message. –If another node tries to send a message at the same time, a collision occurs –Both nodes detect the collision and try to resend their messages after a random amount of time –Subsequent collisons result in exponentially increasing random time between attempts to resend data –What are advantages and disadvantages of this method?

5. Switch Topology Switched media have a topology that indicate how nodes are connected Topology determines –Degree: number of links from a node –Diameter: max number of links crossed between nodes –Average distance: number of links to random destination –Bisection: minimum number of links that separate the network into two halves –Bisection bandwidth: link bandwidth x bisection Warning: Three-dimensional drawings must be mapped onto chips and boards which are essentially two-dimensional media –Elegant when sketched on the blackboard may look awkward when actually constructed

6. Common Topologies TypeDegreeDiameterAve DistBisection 1D mesh 2N-1N/31 2D mesh 42(N 1/2 - 1)2N 1/2 / 3N 1/2 3D mesh 63(N 1/3 - 1)3N 1/3 / 3N 2/3 nD mesh 2nn(N 1/n - 1)nN 1/n / 3N (n-1) / n Ring2N/2N/42 2D torus4N 1/2 N 1/2 / 22N 1/2 HypercubeLog 2 N n=Log 2 N n/2N/2 2D Tree32Log 2 N ~2Log 2 N 1 CrossbarN-11 1N 2 /2 N = number of nodes, n = dimension

7. Butterfly or Omega Network N/2 Butterfly °°°°°° N/2 Butterfly °°°°°° All paths equal length Unique path from any input to any output Try to avoid conflicts

8. Multistage Fat Tree A multistage fat tree (CM-5) avoids congestion at the root node Randomly assign packets to different paths on way up to spread the load Increase degree, decrease congestion

9. Example Networks (Figure 7.19, pg 591) NameNumberTopologyBitsClock Link Bis. BWYear nCube/ten1-102410-cube1 10 MHz 1.26401987 iPSC/216-1287-cube116 MHz23451988 MP-121632-5122D grid125 MHz31,3001989 Delta5402D grid1640 MHz406401991 CM-532-2048fat tree440 MHz2010,2401991 CS-232-1024fat tree870 MHz5050,0001992 Paragon4-10242D grid16100 MHz2006,4001992 T3D16-10243D Torus16150 MHz30019,2001993 No standard topology! MBytes/second

10. Connection-Based vs. Connectionless Connection-based system: –Telephone: operator sets up connection between the caller and the receiver –Once the connection is established, the line is held until it completes –Share transmission lines over long distances by using switches to multiplex several conversations on the same lines »“Time division multiplexing” divide B/W transmission line into a fixed number of slots, with each slot assigned to a conversation –Often referred to as circuit-swtiching –Problem: lines busy based on number of conversations, not amount of information sent –Advantage: reserved bandwidth - call not interrupted once started

11. Connection-Based vs. Connectionless Connectionless system –Postal service : package routed to destination based on address –Each message is divided into packets, with an address per packet –Packets can each take different routes through the network –Often referred to as packet-switching –Disadvantage : additional information sent with each message

12. Store and Forward vs. Cut-Through Store-and-forward policy: each switch waits for the full packet to arrive in the switch before it is sent on to the next switch Cut-through routing or worm hole routing: switch examines the header, decides where to send the message, and then starts forwarding it immediately –In worm hole routing, when the head of the message is blocked the message stays strung out over the network, potentially blocking other messages (needs only buffer the size of the packet that is sent between switches). –Cut through routing lets the tail continue when the head is blocked, storing the whole message into a single switch. (Requires a buffer large enough to hold the largest packet).

13. Packet Formats Fields: Destination, Checksum(C), Length(L), Type(T) Data/Header Sizes in bytes: –CM5: (4 to 20)/4, Ethernet: (0 to 1500)/26, ATM: 48/5

14. Congestion Control Packet switched networks do not reserve bandwidth; this leads to contention Solution: prevent packets from entering until contention is reduced (e.g., entrance lights on interstate) Options: –Packet discarding: If a packet arrives at a switch and there is no room in the buffer, the packet is discarded, SW detects and resends –Flow control: between pairs of receivers and senders; use feedback to tell the sender when it is allowed to send the next packet »Back-pressure: separate wires to tell to stop »Window: give the original sender the right to send N packets before getting permission to send more (TCP uses) –Choke packets: Each packet received by busy switch in warning state sent back to the source via choke packet. Source reduces traffic to that destination by a fixed % (ATM Forum)

15. Example Architectures InterconnectMPPLANWAN ExampleCM-5EthernetATM Topology“Fat” treeLine, BusVariable, constructed from multistage switches Connection based?NoNoYes Data Transfer SizeVariable: Variable: Fixed: 4 to 20B0 to 1500B48B Store & Forward?NoN/AYes Congestion controlAt source: At source: Rate based Flow Listen forvia choke control E-net idle packets via back pressure

16. Deterministic—follows a pre- specified route –mesh: dimension-order routing »(x 1, y 1 ) -> (x 2, y 2 ) »first  x = x 2 - x 1, »then  y = y 2 - y 1, –hypercube: edge-cube routing »X = x o x 1 x 2... x n -> Y = y o y 1 y 2... y n »R = X xor Y »Traverse dimensions of differing address in order Adaptive—route based on network state (e.g., contention) Routing 001 000 101 100 010 110 111 011

17. Practical Issues InterconnectionMPPLANWAN ExampleCM-5EthernetATM StandardNoYesYes Fault Tolerance?NoYesYes Hot Insert?NoYesYes Standards: required for WAN, LAN! Fault Tolerance: Can nodes fail and still deliver messages to other nodes? required for WAN, LAN! Hot Insert: If the interconnection can survive a failure, can it also continue operation while a new node is added to the interconnection? required for WAN, LAN!

18. Protocols Internetworking: allows computers on independent and incompatible networks to communicate reliably and efficiently; –Enabling technologies: SW standards that allow reliable communications without reliable networks –Hierarchy of layers, giving each layer responsibility for portion of overall communications task, called protocol families or protocol suites Transmission Control Protocol/Internet Protocol (TCP/IP) –This protocol family is the basis of the Internet –IP : layer above interconnection network - routes datagram to destination machine –TCP : layer above IP - guaranees reliable, in-order delivery and prevents corruption of data

19. TCP/IP packet Application sends message TCP breaks into 64KB segements, adds 20B header IP adds 20B header, sends to network If Ethernet, broken into 1500B packets with headers, trailers Header, trailers have length field, destination, window number, version,... TCP data (< 64KB) TCP Header IP Header IP Data Ethernet

20. Protocol Key to protocol families is that communication occurs logically at the same level of the protocol, called peer-to- peer, but is implemented via services at the lower level Danger is each level increases latency

21. Connecting Networks Bridges: connect LANs together, passing traffic from one side to another depending on the addresses in the packet. –operate at the Ethernet protocol level –usually simpler and cheaper than routers Routers or Gateways: these devices connect LANs to WANs or WANs to WANs and resolve incompatible addressing. –Generally slower than bridges, they operate at the internetworking protocol (IP) level –Routers divide the interconnect into separate smaller subnets, which simplifies manageability and improves security

22. Summary Interconnection Networks Shared vs. Switched media Switch topology indicates how nodes are connected Switching network can be either connection-based (circuit switched) or connectionless (packet switched) In packet-switched networks congestion control is required Protocols allow different computers on different networks to communicate reliably