1. ENGS 116 Lecture 191 Interconnection Networks Vincent H. Berk November 14, 2005 Reading for Friday: 8.1-8.7 Reading for Monday: 8.8-8.13

2. ENGS 116 Lecture 192 Project Reports Due by Beginning of class next Monday, November 24 Content –Introduction and description of the topic –Coverage of topic: breadth/depth, appropriate background information –Analysis and discussion –References: correct citations, proper form Writing –Spelling –Grammar –Style and presentation Assume that the reader is familiar with basic architecture concepts Must use appropriate citations. Argument all your decisions. Email all source code to

3. ENGS 116 Lecture 193 Project Presentations 15 minutes – absolutely no more – Practice your timing! All group members talk 24 th (4) of November, 1 st (4) of December Present: –research question –approach –results –conclusions EVERYONE ATTENDS these presentations

4. ENGS 116 Lecture 194 Networks Common topics of conversation: –direct (point-to-point) vs. indirect (multi-hop) –topology (e.g., bus, ring, directed acyclic graph, star) –routing algorithms –switching (aka multiplexing) –wiring (e.g., choice of media, copper, coax, fiber) What really matters: –latency –bandwidth –cost –reliability

5. ENGS 116 Lecture 195 ABCs of Networks Starting point: Send bits between 2 computers Queue on each end Can send both ways (“Full Duplex”) Rules for communication? “protocol” –Inside a computer: Loads/Stores: Request (Address) & Response (Data) Need request & response signaling –Name for standard group of bits sent: packet

6. ENGS 116 Lecture 196 A Simple Example What is the format of a packet? (Protocol) –Fixed? Number of bytes? Request/ Response Address/Data 1 bit 32 bits 0: Please send data from address 1: Packet contains data corresponding to request

7. ENGS 116 Lecture 197 Questions About Simple Example What if more than 2 computers want to communicate? –Need computer address field (destination) in packet What if packet is garbled in transit? –Add error detection field in packet (e.g., CRC) What if packet is lost? –More elaborate protocols to detect loss (e.g., NAK, ARQ, time outs) What if multiple processes per machine? –Queue per process Questions such as these lead to more complex protocols and packet formats

8. ENGS 116 Lecture 198 A Simple Example Revisited What is the format of a packet? –Fixed? Number of bytes? Request/ Response Address/Data 2 bits32 bits 00: Request—Please send data from address 01: Reply—Packet contains data corresponding to request 10: Acknowledge request 11: Acknowledge reply 4 bits CRC

9. ENGS 116 Lecture 199 Additional Background Connection of 2 or more networks: Internetworking 3 cultures for 3 classes of networks –SAN: server (storage) networks, performance –LAN: workstations, cost –WAN: telecommunications, long range Cost Performance (BW, latency) Reliability

10. ENGS 116 Lecture 1910 Interconnections (Networks) Examples: –SAN networks (infiniband): 100s nodes; ≤ 100 meters per link –Local Area Networks (Ethernet): 100s nodes; ≤ 1000 meters –Wide Area Network (ATM): 1000s nodes; ≤ 5,000,000 meters Interconnect SW Interface HW Interface Node Link SW Interface HW Interface Node Link SW Interface HW Interface Node Link SW Interface HW Interface Node Link...

11. ENGS 116 Lecture 1911 Software to Send and Receive SW Send steps 1: Application copies data to OS buffer 2: OS calculates checksum, starts timer 3: OS sends data to network interface HW and says start SW Receive steps 3: OS copies data from network interface HW to OS buffer 2: OS calculates checksum, if matches send ACK; if not, deletes message (sender resends when timer expires) 1: If OK, OS copies data to user address space and signals application to continue Sequence of steps for SW: protocol –Example similar to UDP/IP protocol in UNIX

12. ENGS 116 Lecture 1912 Network Performance Measures SW Interface HW Interface Node Link Bandwidth Overhead Link SW Interface HW Interface Node Link Bandwidth Overhead Link Bisection Bandwidth Interconnect Latency...

13. ENGS 116 Lecture 1913 Universal Performance Metrics Sender Receiver Sender Overhead Transmission time (size ÷ bandwidth) Transmission time (size ÷ bandwidth) Time of Flight Receiver Overhead Transport Latency Total Latency = Sender Overhead + Time of Flight + Message Size ÷ BW + Receiver Overhead Includes header/trailer in BW calculation? Total Latency (processor busy) (processor busy)

14. ENGS 116 Lecture 1914 Simplified Latency Model Total Latency ≈ Overhead + Message Size / BW Overhead = Sender Overhead + Time of Flight + Receiver Overhead Example: show what happens as we vary the following –Overhead: 1, 25, 500 µsec –BW: 10, 100, 1000 Mbit/sec (factors of 10) –Message Size: 16 Bytes to 4 MB (factors of 4) If overhead is 500 µsec, how big a message is needed to get > 10 Mb/s of bandwidth?

15. ENGS 116 Lecture 1915 0.01 0.1 1 10 100 1000 1664 256 10244096 1638465536 262144 10485764194304 Message Size (bytes) Effective Bandwidth (Mbits/sec) o1, bw10 o1, bw100 o1, bw1000 o25, bw10 o25, bw100 o25, bw1000 o500, bw10 o500, bw100 o500, bw1000 o1, bw1000 o500, bw1000 o25, bw1000 o500, bw100 o500, bw10 bw1000 bw100 bw10 o1, bw100 Overhead, Bandwidth, Size

16. ENGS 116 Lecture 1916 Measurement: Sizes of Message for NFS 95% messages, 30% bytes for packets ≤ 200 bytes > 50% data transferred in packets = 8KB Packet size Cumulative % 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 010242048307240965120614471688192 Msgs Bytes Why?

17. ENGS 116 Lecture 1917 HW Interface Issues Where to connect network to computer? –Cache consistent to avoid flushes? (  memory bus) –Latency and bandwidth? (  memory bus) –Standard interface card? (  I/O bus) –MPP  memory bus; LAN, WAN  I/O bus $ CPU L2 $ Memory Bus MemoryBus Adaptor I/O bus I/O Controller I/O Controller Network ideal: high bandwidth, low latency, standard interface Network

18. ENGS 116 Lecture 1918 SW Interface Issues How to connect network to software? –Programmed I/O? (low latency) –DMA? (best for large messages) –Receiver interrupted or received polls? Things to avoid –Invoking operating system in common case –Operating at uncached memory speed (e.g., check status of network interface)

19. ENGS 116 Lecture 1919 CM-5 Software Interface CM-5 example (MPP) –Time per poll 1.6  secs; time per interrupt 19  secs –Minimum time to handle message: 0.5  secs –Enable/disable 4.9/3.8  secs As rate of messages arriving changes, use polling or interrupt? –Solution: Always enable interrupts, have interrupt routine poll until no messages pending –Low arrival rate  interrupt –High arrival rate  polling 0 10 20 30 40 50 60 70 80 90 100 0102030405060708090100 message interarrival (µsecs) message overhead (µsecs) Polling Interrupts Time between messages Overhead

20. ENGS 116 Lecture 1920 Network Media Copper, 1mm thick, twisted to avoid antenna effect (telephone) Used by cable companies: high BW, good noise immunity 3 parts are cable, light source, light detector. Twisted Pair: Coaxial Cable: Copper core Insulator Braided outer conductor Plastic Covering Fiber Optics Transmitter – L.E.D – Laser Diode Receiver – Photodiode light source Silica Total internal reflection Air

21. ENGS 116 Lecture 1921 Connecting Multiple Computers Shared Media vs. Switched: pairs communicate at same time, “point-to-point” connections Aggregate BW in switched network is many times that of shared –point-to-point faster since no arbitration, simpler interface Arbitration in shared network? –Central arbiter for LAN? –Listen to check if being used (“Carrier Sensing”) –Listen to check if collision (“Collision Detection”) –Random resend to avoid repeated collisions; not fair arbitration; –OK if low utilization (a.k.a. data switching interchanges, multistage interconnection networks, interface message processors) Switch Node Switched Media (CM-5, ATM) Node Shared Media (Ethernet)

22. ENGS 116 Lecture 1922 Switch Topology Structure of the interconnect Determines –Degree: number of links from a node –Diameter: max number of links crossed between nodes –Average distance: number of hops to random destination –Bisection: minimum number of links that separate the network into two halves (worst case) Warning: these three-dimensional drawings must be mapped onto chips and boards which are essentially two-dimensional media –Elegant when sketched on the blackboard may look awkward when constructed from chips, cables, boards, and boxes (largely 2D)

23. ENGS 116 Lecture 1923 Figure 8.15 A ring network topology. A Simple Example

24. ENGS 116 Lecture 1924 Examples of Static Interconnection Network Topologies c) Star a) Bus b) Linear array e) Tree f) Near-neighbor mesh d) Ring g) Completely connected h) 3–cube (hypercube)

25. ENGS 116 Lecture 1925 Figure 8.16: Network topologies that have appeared in commercial MPPs.

26. ENGS 116 Lecture 1926 Important Topologies N = 1024 TypeDegreeDiameterAvg DistBisectionDiamAvg D 1D mesh≤ 2N-1N/31 2D mesh≤ 42(N 1/2 - 1)2N 1/2 / 3N 1/2 6321 3D mesh≤ 63(N 1/3 - 1)3N 1/3 / 3N 2/3 ~ 30~ 10 Ring2N / 2N/42 2D torus4N 1/2 N 1/2 / 22N 1/2 3216 Hypercubenn = LogNn/2N/2 105

27. ENGS 116 Lecture 1927 Figure 8.14 A fat-tree topology for 16 nodes. 0 1 2 3 0 0 0 1 4 5 6 7 0 1 1 1 8 9 10 11 0 2 2 1 12 13 14 15 0 3 3 1 0 0 0 1 0 2 0 3 0

28. ENGS 116 Lecture 1928 Figure 8.13 Popular switch topologies for eight nodes.

29. ENGS 116 Lecture 1929 Examples of dynamic interconnection network topologies = processor= switch a) Crossbar switch b) 8  8 Baseline