1. 1Thu D. NguyenCS 545: Distributed Systems CS 545: Distributed Systems Spring 2002 Communication Medium Thu D. Nguyen http://paul.rutgers.edu/cs545/S02/

2. 2Thu D. NguyenCS 545: Distributed Systems Administrative Details Remember no class next week We will be in CoRE A for the remainder of the term I will always bring my laptop with your presentations loaded

3. 3Thu D. NguyenCS 545: Distributed Systems Introduction Today, we are covering 3 papers covering 3 different points in the evolution of local area networking Ethernet: one of the first and most successful LAN VIA: standard defined in the last 5 years for low latency high throughput networking performance for clustering InfiniBand: the future? I/O interconnect and SAN

4. 4Thu D. NguyenCS 545: Distributed Systems Ethernet So, what was so great about Ethernet? Clearly, Ethernet has been wildly successful, being the standard LAN technology for the last 20 years 3 Mb/s -> 10 Mb/s -> 100 Mb/s -> 1 Gb/s -> 10 Gb/s? Embodies many of the “principles” that lead to early success in designing/implementation of distributed systems

5. 5Thu D. NguyenCS 545: Distributed Systems Why VIA? Originally, communication protocols such as TCP/IP and UDP/IP were designed for WAN-connected systems Faults common because of line noise and router congestion When parallel computing and clustering first came about, these communication protocols were used by default These new systems have radically different interconnects, however Particular, faults were much less common Also, the applications were much more “tightly” coupled than those running over the Internet The software overhead of standard protocols became an unacceptable performance overhead

6. 6Thu D. NguyenCS 545: Distributed Systems Why VIA? People came to believe in two things Software overhead of “thick” protocols is too costly for parallel and clustering applications Overhead of trapping into the kernel for every message send/receive was also unacceptably high Particularly because trapping into the kernel often requires copying from the user’s address space into kernel buffers The system bus became the performance bottleneck I assume you are familiar with this from readings in 519 VIA Protected direct user-level access to network 0-copy and avoids trap into kernel

7. 7Thu D. NguyenCS 545: Distributed Systems Why InfiniBand? Gee, VIA just arrived. Why the hurry to InfiniBand? Now that the network is providing low latency and high throughput, the I/O interconnect is becoming the performance bottleneck

8. 8Thu D. NguyenCS 545: Distributed Systems Specialized Messaging Performance 1992199419972000

9. 9Thu D. NguyenCS 545: Distributed Systems The Intuition Specialized Messaging focuses on reducing instructions the ratio of processor speed to I/O bus speed grows with time  gain from specialized messaging should also diminish with time Application of Amdahl’s law For example, we found that relative gain dropped from ~10 to ~2 for small messages

10. 10Thu D. NguyenCS 545: Distributed Systems Short Message Methodology Measured real system Used Architectural parameters Built Abstract model based on parameters Predicted future trend of parameters Used model and trend to derive future performance improvements

11. 11Thu D. NguyenCS 545: Distributed Systems Experimental Setup Giganet VIA AM-II Pairs of computers were used: Intel 233MHz, 300MHz, 400MHz, 550MHz OS:Linux 2.2.14 Ethernet UDP AM-II

12. 12Thu D. NguyenCS 545: Distributed Systems The LogGP Model Latencyoverhead -Sending- overhead -Receiving- gap Processor Source Processor Sink

13. 13Thu D. NguyenCS 545: Distributed Systems Comparison ratio=3.3 3.4 2.75 2.40

14. 14Thu D. NguyenCS 545: Distributed Systems Architectural Model Characterize sending o s and o r in terms of architectural parameters Measured Protocol Instructions User/Kernel Mode I/O instructions Memory references Cache hits and misses CPI Interrupts

15. 15Thu D. NguyenCS 545: Distributed Systems Predictions 40% compound growth for processor speed over the next 5 years. I/O Bus Speed from 33MHz to 66MHz Number of PCI instructions may drop 133MHz not likely due to cost

16. 16Thu D. NguyenCS 545: Distributed Systems Comparison

17. 17Thu D. NguyenCS 545: Distributed Systems Projected Values 2.1 ratio=2.23 2.25

18. 18Thu D. NguyenCS 545: Distributed Systems AM-UDP

19. 19Thu D. NguyenCS 545: Distributed Systems AM-VIA

20. 20Thu D. NguyenCS 545: Distributed Systems Short Message Summary Predicted relative gain of specialized messaging layers as compared to general messaging will drop to a factor of ~2 by 2005. Processor cycles becoming cheaper compared to I/O cycles. Possibly add overhead to add connectivity/robustness Need radical restructuring of I/O interconnect

21. 21Thu D. NguyenCS 545: Distributed Systems InfiniBand InfiniBand is a radical restructuring of the I/O interconnect But … InfiniBand really concentrates on bandwidth rather than latency! Possible implications Robustness and connectivity given by standard protocols are needed Death of VIA? CPU is so fast that the overhead of processing communication protocols no longer matters Machines are only doing I/O anyways so if the CPU is 100% busy with this task, it’s ok

22. 22Thu D. NguyenCS 545: Distributed Systems InfiniBand Everything and the kitchen sink Does it have a hope of working?