1. High Performance Messaging on Workstations
Illinois Fast Messages (FM) for Myrinet
Presented By: Scott Thomas
11/12/2018
Cal Poly Network Performance Research Group

2. Cal Poly Network Performance Research Group
Introduction
Workstations clusters
Advantages
Lower Cost
Larger Software Base
Greater Accessibility
Disadvantages
Protocol Overhead
Buffer Management
Operating System Overhead
11/12/2018
Cal Poly Network Performance Research Group

3. Cal Poly Network Performance Research Group
Background
Previous methods:
Mapping network device interface into user space directly via load/store operations
Myrinet Network Features
Uses LANai network coprocessor (5 MIPS)
3 DMA engines (in, out, host)
Workstation Features
Memory bus is fast, I/O Bus is slow!
11/12/2018
Cal Poly Network Performance Research Group

4. Cal Poly Network Performance Research Group
11/12/2018
Cal Poly Network Performance Research Group

5. Cal Poly Network Performance Research Group
Critical Issues
Critical Issues
Division of labor between the host and network coprocessor
Assign as much functionality to the host as possible
Management of the input/output bus
Use DMA as much as possible
Buffer management
Minimal amount of queues
11/12/2018
Cal Poly Network Performance Research Group

6. The Fast Messages Approach
Illinois Fast Messages 1.0 methods
FM_send_4() - Send a four word message
FM_send() - Send a longer message
FM_extract() - Process a received message
11/12/2018
Cal Poly Network Performance Research Group

7. Cal Poly Network Performance Research Group
FM Implementation
FM 1.0
Host Program
LANai Control Program (LCP)
Message Transmission
1. Get data to the LANai
2. Put the data on the line
3. Remove data from the line at receiver
4. Transport data to host memory
11/12/2018
Cal Poly Network Performance Research Group

8. Network Coprocessor Program
Baseline vs. Streamed
11/12/2018
Cal Poly Network Performance Research Group

9. Cal Poly Network Performance Research Group
SBus Management
Methods of Interaction
all-DMA
outgoing: host copies data into DMA, give LANai pointers, triggers send
incoming: uses pointer to write data into host memory
hybrid
outgoing: host copies data into LANai’s memory, triggers send
incoming: DMAs messages into host memory
11/12/2018
Cal Poly Network Performance Research Group

10. Cal Poly Network Performance Research Group
SBus Management
11/12/2018
Cal Poly Network Performance Research Group

11. Buffer Management / Flow Control
4 queues
LANai send
LANai receive
host receive
host reject
Return-to-sender flow control
Receiver returns packet to sender if no memory available
11/12/2018
Cal Poly Network Performance Research Group

12. Comparative Performance
Adding small features can cause huge performance hits.
11/12/2018
Cal Poly Network Performance Research Group

13. Comparative Performance
11/12/2018
Cal Poly Network Performance Research Group

14. Cal Poly Network Performance Research Group
Conclusion
Improvements
improved I/O bus performance for non-DMA operations
faster network interface processor
11/12/2018
Cal Poly Network Performance Research Group

15. Cal Poly Network Performance Research Group
11/12/2018
Cal Poly Network Performance Research Group