1. Protocols and software for exploiting Myrinet clusters Congduc Pham and the main contributors P. Geoffray, L. Prylli, B. Tourancheau, R. Westrelin

2. Parallel machines and clusters Cplant Standalone workstation

3. Pros for clusters n Large supercomputers are expensive and suffer from a short useful life span n Performance of workstations and PCs is rapidly improving n The communications bandwidth between workstations is increasing as new networking technologies and protocols are implemented in LANs and WANs. n Workstation clusters are easier to integrate into existing networks than special parallel computers. n Use of clusters of workstations as a distributed computing resource is very cost effective - incremental growth or update of system!!!

4. No polemical discussion, just statement… Mainframe Vector Supercomputer Mini Computer Workstation PC 1984 from R. Buyya GigaEthernet Giganet SCI Myrinet …

5. The Myrinet technology n Switch –full crossbar –wormhole source routing –small latency n Network interface –embedded RISC processor –programmable –local memory –several DMA engines Current specifications: Up to 200Mhz processor Up to 8MB local memory 64bit/66Mhz PCI bus (528 MB/s peak) 250 MB/s full duplex links

6. The raw performance is here, but… n the traditional communication software fail to bring the hardware performance to the applications Myrinet Traditional communication layers Optimized communication layers 200mph40mph 180mph 35mph 175mph

7. Going faster by taking shortcuts

8. Our communication architecture n Provides a complete suite for high-performance communications.Focus on Myrinet-based clusters n Viewed as layers, but by-passes as much as possible the OS Myrinet physical layer BIP BIP-SMP MPI-BIP programmable NICs break the traditional spatial distribution of tasks

9. BIP, the lowest protocol level n Basic Interface for Parallelism –very basic API –provides a library, a kernel module and a MCP –definitely not for the end-user n Optimizations for –latency –maximum throughput –the throughput increase n The implementation performs –reduction of the data critical path –distinction between small and large messages –burst or write combining for host  NIC –optimal cache usage –cache snooping for NIC  host (monitoring of the PCI bus) –buffer alignment –optimal fragment size… Myrinet BIP BIP-SMP MPI-BIP

10. n Avoids handshakes between the host and the NIC n Uses PIO to a NIC FIFO on the sending side and an extra memory copy on the receiving side BIP, small message strategy

11. n Use DMA both on the send side and receive side: higher bandwidth, offload the CPU n Zero-copy mechanism, pipelined transmission BIP, large message strategy

12. BIP-SMP: a low level for SMP machines n SMP viewed as best performance/price ratio architectures (2 or 4 proc.) n BIP-SMP provides –manage concurrent accesses to the NIC –low latency intra-node communications –BIP equivalent inter-node communication –total transparency for the applications and end-users 0 1 2 3

13. BIP-SMP: Moving data between processes

14. MPI-BIP: the communication middleware n MPI-BIP adds high-level features to BIP –based on the MPICH implementation –provides a portable and widely-used API –implements a credit-based flow control for small messages –request FIFO for multiple non-blocking operations –provides segmentation/reassembly features to avoid timeouts

15. Working with the BIP software suite n installation –run configure n compilation and linkage –several libraries: bip, bip-smp, mpi –compile with bipcc n Submitting jobs and monitoring nodes –run myristat to know which nodes are available –run bipconf to configure the virtual machine –use bipload to lunch programs

16. WebCM: a high level management tool n web-based management tool n integrates existing solutions into a common framework

17. The WebCM user interface graphical interface for myristat and bipconf n allows submission of jobs through batch packages n shows the user's virtual machine definition and the user's runnning processes n addition of fonctionnalities is performed by incorporating new software packages

18. Latency: BIP and MPI-BIP

19. Throughput: BIP and MPI-BIP

20. BIP-SMP: intra-node communications

21. BIP-SMP: inter-node communications

22. What run on our clusters? n Genomic simulation n Fluid dynamic n Discrete Event Parallel Simulation n Distributed Shared Memory System n Want to know more? –getting the distribution –getting the documentation http://resam.univ-lyon1.fr