An Overview of Myrinet By: Ralph Zajac

What is Myrinet? n LAN designed for clusters n Based on USCD’s ATOMIC LAN n Has many characteristics of MPP message-passing networks

MPP Network Characteristics n High data rate n Regular topology and scalability n Very low error rate n Cut-through routing n Flow control

ATOMIC LAN: Overview n Based on technology from Caltech’s Mosaic project n Used TCP/IP at burst rates of 400 Mb/s n Error rate less than

ATOMIC LAN: Limitations n Asynchronous communication could lead to deadlock n Network topology –Complex –Hidden computing power n Performance limited by lack of DMA in host interface n End-to-end data rates limited by TCP/IP stack of OS

Myrinet n Designed to address the limitations of ATOMIC LAN n Not limited to an existing MPP network n Specifications published in ANSI/VITA

Myrinet: Links n Can be copper (up to 25m) or optical- fiber n Full duplex n Uses non-return-to- zero encoding n Flow control run by a slack buffer STOP GO Slack Buffer

Myrinet: Packets and Routing n Leading byte determines the outgoing port n MSB of each header byte distinguishes between “host” and “switch” packets n Variable payload length n 8-bit CRC

Myrinet: Switches n Use blocking-cut-through routing n Each contains two custom VLSI chips –crossbar-switch –dual interface n Topology is arbitrary

Myrinet: Host Interface n Available for PCI and Sbus n 32-bit SRAM for MCP n Can act as a slave or a bus master

Myrinet: Software n MCP –Handles all interfacing with the network –Continuously mapping and monitoring the network n Host Software –TCP/IP and UDP/IP –Myrinet API

High Performance Virtual Machine n Windows NT 4.0 cluster built at UCSD and UIUC for the NCSA n 96 dual processor nodes n Used a fat-tree network with 8-port Myrinet switches n Was a 56 GFlops system with: –45 GB of DRAM –1.6 MB/s bisection bandwidth –400 GB of disk storage

HPVM: Testing n Tested on four programs –ZeusMP: astrophysical phenomena in 3D –Cactus: modular 3D numerical relativity –AS-PCG kernel: linear systems solver –QMC kernel: Quantum Monte Carlo simulator n Compared with a Cray T3E and an Origin 2000

HPVM: Results n Performed from 2 to 4 times slower –Slower floating point calculations –Flow control limitations of Myrinet n Much better price/performance ratio

Conclusions n Myrinet is: –Flexible –Fast n Affordable systems with acceptable performance