System Area Network Abhiram Shandilya 12/06/01

Overview Introduction to System Area Networks SAN Design and Examples SAN Applications

Introduction What is a SAN? Overview of Cluster Computing Advent of Clusters Advantages of Clusters Communication in Clusters SAN design goals

What is a SAN? Primarily designed for communication between nodes in a cluster. Other SANs: Storage, Server and Small A Cluster

Cluster Computing: Advent of…. Early parallel computers: MPPs Custom-built Expensive Intel Paragon, CM-1, nCube etc Changes in technology: Faster processors Faster networking technologies Cheaper Commodity off-the-shelf (COTS) hardware A cheaper option: clusters

Cluster Computing: Advantages Better price/performance ratios Better availability Survive node failures with appropriate software support Impressive scaling 100s to 1000s of processors Easy and economical upgrading and technology migration.

Cluster Computing: Communication Parallel computation is communication intensive Dominant contributors for message latency Large messages: Bandwidth Small messages: Software Overhead Cluster communication: small messages Software overhead on traditional networks due to: Multiple data copies through protocol layers OS interrupts and context switching.

Software overhead due to multiple copies

SAN Design Goals Minimize latency Avoid operating system by providing user application direct access to the network interface Avoid multiple copies of data High Bandwidth

SAN Design and Examples Key Issues General Architecture Example 1: Virtual Interface Architecture (VIA) Example 2: Memory Channel

Key Issues Design of direct user access to network interface Multiplexing between processes for access to network interface Providing security: avoiding inter-process interference Managing limited communication resources Efficient programming interface

General Architecture Communication model Virtual Interface Architecture (VIA) User-Level Network Interface (U-Net) Virtual Memory Mapped Communication (VMMC) Network Interconnect (NI) Memory Channel 1 & 2 (DIGITAL) ServerNet-II (Compaq)

Virtual Interface Architecture (VIA): An Overview  Promoted by Compaq, Intel and Microsoft  Defines an industry-standard architecture for communication within clusters  Increase scalability  Allows a single parallel application to run efficiently over dozens of clustered nodes  Speeds up exchange of data between distributed application modules running on different servers

VIA: Goals Protected communication Latencies like memory operation Eliminate intermediate data copies Avoid traps into the OS Eliminate driver running in protected kernel mode Minimize number of instructions for send/receive Remove the reliance on interrupts for communication Keep architecture simple to allow software and hardware implementations Three types of services provided: Unreliable datagram service Reliable delivery service Reliable reception service

A Virtual Interface (VI) Creates illusion that a process has its own network interface. User Proces s Send Queue Receive Queue CNTL VI Network Interface User Proces s Send Queue Receive Queue CNTL VI Kerne l Agent Control and interrupts Per VI control and synchronization

VIA: Architecture Virtual Interface Represents a connection Linked list of descriptors Send, remote-DMA/write and remote-DMA/read on send queue Receive on receive queue Maintains data ordering for a connection using VI Asynchronous I/O between different VIs Kernel agent Unless required by user, not involved in communication Initializes tables and structures for communication Provides memory protection Pins physical memory for each connection. Necessary to avoid delay on receiver’s side due to address translation

Memory Channel 1: An Overview  Developed at DIGITAL as a dedicated network interconnect  Designed to improve communication performance in a cluster of at most 8 nodes  Can support clusters as large as 8 nodes by 12 processors  Implements Virtual Shared Memory  Supports any computer using industry- standard peripheral-component interconnect (PCI) bus

Memory Channel 1: Goals Minimize cluster communication latencies Maintain the ability to repair hardware while cluster is running Strict message ordering Support full detection and recovery from communication errors Connect to PCI bus for compatibility with all PCI-based Alphas and Intel computers. Support broadcast, multicast and point-to-point communication

Memory Channel 1: Architecture (1) Address Space Mapping Memory-mapped circuit Connection between virtual-address space on sending node and a page of physical memory on receiver’s node. Write-only connection

Memory Channel 1: Architecture (2) Hardware Implementation MC Network I/O Bridge Transmit path Receive Path Processor Memory I/O Device I/O Bridge Transmit path Receive Path Processor Memory I/O Device PCI Bus System Bus System Bus PCI Bus Node 2 Node 1 MC Adaptor

SAN Applications File Server Clusters Database Server Clusters Storage Area Networks (SAN) Web Servers Clusters Scientific Applications Beowulf clusters

Conclusions Introduction to SAN SAN Design and examples VIA Memory Channel 1 SAN Applications