NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 1 Cluster Archtectures and the NPACI Berkeley NOW David E. Culler Computer Science Division U.C. Berkeley

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 2 Architectural Drivers Node architecture dominates performance –processor, cache, bus, and memory –design and engineering $ => performance Greatest demand for performance is on large systems –must track the leading edge of technology without lag MPP network technology => mainstream –system area networks System on every node is a powerful enabler –very high speed I/O, virtual memory, schedulings, … Incremental scalability (up, down, and across) Complete software tools Wide class of applications

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 3 Berkeley NOW 100 Sun UltraSparcs –200 disks Myrinet SAN –160 MB/s Fast comm. –AM, MPI,... Ether/ATM switched external net Global OS Self Config

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 4 Basic Components $ P M I/O bus MyriNet P Sun Ultra 170 Myricom NIC 160 MB/s M

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 5 Massive Cheap Storage Cluster Basic unit: 2 PCs double-ending four SCSI chains of 8 disks each Currently serving Fine Art at

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 6 Cluster of SMPs (CLUMPS) Four Sun E5000s –8 processors –4 Myricom NICs each Multiprocessor, Multi- NIC, Multi-Protocol NPACI => Sun 450s

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 7 Millennium PC Clumps Inexpensive, easy to manage Cluster Replicated in many departments Prototype for very large PC cluster

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 8 So What’s So Different? Commodity parts? Communications Packaging? Incremental Scalability? Independent Failure? Intelligent Network Interfaces? Complete System on every node –virtual memory –scheduler –files –...

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 9 Communication Performance  Direct Network Access LogP: Latency, Overhead, and Bandwidth Active Messages: lean layer supporting programming models Latency1/BW

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 10 MPI Performance

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 11 NAS Parallel Benchmarks

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 12 World-Record Disk-to-Disk Sort Sustain 500 MB/s disk bandwidth and 1,000 MB/s network bandwidth

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 13 General purpose Parallel System Many timeshared processes –each with direct, protected access –partition it any way you like User and system Client/Server, Parallel clients, parallel servers –they grow, shrink, handle node failures Multiple packages in a process –each may have own internal communication layer Use communication as easily as memory

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 14 Virtual Networks Endpoint abstracts the notion of “attached to the network” Virtual network is a collection of endpoints that can name each other. Many processes on a node can each have many endpoints, each with own protection domain.

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 15 Process 3 How are they managed? How do you get direct hardware access for performance with a large space of logical resources? Just like virtual memory –active portion of large logical space is bound to physical resources Process n Process 2 Process 1 *** Host Memory Processor NIC Mem Network Interface P

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 16 Network Interface Support NIC has endpoint frames Services active endpoints Signals misses to driver –using a system endpont Frame 0 Frame 7 Transmit Receive EndPoint Miss

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 17 Communication under Load Client Server Msg burst work

NPACI: National Partnership for Advanced Computational Infrastructure August 17-21, 1998 NPACI Parallel Computing Institute 18 Beyond the Personal Supercomputer Able to timeshare parallel programs –with fast, protected communication Mix with sequential and interactive jobs Use fast communication in OS subsystems –parallel file system, network virtual memory, … Nodes have powerful, local OS scheduler Simple implicit scheduling techniques provide coordinated scheduling => ride workstation/PC nodes and internet server systems technology => focus CS partners on RAS for long running apps