Evolution of High Performance Cluster Architectures David E. Culler NPACI 2001 All Hands Meeting

Much has changed since “NOW” NOW0 HP+medusa FDDI NOW1 SS+ATM/Myrinet NOW 110 UltraSparc +Myrinet inktomi.berkeley.edu

Millennium Cluster Editions

The Basic Argument performance cost of engineering lag –miss the 2x per 18 months –=> rapid assembly of leading edge HW and SW building blocks –=> availability through fault masking, not inherent reliability emergence of the “killer switch” opportunities for innovation –move data between as fast as within machine –protected user-level communication –large-scale management –fault isolation –novel applications

Clusters Took Off scalable internet services –only way to match growth rate changing supercomputer market web hosting

Engineering the Building Block argument came full circle in ~98 wide-array of 3U, 2U, 1U rack-mounted servers –thermals and mechanicals –processing per square-foot –110 AC routing a mixed blessing –component OS & drivers became the early entry to the market

Emergence of the Killer Switch ATM, Fiberchannel, FDDI “died” ServerNet bumps along IBM, SGI do the proprietary thing little Myrinet just keeps going –quite nice at this stage SAN standards shootout –NGIO + FutureIO => Infiniband –specs entire stack from phy to api »nod to IPv6 –big, complex, deeply integrated, DBC Gigabit EtherNet steamroller... –limited by TCP/IP stack, NIC, and cost

Opportunities for Innovation

Unexpected Breakthru: layer-7 switches fell out of modern switch design –process packets in chunks vast # of simultaneous connections many line-speed packet filters per port can be made redundant => multi-gigabit cluster “front end” –virtualize IP address of services –move service within cluster –replicate it, distribute it  high-level xforms  fail-over,  load management Layer-7Switch Network Switch

e-Science any useful app should be a service

Protected User-level messaging  Virtual Interface Architecture (VIA) emerged  primitive & complex relative to academic prototypes  industrial compromise  went dormant  Incorporated in Infiniband  big one to watch  Potential breakthrough  user-level TCP, UDP with IP NIC  storage over IP

Management workstation -> PC transition a step back –boot image distribution, OS distribution –network troubleshoot and service multicast proved a powerful tool emerging health monitoring and control –HW level –service level –OS level still a problem

Rootstock Local Rootstock Server Internet Rootstock Server Local Rootstock Server UC Berkeley

Ganglia and REXEC rexec d vexecd (Policy A) rexec Cluster IP Multicast Channel %rexec –n 2 –r 3 indexer minimum $ vexecd (Policy B) Node ANode BNode CNode D “Nodes AB” Also: bWatch BPROC: Beowulf Distributed Process Space VA Linux Systems: VACM, VA Cluster Manager

Network Storage state-of-practice still NFS + local copies local disk replica management lacking NFS doesn’t scale –major source of naive user frustration limited structured parallel access SAN movement only changing the device interface Need cluster content distribution, caching, parallel access and network striping see: GPFS, CFS, PVFS, HPSS, GFS,PPFS,CXFS, HAMFS,Petal, NASD...

Distributed Persistent Data Structure Alternative Service DDS lib Storage “brick” Service DDS lib Service DDS lib Storage “brick” Storage “brick” Storage “brick” Storage “brick” Storage “brick” System Area Network Clustered Service Distr Hash table API Single-node durable hash table Redundant low latency high xput network

Scalable Throughput

“Performance Available” Storage D A D A D A D A Static Parallel Aggregation D A D A D A D A Distributed Queue Adaptive Parallel Aggregation

Application Software very little movement towards harnessing architectural potential application as service –process stream of requests (not shell or batch) –grow & shrink on demand –replication for availability »data and functionality –tremendous internal bandwidth outer-level optimizations, not algorithmic

Time is NOW  f inish the system area network  tackle the cluster I/O problem  come together around management tools  get serious about application services