1. Clusters of Multiprocessor Systems

2. Multiprocessing ■ Multiprocessor
- Computer system containing more than one
processor
■ Reasons
- Increase the processing power of a system
- Parallel processing

3. Clusters ■ A collection of workstations of PCs that are
interconnected by a high-speed network
■ Work as an integrated collection of resources
■ Have a single system image spanning all its
nodes

4. Cluster Computer Architecture

5. Cluster Models

6. Tightly Coupled Systems

7. Memory-Coupling, Message-Coupling and DSM
■ UMA – Uniform Memory Access
■ NUMA – Non-Uniform Memory Access
■ NORMA – No Remote Memory Access

8. Memory a) All main memory at the global bus (B2)
b) Main memory distributed among the clusters

9. Components of Cluster Computers
■ Multiple High Performance Computers
- PCs
- Workstations
- SMPs (CLUMPS)
- Distributed HPC Systems leading to Metacomputing
■ State of the art Operating Systems
- Linux (Beowulf)
- Microsoft NT (Illinois HPVM)
- SUN Solaris (Berkeley NOW)
- IBM AIX (IBM SP2)
- HP UX (Illinois-PANDA)
- Mach (Microkernel based OS) (CMU)
- OS gluing layers (Berkeley Glunix)
- Cluster Operating Systems (Solaris MC, Compaq
TruClusters, MOSIX (academic project)

10. Components of Cluster Computers
■ High Performance Networks/Switches
- Ethernet (10Mbps)
- Fast Ethernet (100Mbps)
- Gigabit Ethernet (1Gbps)
- SCI (Dolphin -MPI-12micro-sec latency)
- ATM
- Myrinet(1.2Gbps)
- Compaq Memory Channel (800 Mbps)
- Quadrics QsNET(340 MBps)
■ Fast Communication Protocols and Services
- Active Messages (Berkeley)
- Fast Messages (Illinois)
- U-net (Cornell)
- XTP (Virginia)

11. Components for Clusters
■ Processors
- Intel x86 Processors (Pentium Pro and
Pentium Xeon, AMD x86, Cyrix x86, etc)
- Compaq Alpha (Alpha processor
integrates processing, memory controller,
network interface into a single chip)
- IBM PowerPC
- Sun SPARC
- SGI MIPS
- HP PA
- Berkeley Intelligent RAM (IRAM) integrates
processor and DRAM onto a single chip

12. Components for Clusters
■ Memory and Cache
- Standard Industry Memory Module (SIMM)
- Dual Inline Memory Module (DIMM)
- RDRAM (Rambus), SDRAM, SLDRAM
(SyncLink)
- Access to DRAM is extremely slow compared
to the speed of the processor (SRAM used for
Cache is fast, but expensive & cache control
circuitry becomes more complex as the size
of the cache grows)
- 64-bit wide memory paths
- ECC and RAID protection (high availability)

13. Components for Clusters
■ System Bus
- PCI bus (Up to 64 bit wide and 66 MHz, Up to 512
Mbytes/s transfer rate, Adopted both in Pentium-
based PC and non-Intel platform)
- PCI-X (Up to 1 GB/s transfer rate)
■ Disk and I/O
- Disk access time improves less than 10% per year
- Amdahl’s law (Make the common case faster)
- Performance – Carry out I/O operations parallel,
supported by parallel file system based on
hardware or software striping
- High Availability – Software RAID

14. DASH CEDAR ■ Cluster-based machine developed at Stanford
■ Each cluster is a 4-CPU bus-based Silicon Graphics Power System/340,
with 32 Mbytes of memory visible to all processors in the machine
■ The clusters are interconnected with a 2-D mesh
■ Each processor has three directmapped caches
■ The CPUs are 33-Mhz MIPS R3000s
■ Cache coherence is supported in hardware: a snoopy-based protocol
within each cluster and a directory-based one across clusters
CEDAR
■ A 4-cluster vector multiprocessor developed at the Univ. of Illinois'
Center for Supercomputing Research and Development
■ Each cluster is an 8-CPU bus-based Alliant FX/8
■ All processors in a cluster share a 512 Kbyte direct-mapped cache and
64 Mbytes of memory visible only to the cluster
■ Fast synchronization is possible via a percluster synchronization bus
■ Each processor has a 4 Kbyte prefetch buffer
■ All processors in the machine are connected to 64 Mbytes of shared
memory via forward and return omega networks and no caches

15. HP/Convex Exemplar HP 9000 V2200 Enterprise Server
■ The Exemplar X-Class is the second generation SPP from HP/Convex
■ A ccNUMA architecture comprised of multiple nodes
■ Each node may contain up to 16 PA-8000 processors, 16 Gbytes of
memory and 8 PCI busses
■ Memory access is UMA within each node and is accomplished via a
nonblocking crossbar
■ Each node can be correctly considered as a symmetric multiprocessor
■ The interconnect between nodes is a derivative of the IEEE standard,
SCI, which permits up to 32 nodes to be connected in a 2 dimensional
topology
■ The system includes features to aid high performance
engineering/scientific computations
HP 9000 V2200 Enterprise Server
■ Includes one CPU, 256 megabytes of memory and an unlimited HP-UX
license
■ The machine features up to a 16-way SMP with up to 32-way SMP
■ Supports nonuniform memory access (NUMA) in the hardware level,
while HP-UX will support NUMA in the operating system, so it can
be added to other HP 9000 machines when it is shipped

16. Motorola/IBM PowerPC 604/604e
■ Four-issue superscalar RISC processors from IBM Microelectronics and
Motorola, implementations of the PowerPC architecture specification
■ The processors are targeted at general-purpose desktop computing and
have found design wins in the Apple Macintosh line of personal
computers and in Macintosh clones
■ The fastest version of the PowerPC 604 operates at a clock speed of 180
MHz with a 3.3-volt supply
■ The PowerPC 604e is an enhanced version of the PowerPC 604 and can
operate at a clock speed of 225 MHz with a 2.5-volt supply for the
processor core and a 3.3-volt supply for I/O
■ The PowerPC 604 uses a superscalar RISC architecture and can
dispatch and complete up to four instructions in a single clock cycle
■ The processor operates on 32-bit instructions and integer data and on
64-bit double-precision or 32-bit single-precision floating-point data.
■ The PowerPC 604 has independent floating-point and integer data paths
■ The PowerPC 604 supports virtual memory via separate instruction and
data TLBs for fast address translations
■ The PowerPC 604 allows both the instruction and data caches to be
locked

17. Motorola/IBM PowerPC 604/604e

18. AlphaServer 8400 ■ Supports up to twelve 21164 microprocessors
and 14 gigabytes of
memory, creating
breakthroughs in very
large database
performance
■ Provides a viable
alternative to
supercomputers and
mainframes, with a
peak throughput of 6.6
GF (gigaflops)

19. Architecture of NOW System

20. Beowulf Clusters ■ Simple and highly configurable ■ Low cost
■ Networked
- Computers connected to one another by a private
Ethernet network
- Connection to an external network is through a single
gateway computer
■ Configuration
- COTS – Commodity-off-the-shelf components such as
inexpensive computers
- Blade components – computers mounted on a
motherboard that are plugged into connectors on a rack
- Either shared-disk or shared-nothing model

21. Blade and Rack of Beowulf Cluster

22. Cluster Model
■ Each node is a SMP containing 4 Pentium Pro Processors ■ 8 nodes connected Communication – Ethernet using MPICH – Myrinet using NICAM (Network Interface Communication using Active Messages) ■ Solaris OS ■ Single user, single job

23. Programming SMP Clusters Shared Memory Architecture
■ Thread synchronization and mutual exclusion
is needed
■ Communication overhead is low
■ Performance is limited by system bus
bandwidth
■ Possibility for full cache utilization
■ Distributed memory architecture
■ Implicit synchronization when exchanging
messages
■ Communication overhead is high
■ Performance limited by network bandwidth

24. Quad Pentium shared memory multiprocessor system

25. General model of a shared memory multiprocessor system with caches

26. Scalable Cache Coherent Systems
■ Scalable, distributed memory plus coherent replication
■ Scalable distributed memory machines
- P-C-M nodes connected by network
- Communication assist interprets network transactions,
forms interface
■ Final point was shared physical address space
- Cache miss satisfied transparently from local or remote
memory
■ Natural tendency of cache is to replicate
- No broadcast medium to snoop on
■ Not only hardware latency, but also protocol must scale

27. Two-level Hierarchies

28. Bus Hierarchies with Distributed Memory
■ Main memory distributed among clusters
- Cluster is a full-fledged bus-based machine, memory and all
- Automatic scaling of memory (each cluster brings some with
it)
- Good placement can reduce global bus traffic and latency,
but latency to far-away memory may be larger than to root

29. Benchmarking HPC (High Performance Computing) Clusters using AMD Opteron Processors
■ The AMD64 architecture - and specifically the AMD Opteron processor - is a great platform for HPC needs ■ Good where are large datasets, extensive memory requirements, and a lot of integer or floating-point arithmetic ■ Opteron processor offers low latency, scalable memory bandwidth with on-chip memory controller ■ Minimizing memory latency in multi-processor systems

30. High Performance Computing
■ Minimize turnaround time to complete specific application problem ■ Maximize the problem size that can be solved in a given amount of time

31. The End