TCCluster: A Cluster Architecture Utilizing the Processor Host Interface as a Network Interconnect Heiner Litz University of Heidelberg

2 Motivation  Future Trends  More cores, 2-fold increase per year [Asanovic 2006]  More nodes, nodes for Exascale [Exascale Rep.]  Consequence  Exploit fine grain parallelisim  Improve serialization/synchronization  Requirement  Low latency communication

Motivation  Latency lags Bandwidth [Patterson, 2004]  Memory vs. Network  Memory BW 10GB/s  Network BW 5 GB/s  Memory Latency 50ns  Network Latency 1us  2x vs. 20x 3

State of the Art 4 Scalability Lower Latency Infiniband Ethernet Quickpath SW DSM HyperTransport Larrabee Tilera Clusters SMPs TCCluster

5 Observation  Today’s CPUs represent complete Cluster nodes  Processor cores  Switch  Links

6 Approach  Use host interface as interconnect  Tightly Coupled Cluster (TCCluster)

Background  Coherent HyperTransport  Shared memory SMPs  Cache coherency overhead  Max. 8 endpoints  Table based routing (nodeID)  Non-coherent HyperTransport  Subset of cHT  I/O devices, Southbridge,..  PCI like protocol  “Unlimited” number of devices  Interval routing (memory address) 7

8 Approach  Processors pretend to be I/O devices  Partitioned global address space  Communicate via PIO writes to MMIO

Routing  9

Programming Model  Remote Store PM  Each process has local private memory  Each process supports remotely writable regions  Sending by storing to remote locations  Receiving by reading from local memory  Synchronization through serializing instructions  No support of bulk transfers (DMA)  No support for remote reads  Emphasis on locality, low latency reads 10

11 Implementation  2x Two-socket Quadcore Shanghai Tyan Box SB HTX node1 node0 HTX SB node0 node1 ncHT link BOX 0 BOX 1 Reset/PWR

12 Implementation

13 Implementation  Software based approach  Firmware  Coreboot (LinuxBIOS)  Link de-enumeration  Force non-coherent  Link frequency & electrical parameters  Driver  Linux based  Topology & Routing  Manages remotely writable regions

14 Memory Layout 0 GB 4 GB 5 GB 6 GB Local DRAM node 0 WB Node1 WB MMIO WC RW mem UC 0 GB 4 GB 5 GB 6 GB Local DRAM node 0 WB Node1 WB RW mem UC MMIO WC DRAM Hole BOX 0 BOX 1

15 Bandwidth – HT800(16bit) Singlethread message-rate: 142 mio

16 Latency – HT800(16bit) 227 ns Software-2-Software Half-Roundtrip

17 Conclusion  Introduced novel tightly coupled interconnect  “Virtually” moved the NIC into the CPU  Order of magnitude latency improvement  Scalable  Next steps:  MPI over RSM support  Own mainboard with multiple links

References  [Asanovic, 2006] Asanovic K, Bodik R, Catanzaro B, Gebis J. The landscape of parallel computing research: A view from berkeley. UC Berkeley Tech Report  [Exascale Rep ] ExaScale Computing Study: Technology Challenges in Achieving Exascale Systems  [Patterson, 2004] Latency lags Bandwidth. Communications of the ACM, vol. 47, number 10, pp , October

UoH confidential and proprietary19 Routing  Traditional system (all nodes have same view of memory) DRAM x00 - x0Fx10 - x1Fx20 - x2F DRAM x30 - x3F IO x50 - x5F IO cHT ncHT

UoH confidential and proprietary20 Routing  Our approach (each CPU has its own view with one coherent node0 and 4 IO links) DRAM x00 - x0Fx10 - x1Fx20 - x2F DRAM x30 - x3F ncHT DRAM

UoH confidential and proprietary21 Routing in the Opteron Fabric  Type of a HT packet (posted, non posted, cHT, ncHT) is determined by SRQ based on:  MTRR  GART  Top of memory register  IO and DRAM range registers  Routing is determined by the NB on:  Routing table registers  MMIO base/limit registers  Coherent link traffic distribution register

UoH confidential and proprietary22 Transaction Example 1. Core 0 performs write to IO address. Forwarded to X-Bar via SRQ 2. X-Bar forwards it to IO bridge to convert into posted Write 3. X-Bar forwards it to IO link 4. X-Bar forwards it to IO bridge to convert into coherent sizedWr 5. X-Bar forwards it to Mem Ctrler

UoH confidential and proprietary23 Topology and Adressing > Top > Left > Right > Down > Top > Left 36 -> Right null -> Down Limited possibilities as Opteron only supports 8 address range registers

UoH confidential and proprietary24 Limitations  Communication is PIO, no DMA, no offloading  No congestion management, no HW barriers, no multicast, limited QoS etc  Synchronous system, all Opterons require same clock, no COTS boxes  Security Issues: Nodes can write directly to phys mem on any node  Posted writes to remote memory do not have the coherency bit set, no local caching possible?

How does it work?  Minimalistic Linux Kernel (MinLin)  100 MB, runs in Ramdisk  Boots over ethernet or FILO  Mount homes over ssh  PCI subsystem, to access NB config  Multicore/processor supported  No harddisk, VGA, keyboard,..  No Module support, no device drivers  No Swapping/paging UoH confidential and proprietary25