WildFire: A Scalable Path for SMPs Erick Hagersten and Michael Koster Sun Microsystems Inc. Presented by Terry Arnold II

Introduction What was the goal? What was the goal? How did they achieve it? How did they achieve it? CMR CMR HAS HAS Competitive Comparisons Competitive Comparisons Results Results Questions Questions

The Goal In the past people have been skeptical about the ability of SMPs to continue to scale due to their bandwidth limitations In the past people have been skeptical about the ability of SMPs to continue to scale due to their bandwidth limitations The trend has been to switch to cc-NUMA The trend has been to switch to cc-NUMA To improve the scalability of SMP technologies To improve the scalability of SMP technologies

Cc-NUMA issues Great scalability but have less than optimal “access patterns” Great scalability but have less than optimal “access patterns” Require high software optimization for capacity and conflict misses Require high software optimization for capacity and conflict misses Non trivial scheduling, etc. (resource and memory management) Non trivial scheduling, etc. (resource and memory management)

How? The answer is the same as the answer to all engineering problems, that is, throwing new acronyms at the problem The answer is the same as the answer to all engineering problems, that is, throwing new acronyms at the problem Coherent Memory Replication (CMR) Coherent Memory Replication (CMR) Hierarchical Affinity Scheduling (HAS) Hierarchical Affinity Scheduling (HAS) Both of these exploit locality as a means of increasing performance (that is for OLTP workloads) Both of these exploit locality as a means of increasing performance (that is for OLTP workloads)

The Overview

The Acronyms: CMR S-COMA with fixed home locations for each address S-COMA with fixed home locations for each address Shadow physical pages Shadow physical pages Coherence at hardware level (64 byte) Coherence at hardware level (64 byte) Things start out cc-NUMA and changed into CMR based on hardware counters that monitor memory access patterns Things start out cc-NUMA and changed into CMR based on hardware counters that monitor memory access patterns Limitations – memory-resident pages and large physical pages can only be replicated explicitly Limitations – memory-resident pages and large physical pages can only be replicated explicitly

The Acronyms: HAS Schedules in the following way: Schedules in the following way: Last processor it ran on Last processor it ran on Same node processor Same node processor Remote node processor (when load balances exceeds “threshold”) Remote node processor (when load balances exceeds “threshold”)

Implementation 2 ASICs – NIAC (coherence), NIDC (bit sliced interconnect) 2 ASICs – NIAC (coherence), NIDC (bit sliced interconnect) These improve upon latency of a switch These improve upon latency of a switch NIAC – Interface and Global-Coherence Layer NIAC – Interface and Global-Coherence Layer Translators and Counters Translators and Counters

Competition The SGI Origin and Sequent’s NUMA-Q The SGI Origin and Sequent’s NUMA-Q

Results 1

Results 2

Questions? Is this “solution” too dependent on the software (kernel modifications)? How compatible are CMR and HAS with the other DSM solutions?