1. NUMA scaling issues
in 10GbE
NetConf 2009
PJ Waskiewicz
Intel Corp.
LAN Access Division

2. NUMA balancing on 10GbE No affinity to a socket when a driver loads
Insmod runs as a single thread, indeterminate of where static structures will be allocated from
No linkage (currently) to aligning where driver buffers are allocated to where userspace apps run. Is this important?

3. Current issues observed in 10GbE
Scaling multiple ports of 10GbE can cause NUMA memory bandwidth bottlenecks
What happens in systems with a PCIe slot affinitized to a socket? How does the driver know, and allocate accordingly?
Kernel currently references everything per-core (per_cpu lists, etc.). Moving to billions of cores, referencing things to node starts to make more sense

4. Tiny snapshot of balancing issue

5. Thoughts on which direction to go
This problem isn't solved yet, and affects almost everyone. Becomes even worse approaching 40GbE 100GbE
Have an API of sorts to “properly” allocate memory for drivers in a NUMA environment
Move towards using a single queue (or small set) per NUMA node, instead of queue per CPU core. Inter- socket performance is much better than inter-node
Is there any benefit in trying to drive NUMA affinitization into userspace (possibly through the recent Flow Steering from Tom Herbert?)