1. Experiences Tuning Cluster Hosts
1GigE and 10GbE
Paul Hyder
Cooperative Institute for Research in Environmental Sciences, CU Boulder
(CIRES at NOAA/ERSL/GSD High Performance Computing)
Paul.Hyder at noaa.gov

2. Tuning Focus Cluster Front Ends and Cron Server Hosts
File transfer servers (scponly)
BWCTL host
Remote client hosts
10GbE Testbed
(7.2 Gb/sec uses ~49% of one 3G CPU)

3. How We Apply the Well Known Rules
Jumbo Frames
8K on hosts
9K on network
Tune TCP to match BDP
Encourage application writers to use large read and write buffers
Install tuned Applications
PSC.edu patch to ssh
OpenSSH:channels.h
#define CHAN_TCP_PACKET_DEFAULT (32*1024)
#define CHAN_TCP_WINDOW_DEFAULT (4*CHAN_TCP_PACKET_DEFAULT)

4. Throughput Testing Iperf (2.0.2) from shell scripts
Vary buffer (-l) and window (-w)
Modify ifconfig and PCI configuration
Loop takes 3 days
Bwctl with remote hosts
?Anyone on NLR?
Use scp/sftp/rsync as final test

5. I’m Curious
How much TCP tuning information do you provide users and admins?
Are hosts being tuned?
Does your internal LAN support jumbo frames?

6. GSD Cluster GigE Defaults
[wr]mem_default 2MB
[wr]mem_max 16MB
ipv4/tcp_[wr]mem 64KB 2MB 16MB
optmem_max 512K
txqueuelen 10000
netdev_max_backlog 3000
ipv4/tcp_sack and ipv4/tcp_timestamps on
Don’t touch ipv4/tcp_mem

7. Jumbo Frame Plot

8. What doesn’t work Jumbo Frames Switch Fabrics Firewalls
High density cards
Complex vLAN configurations
Stand alone GigE switches
Firewalls
ICMP for path mtu discovery
Disabled completely
Network devices don’t respond

9. Linux 2.6 and Jumbos
IP hostA > hostB.22: S 544:544(0) win <mss 8152,...>
IP hostB.22 > hostA.52434: S 207:207(0) ack 545 win 5792 <mss 1460,...>
...
IP hostA > hostB.22: :6599(4344) ack 2293 win <...>
IP hostA > hostB.22: P 6599:10943(4344) ack 2293 win <...>
IP router > hostA: icmp 36: hostB unreachable - need to frag (mtu 1500)
IP hostA > hostB.22: :3703(1448) ack 2293 win <...>

10. Host Side Checks Interrupt Aggregation (Linux NAPI)
Memory to match buffer tuning
More than one CPU
Static ARP entries

11. Network Device Settings
Static ARP entries or increase timeout
Increase FDB timeouts
Verify jumbo frame configuration

12. 10GbE Quick Notes
Know your PCI hardware (MMRBC, Latency timer, and Splits)
TCP stack is ~0.200ms
Increase netdev_max_backlog to 30000
(throughput = backlog * 100MHz * ave_bytes_pkt)
Set *_cong to CERN values
Write buffers in code ~128KB

13. 10G buffer plot

14. Questions?

15. Reference URLs http://www.psc.edu/networking/projects/hpn-ssh/