1. Building a High-performance Computing Cluster Using FreeBSD BSDCon '03 September 10, 2003 Brooks Davis, Michael AuYeung, Gary Green, Craig Lee The Aerospace Corporation El Segundo, CA {brooks,lee,mauyeung}@aero.org, Gary.B.Green@aero.org

2. HPC Clustering Basics ● HPC Cluster features: – Commodity computers – Networked to enable distributed, parallel computations – Vastly lower cost compared to traditional supercomputers ● Many, but not all HPC applications work well on clusters

3. Cluster Overview ● Fellowship is the Aerospace Corporate Cluster – Name is short for "The Fellowship of the Ring" ● Running FreeBSD 4.8-STABLE ● Over 183GFlops of floating point performance using the LINPACK benchmark

4. Cluster Overview Nodes and Servers ● 160 Nodes (320 CPUs) – dual CPU 1U systems with Gigabit Ethernet – 86 Pentium III (7 1GHz, 40 1.26GHz, 39 1.4GHz – 74 Xeon 2.4GHz ● 4 Core Systems – frodo – management server – fellowship – shell server – gamgee – backup, database, monitoring server – legolas – scratch server (2.8TB)

5. Cluster Overview Network and Remote Access ● Gigabit Ethernet network – Cisco Catalyst 6513 switch – Populated with 11 16-port 10/100/1000T blades ● Serial console access – Cyclades TS2000 and TS3000 Terminal Servers ● Power control – Baytech RPC4 and RPC14 serial power controllers

6. Cluster Overview Physical Layout

7. Design Issues ● Operating System ● Hardware Architecture ● Network Interconnects ● Addressing and Naming ● Node Configuration Management ● Job Scheduling ● System Monitoring

8. Operating System ● Almost anything can work ● Considerations: – Local experience – Needed applications – Maintenance model – Need to modify OS ● FreeBSD – Diskless support – Cluster architect is a committer – Ease of upgrades – Linux Emulation

9. Hardware Architecture ● Many choices: – i386, SPARC, Alpha ● Considerations: – Price – Performance – Power/heat – Software support (OS, apps, dev tools) ● Intel PIII/Xeon – Price – OS Support – Power

10. Network Interconnects ● Many choices – 10/100 Ethernet – Gigabit Ethernet – Myrinet ● Issues – price – OS support – application mix ● Gigabit Ethernet – application mix ● middle ground between tightly and loosely coupled applications – price

11. Addressing and Naming Schemes ● To subnet or not? ● Public or private IPs? ● Naming conventions – The usual rules apply to core servers – Large cluster probably want more mechanical names for nodes ● 10.5/16 private subnet ● Core servers named after Lord of the Rings characters ● Nodes named and numbed by location – rack 1, node 1: ● r01n01 ● 10.5.1.1

12. Node Configuration Management ● Major methods: – individual installs – automated installs – network booting ● Automation is critical ● Network booted nodes – PXE ● Automatic node disk configuration – version in MBR – diskprep script ● Upgrade using copy of root

13. Job Scheduling ● Options – manual scheduling – batch queuing systems (SGE, OpenPBS, etc.) – custom schedulers ● Sun Grid Engine – Ported to FreeBSD starting with Ron Chen's patches

14. System Monitoring ● Standard monitoring tools: – Nagios (aka Net Saint) – Big Sister ● Cluster specific tools: – Ganglia – Most schedulers ● Ganglia – port: sysutils/ganglia- monitor-core ● Sun Grid Engine

15. System Monitoring Ganglia

16. Lessons Learned ● Hardware attrition can be significant ● Neatness counts in cabling ● System automation is very important – If you do it to a node, automate it ● Much of the HPC community thinks the world is a Linux box

17. FY 2004 Plans ● Switch upgrades: Sup 720 and 48-port blades ● New racks: another row of racks adding 6 more node racks (192 nodes) ● More nodes: either more Xeons or Opterons ● Upgrade to FreeBSD 5.x

18. Future Directions ● Determining a node replacement policy ● Clustering on demand ● Schedular improvements ● Grid integration (Globus Toolkit) ● Trusted clusters

19. Wish List ● Userland: – Database driven, PXE/DHCP server ● Kernel: – Distributed files system support (i.e. GFS) – Checkpoint and restart capability – BProc style distributed process management

20. Acknowledgements  Aerospace – Michael AuYeung – Brooks Davis – Alan Foonberg – Gary Green – Craig Lee  Vendors – iXsystems – Off My Server – Iron Systems – Raj Chahal ● iXsystems, Iron Systems, ASA Computers

21. Resources ● Paper and presentation: – http://people.freebsd.org/~brooks/papers/bsdcon2003/