Notes from Installing a Mac G5 Cluster at SLAC Chuck Boeheim SLAC Computing Services

Background New joint Stanford/SLAC department: Kavli Institute for Particle Astrophysics Choices for compute cluster included joining Opteron purchase, or Mac G5 cluster Astronomers like Macs –Compatible with laptops and desktops they were buying

Configuration 2 Mac G5 file servers –2 2.5TB Apple XServ Raid units each 2 Mac G5 interactive servers 10 Mac G5 compute nodes –No graphics cards –No CD drives Plus two development servers for us

Physical Good overall engineering Good airflow; quiet Hot swap disks in server nodes Rails fit standard racks well Easy to mount

Management Supports serial console (57,600 baud) –No graphics card in G5 server Supports serial bios –But must press button to boot in bios No power management Supports network install Initial disk partition and boot a little tricky Subsequent installs can be fully automated

Installing the first server “Secret handshake” –Install from CD Run Server Assistant on another Mac to start install, and again to configure Run Apple Remote Desktop to get graphic login Command line alternatives are available, though obscure

The Partitioning Conundrum System has initial OS with one 250GB partition Cannot re-partition the boot drive One solution: –Boot server from CD –Ssh to server –Partition the disk Diskutil partitionDisk disk0 3 “JournaledHFS+” System 10G “JournaledHFS+” Cache 1G “JournaledHFS+” Work 200G Netboot may also be possibility Fastest solution: insert disks in 2nd and 3rd bays of server and format

Reference Server Install second server –Mac OS X Server –All patches –Xcode tools –Local directory configuration –AFS –Fink Shutdown, move hard disk to bay 2 of netboot server

Make a Network Install Master Netboot serves disk images Uses DHCP, BSDP 1, TFTP, and NFS to serve disk image to target machine Use Network Image Utility to make image of reference disk Turn on Netboot in Server Admin Boot client from network (different “secret handshake”) 1 Boot Server Discovery Protocol

Network Install Sequence Target Server Netboot Server DHCP broadcast Configure with DHCP helper address DHCP response BSDP broadcast BSDP response Router TFTP bootloader NFS disk image DHCP Server

Netboot Issues Our standard DHCP server worked just fine. BSDP is an extension of DHCP –We put on same subnet as cluster for simplicity –Should be able to use router helper to put on a different subnet Network Image Utility had some conflicts

Users: Looks like Unix Use NIS, LDAP, Kerberos for accounts Most utilities present: ssh, bash, perl, emacs, X-windows Fink supplies most gnu tools

Sysadmins: Sorta like Unix Strong BSD heritage Some things in different places than Linux GUI tools for configuring –Most, but not all, can configure a remote server Serversetup and networksetup commands can configure most settings Directory Access (setup NIS or LDAP) seems not to be scriptable

Startup Befuddlement Not inittab or /etc/rc.d based SystemStarter starts many daemons Ssh starts out of xinetd Ypbind started when needed by Directory Services Watchdog starts server processes –/etc/watchdog.conf –Sorta like inittab Mac OS X 10.4 (Tiger) is going to introduce yet another way

Authentication Compatibility Can use NIS, LDAP, Kerberos, Active Directory Supports PAM, but not everything uses it LoginWindow (and a few other things) are directly kerberos-aware and don’t use PAM

Infrastructure Fit Taylor (SLAC’s config tool) ran with little modification About half the config modules worked with little change The remainder took completely Mac OS X specific rewrites

Nits /etc/passwd is there, but not used. Uses netinfo instead for local accounts Shadow passwords in different place and format than linux, solaris afs permissions: default is to copy owner mode bits to group and other –Copy file from afs to local, ends up world-readable –Change ‘realmodes’ in /var/db/openafs/etc/config/settings.plist Mac HFS file system is not case-sensitive: Makefile and makefile are the same!

Conclusions Good hardware package, but lacking power management Network install suitable for cluster operation, but still a few wrinkles Generally good configuration management tools, but some divergence from standard Unix tools