1. Console Infrastructure in the CERN Computer Centre Helge.Meinhard@cern.ch HEPiX / HEPNT Autumn 2003 Vancouver Mostly work done by Andras.Horvath@cern.ch

2. HEPiX Vancouver: Console management at CERN2 Helge.Meinhard (at) cern.ch The problem CERN CC is running large farms CPU servers: now 1500 boxes, 6000* in 2006 Disk/tape servers: now 300 boxes, 1200* in 2006 *) Error bar: ~ factor 2 Attempt at high-level management solution: ELFms  T. Kleinwort Low-level problems E.g. machine unpingable Console access and/or reset required

3. HEPiX Vancouver: Console management at CERN3 Helge.Meinhard (at) cern.ch Existing solutions… … do not scale

4. HEPiX Vancouver: Console management at CERN4 Helge.Meinhard (at) cern.ch Requirements Considered systematically in summer 2003 Main points: Remote console access To boot loader and operating system (Linux) Preferably to BIOS as well Remote reset ATX reset and/or ATX power on/off and/or Remote power cycling

5. HEPiX Vancouver: Console management at CERN5 Helge.Meinhard (at) cern.ch Options (1 CHF = 0.75 USD = 0.65 EUR = 1 CAD) † : yes, but…

6. HEPiX Vancouver: Console management at CERN6 Helge.Meinhard (at) cern.ch Prototypes Serial daisy-chaining Up to 4 nodes BIOS, boot loader, OS Console: minicom But few boards come with two serial lines these days… Remote reset port 0 port 1 port 0 port 1 port 0 port 1 port 0 port 1

7. HEPiX Vancouver: Console management at CERN7 Helge.Meinhard (at) cern.ch Decisions Infrastructure for serial console via serial cards in PCs to be deployed Nothing else for now (no remote reset etc.) 24 x 7 operator coverage can step in Many services are redundant Specs for all new servers require support for Redirection of BIOS to serial line… and controllable system behaviour (stay off vs. previous state) on power cycle

8. HEPiX Vancouver: Console management at CERN8 Helge.Meinhard (at) cern.ch Serial infrastructure: head nodes Dedicated head nodes vs. worker nodes serving as heads for small number of peers +Cleaner – all worker nodes remain the same +Can be used for other head node applications (e.g. software distribution) if desired –Extra investment, extra space –If down, larger number of machines inaccessible via serial console Decided in favour of dedicated head nodes

9. HEPiX Vancouver: Console management at CERN9 Helge.Meinhard (at) cern.ch Concentration factor, scope Head nodes equipped with 6 8-port cards Complete head node (w/o serial cables) is about 1800 CHF By far cheaper than higher number of ports per console server, even though more console servers needed Will equip all CERN computer centre Machine rooms on ground floor and basement Except Windows machines, machines dedicated to network services Procurement running for 75 head nodes Cross-connection of head nodes not decided yet Some free ports on head nodes

10. HEPiX Vancouver: Console management at CERN10 Helge.Meinhard (at) cern.ch Software Need a bit more than minicom Logging into one of ~75 servers and requesting /dev/ttyS25 not going to scale Authentication and authorisation Logging of console output Started prototyping our own solution (Andras Horvath / CERN) Put on hold when we learned (at HEPiX Amsterdam) of … Software by Chuck Boeheim (SLAC) used at SLAC, Fermi, LBL, … Provides most of the functionality we require CERN-specific extensions can be easily added (wrapper scripts) Constructive discussions with Chuck, expect to share the work Aim is one common code base

11. HEPiX Vancouver: Console management at CERN11 Helge.Meinhard (at) cern.ch xxx pcitfionnn Software schematics lxplusnnn User app CDB – config service Machine – port @ head node mapping User – machine authorisations Console server 1 Server proc conf log Machine 1.1 Machine 1.44........ RS/232 Console server 75 Server proc conf log Machine 75.1 Machine 75.44........ … Console log repository

12. HEPiX Vancouver: Console management at CERN12 Helge.Meinhard (at) cern.ch Software components User application Should run on all on-site Linux machines; Windows, Solaris? Console application on head nodes Grants and logs access to serial lines Logs console output Configuration service Machine – port @ head node mapping User – machine mapping (authorisation to access serial line) Store for console logs Nothing on machines…

13. HEPiX Vancouver: Console management at CERN13 Helge.Meinhard (at) cern.ch Software: TBD On our wishlist: Authentication of head node towards user app, and of user towards server process on head node Per-line control of access right (Possibility of) logging via syslog CERN-specific extensions being designed Machine detection, feedback to config service Wrapper around user app asking config service to provide mapping of machine to (port @) head node Automatic creation of local config files on head nodes Collection of console logs in central repository

14. HEPiX Vancouver: Console management at CERN14 Helge.Meinhard (at) cern.ch Status, outlook HW: Orders for head nodes, serial cards, cables out or being finalised Expected delivery: 2H November 2003 SW: Started discussing and investigating adaptations, CERN-specific elements being designed Hope to have first head node ready in time for next disk server delivery (early December; no KVM switches!) Full deployment will run well into 2004

15. HEPiX Vancouver: Console management at CERN15 Helge.Meinhard (at) cern.ch Options Serial line PCI cards in PCs Dedicated console server PC Weasel card Keyboard / video / mouse multiplexers Analog Digital Relay boxes (for reset / ATX power on-off) Remotely controllable power strips IPMI

16. HEPiX Vancouver: Console management at CERN16 Helge.Meinhard (at) cern.ch Concentration factor Price comparison for head nodes + serial cards + patch panel boxes (if required) for basement machine room – 792 ports (18 racks for white box CPU servers) List prices for cards and boxes, assumed 1500 CHF per head node (1 CHF = 0.75 USD = 0.65 EUR = 1 CAD) Newest price estimate (head node + 6 x Exsys 41098): 1800 CHF = 41 CHF/port (44 ports) Head node configurationSystems / head node No of head nodes Price per port / CHF 1 PC + 6 x 8port441870 1 PC + 2 x 48port8811140 1 PC + 2 x 48port + 1 x 96port1766170 1 PC + 2 x 48port + 3 x 96port3523175

17. HEPiX Vancouver: Console management at CERN17 Helge.Meinhard (at) cern.ch Software requirements User application requests machine, not port @ head node Traffic between user application and head node Encrypted Head node authenticates to user application User authenticates to head node application Access controllable by user and machine / port @ head node Complete logging of console messages of all used ports Detect connected machines, i.e. establish machine – port @ head node mapping automatically Able to send magic keys, breaks etc. to machines No additional software to be installed on machines Logging who uses a given line Software on head nodes reconfigurable (SIGHUP)