M.Biasotto, CERN, 5 november Fabric Management Massimo Biasotto, Enrico Ferro – INFN LNL

M.Biasotto, CERN, 5 november Legnaro CMS Farm Layout FastEth 32 – GigaEth 1000 BT SWITCH N1 FastEth SWITCH 1 8 S1 S16 N24 N1 Nx – Computational Node Dual PIII – 1 GHz 512 MB 3x75 GB Eide disk + 1x20 GB for O.S. Sx – Disk Server Node Dual PIII – 1 GHz Dual PCI (33/32 – 66/ MB 4x75 GB Eide Raid disks (exp up to 10) 1x20 GB disk O.S. FastEth SWITCH N1 2 N Nodes 4000 SI95 9 TB up to 190 Nodes S Servers 3.3 TB To WAN 34 Mbps Mbps 2002

M.Biasotto, CERN, 5 november Datagrid  Project structured in many “Work Packages”: –WP1: Workload Management –WP2: Data Management –WP3: Monitoring Services –WP4: Fabric Management –WP5: Mass Storage Management –WP6: Testbed –WP7: Network –WP8-10: Applications  3 year project ( ).  Milestones: month 9 (Sept 2001), month 21 (Sept 2002), month 33 (Sept 2003)

M.Biasotto, CERN, 5 november Overview  Datagrid WP4 (Fabric Management) overview  WP4 software architecture  WP4 subsystems and components  Installation and software management  Current prototype: LCFG  LCFG architecture  LCFG configuration and examples

M.Biasotto, CERN, 5 november WP4 overview  Partners: CERN, INFN (Italy), KIP (Germany), NIKHEF (Holland), PPARC (UK), ZIB (Germany)  WP4 website: grid-fabric/  Aims to deliver a computing fabric comprised of all the necessary tools to manage a centre providing Grid services on clusters of thousands of nodes

M.Biasotto, CERN, 5 november WP4 structure  WP activity divided in 6 main ‘tasks’ –Configuration management (CERN + PPARC) –Resource management (ZIB) –Installation & node management (CERN + INFN + PPARC) –Monitoring (CERN + INFN) –Fault tolerance (KIP) –Gridification (NIKHEF)  Overall WP4 functionality structured into units called ‘subsystems’, corresponding to the above tasks

M.Biasotto, CERN, 5 november Architecture overview  WP4 architectural design document (draft): – fabric/architecture/eu/default.htm  Still work in progress: open issues that need further investigation  Functionalities classified into two main categories: –User job control and management  handled by Gridification and Resource Management subsystems –Automated system administration  handled by Configuration Mgmt, Installation Mgmt, Fabric Monitoring and Fault Tolerance subsystems

M.Biasotto, CERN, 5 november Farm A (LSF)Farm B (PBS ) Grid User (Mass storage, Disk pools) Local User Installation & Node Mgmt Configuration Management Monitoring & Fault Tolerance Fabric Gridification Resource Management Grid Info Services (WP3) WP4 subsystems Other Wps Resource Broker (WP1) Data Mgmt (WP2) Grid Data Storage (WP5) Architecture overview - Interface between Grid- wide services and local fabric; - Provides local authentication, authorization and mapping of grid credentials. - Interface between Grid- wide services and local fabric; - Provides local authentication, authorization and mapping of grid credentials. - provides transparent access to different cluster batch systems; - enhanced capabilities (extended scheduling policies, advanced reservation, local accounting). - provides transparent access to different cluster batch systems; - enhanced capabilities (extended scheduling policies, advanced reservation, local accounting). - provides a central storage and management of all fabric configuration information; - central DB and set of protocols and APIs to store and retrieve information. - provides a central storage and management of all fabric configuration information; - central DB and set of protocols and APIs to store and retrieve information. - provides the tools to install and manage all software running on the fabric nodes; - bootstrap services; software repositories; Node Management to install, upgrade, remove and configure software packages on the nodes. - provides the tools to install and manage all software running on the fabric nodes; - bootstrap services; software repositories; Node Management to install, upgrade, remove and configure software packages on the nodes. - provides the tools for gathering and storing performance, functional and environmental changes for all fabric elements; - central measurement repository provides health and status view of services and resources; - fault tolerance correlation engines detect failures and trigger recovery actions. - provides the tools for gathering and storing performance, functional and environmental changes for all fabric elements; - central measurement repository provides health and status view of services and resources; - fault tolerance correlation engines detect failures and trigger recovery actions.

M.Biasotto, CERN, 5 november Resource Management diagram Stores static and dynamic information describing the states of the RMS and its managed resources Accepts job requests, verifies credentials and schedules the jobs Assigns resources to incoming job requests, enhancing fabric batch systems capabilities (better load balancing, adapts to resource failures, considers maintenance tasks) proxies provide uniform interface to underlying batch systems (LSF, Condor, PBS)

M.Biasotto, CERN, 5 november Monitoring & Fault Tolerance diagram Local node MSA MR cache FTA FTCEAD MS Central repository Data Base MR server Service master node FTCE Control flow Data flow Human operator host MUI MS Monitoring Sensor Agent - collects data from Monitoring Sensors and forwards them to the Measurement Repository Measurement Repository - stores timestamped information; it consists of local caches on the nodes and a central repository server Monitoring User Interface - graphical interface to the Measurement Repository Monitoring Sensor - performs measurement of one or several metrics; Fault Tolerance Correlation Engine - processes measurements of metrics stored in MR to detect failures and possibly decide recovery actions Actuator Dispatcher - used by FTCE to dispatch Fault Tolerance Actuators; it consists of an agent controlling all actuators on a local node Fault Tolerance Actuator - executes automatic recovery actions

M.Biasotto, CERN, 5 november Configuration Management diagram High Level Description High Level Description Low Level Description Low Level Description Cache Configuration Manager Cache Configuration Manager Local Process Local Process Configuration Database APIAPI Client Node Configuration Database: stores configuration information and manages modification and retrieval access Cache Configuration Manager: downloads node profiles from CDB and stores them locally

M.Biasotto, CERN, 5 november Configuration DataBase All computing nodes of CMS Farm #3 use cmsserver1 as Application Server cmsserver1 /etc/exports /app cmsnode1, cmsnode2,.. cmsserver1 /etc/exports /app cmsnode1, cmsnode2,.. cmsnode3 /etc/fstab cmsserver1:/app /app nfs.. cmsnode3 /etc/fstab cmsserver1:/app /app nfs.. cmsnode2 /etc/fstab cmsserver1:/app /app nfs.. cmsnode2 /etc/fstab cmsserver1:/app /app nfs.. cmsnode1 /etc/fstab cmsserver1:/app /app nfs.. cmsnode1 /etc/fstab cmsserver1:/app /app nfs.. High Level Description Low Level Description

M.Biasotto, CERN, 5 november Installation Management diagram Node Management Agent - manages installation, upgrade, removal and configuration of software packages Software Repository - central fabric store for Software Packages Bootstrap Service - service for initial installation of nodes

M.Biasotto, CERN, 5 november Distributed design  Distributed design in the architecture, in order to ensure scalability: –individual nodes as much autonomous as possible –local instances of almost every subsystem: operations performed locally where possible –central steering for control and collective operations Config DB Local Cache Config DB Local Cache Monitoring Local Repository Monitoring Local Repository Config DB Local Cache Config DB Local Cache Monitoring Local Repository Monitoring Local Repository Monitoring Central Repository Monitoring Central Repository Central Config DB Central Config DB Local Cache Config DB Local Cache Monitoring Local Repository Monitoring Local Repository

M.Biasotto, CERN, 5 november Scripting layer  All subsystems are tied together using a high level ‘scripting layer’: –allows administrators to code and automate complex fabric-wide management operations –coordination in execution of user jobs and administrative task on the nodes –scripts can be executed by Fault Tolerance subsystem to automate corrective actions  All subsystems provide APIs to control their components  Subsystems keep their independence and internal coherence: the scripting layer only aims at connecting them for building high-level operations

M.Biasotto, CERN, 5 november Maintenance tasks  Control function calls to NMA are known as ‘maintenance tasks’ –non intrusive: can be executed without interfering with user jobs (e.g. cleanup of log files) –intrusive: for example kernel upgrades or node reboots  Two basic node states from the administration point of view –production: node is running user jobs or user services (e.g. NFS server). Only non intrusive tasks can be executed –maintenance: no user jobs or services. Both intrusive and non intrusive tasks can be executed  Usually a node is put into maintenance status only when it is idle, after draining the job queues or switching the services to another node. But there can be exceptions to immediately force the status change.

M.Biasotto, CERN, 5 november Use Case example: upgrade of NFS server (I)  Kernel version of a given type of NFS server is to be upgraded: this affects server S1, which is accessed by N production nodes  An administrator (Product Maintainer) runs a ‘configuration change’ script, with the following steps:  1. add the SP with the new kernel to the SR  2. change in CDB the default configuration for S1 type machines for upgrading the kernel version  Possibly another admin (Service Manager), when appropriate, runs the following ‘deployment’ script:  3. query to CDB to get list of nodes N1,...,Nn that depend on server S1  4. change in CDB the configuration of nodes N1,...Nn to point from S1 to S2

M.Biasotto, CERN, 5 november Use Case example: upgrade of NFS server (II)  5. for each node Ni in N1,...,Nn, do (in parallel): –5.1 ask RMS scheduler to free node Ni from user jobs –5.2 ask NMA of Ni to reconfigure itself (change its NFS mount table from S1 to S2) as soon as node enters maintenance state (this is an intrusive task) –5.3 if Ni is in production state, ask NMA to put node in maintenance state –5.4 wait for reconfiguration to terminate –5.5 if Ni was in production, ask NMA to put node back in production state –5.6 if Ni was in production, notify RMS scheduler that the node is again available  6. wait for step 5 to complete on all nodes N1,...,Nn

M.Biasotto, CERN, 5 november Use Case example: upgrade of NFS server (III)  7. On server S1, ask NMA to reconfigure itself. As soon as S1 enters maintenance state the NMA will upgrade the kernel SP and reboot the node  8. wait for reconfiguration to terminate  9. if S1 was in production, ask NMA to put it in production state  Step 5 is not specific to this kind of operation: all its sub- steps can be coded in a generic library and reused in other scripts  Also the two administrative scripts could be generically written and reused with appropriate parameters

M.Biasotto, CERN, 5 november Installation & Software Mgmt Prototype  The current prototype is based on a software tool originally developed by the Computer Science Department of Edinburgh University: LCFG (Large Scale Linux Configuration)  Handles automated installation, configuration and management of machines  Basic features: –automatic installation of O.S. –installation/upgrade/removal of all (rpm-based) software packages –centralized configuration and management of machines –extendible to configure and manage custom application software

M.Biasotto, CERN, 5 november A collection of agents read configuration parameters and either generate traditional config files or directly manipulate various services Abstract configuration parameters for all nodes stored in a central repository ldxprof Load Profile Generic Component Profile Object rdxprof Read Profile LCFG Objects Local cache Client nodes Web Server HTTP XML Profile LCFG Config Files Make XML Profile Server LCFG diagram +inet.services telnet login ftp +inet.allow telnet login ftp sshd +inet.allow_telnet ALLOWED_NETWORKS +inet.allow_login ALLOWED_NETWORKS +inet.allow_ftp ALLOWED_NETWORKS +inet.allow_sshd ALL +inet.daemon_sshd yes auth.users myckey +auth.userhome_mickey /home/mickey +auth.usershell_mickey /bin/tcsh +inet.services telnet login ftp +inet.allow telnet login ftp sshd +inet.allow_telnet ALLOWED_NETWORKS +inet.allow_login ALLOWED_NETWORKS +inet.allow_ftp ALLOWED_NETWORKS +inet.allow_sshd ALL +inet.daemon_sshd yes auth.users myckey +auth.userhome_mickey /home/mickey +auth.usershell_mickey /bin/tcsh Config files , /home/MickeyMouseHome /bin/tcsh , /home/MickeyMouseHome /bin/tcsh XML profiles Profile Object inet auth /etc/services /etc/inetd.conf /etc/hosts.allow in.telnetd : , in.rlogind : , in.ftpd : , sshd : ALL /etc/hosts.allow in.telnetd : , in.rlogind : , in.ftpd : , sshd : ALL /etc/shadow /etc/group /etc/passwd.... mickey:x:999:20::/home/Mickey:/bin/tcsh.... /etc/passwd.... mickey:x:999:20::/home/Mickey:/bin/tcsh....

M.Biasotto, CERN, 5 november LCFG: future development API Generic Component Profile Object Cache Manager Cache Manager LCFG Objects Web Server XML Profile HTTP Configuration Database Cache Current Prototype Future Evolution ldxprof Load Profile Generic Component Profile Object rdxprof Read Profile LCFG Objects Local cache Client nodes Web Server HTTP XML Profile LCFG Config Files Make XML Profile Server

M.Biasotto, CERN, 5 november LCFG configuration (I)  Most of the configuration data are common for a category of nodes (e.g. diskservers, computing nodes) and only a few are node-specific (e.g. hostname, IP-address)  Using the cpp preprocessor it is possible to build a hierarchical structure of config files containing directives like #define, #include, #ifdef, comments with /* */, etc...  The configuration of a typical LCFG node looks like this: #define HOSTNAME pc239 /* Host specific definitions */ #include "site.h" /* Site specific definitions */ #include "linuxdef.h" /* Common linux resources */ #include "client.h" /* LCFG client specific resources */

M.Biasotto, CERN, 5 november LCFG configuration (II) From "site.h" #define LCFGSRV grid01 #define URL_SERVER_CONFIG #define LOCALDOMAIN.lnl.infn.it #define DEFAULT_NAMESERVERS [...] [...] From "linuxdef.h" update.interfaces eth0 update.hostname_eth0 HOSTNAME update.netmask_eth0 NETMASK [...] [...] From "client.h" update.disks hda update.partitions_hda hda1 hda2 update.pdetails_hda1 free / update.pdetails_hda1 free / update.pdetails_hda2 128 swap auth.users mickey auth.usercomment_mickey Mickey Mouse auth.userhome_mickey /home/Mickey [...]

M.Biasotto, CERN, 5 november LCFG: configuration changes  Server-side: when the config files are modified, a tool (mkxprof) recreates the new xml profile for all the nodes affected by the changes –this can be done manually or with a daemon periodically checking for config changes and calling mkxprof –mkxprof can notify via UDP the nodes affected by the changes  Client-side: another tool (rdxprof) downloads the new profile from the server –usually activated by an LCFG object at boot –can be configured to work as  daemon periodically polling the server  daemon waiting for notifications  started by cron at predefined times

M.Biasotto, CERN, 5 november LCFG: what’s an object?  It's a simple shell script (but in future it will probably be a perl script)  Each object provides a number of “methods” (start, stop, reconfig, query,...) which are invoked at appropriate times  A simple and typical object behaviour: –Started by profile object when notified of a configuration change –Loads its configuration from the cache –Configures the appropriate services, either translating config parameters into a traditional config file or directly controlling the service (e.g. starting a daemon with command-line parameters derived from configuration).

M.Biasotto, CERN, 5 november LCFG: custom objects  LCFG provides the objects to manage all the standard services of a machine: inet, syslog, auth, nfs, cron,...  Admins can build new custom objects to configure and manage their own applications: –define your custom “resources” (configuration parameters) to be added to the node profile –include in your script the object “generic”, which contains the definition of common function used by all objects (config loading, log, output,...) –overwrite the standard methods (start, stop, reconfig,...) with your custom code –for simple objects usually just a few lines of code

M.Biasotto, CERN, 5 november LCFG: Software Packages Management  Currently it is RedHat-specific: heavily dependent on the RPM tool  The software to install is defined in a file on the server containing a list of RPM packages (currently not yet merged in the XML profile)  Whenever the list is modified, the required RPM packages are automatically installed/upgraded/removed by a specific LCFG object (updaterpms), which is started at boot or when the node is notified of the change

M.Biasotto, CERN, 5 november First boot via floppy or via network Initialization script starts First boot via floppy or via network Initialization script starts LCFG: node installation procedure DHCP Server Software Packages Software Packages IP address Config URL IP address Config URL Root Image with LCFG environment NFS Server LCFG Config Files LCFG Config Files XML Profiles XML Profiles LCFG ServerWEB Server Software Repository Client Node After reboot LCFG objects complete the node configuration Root Image complete with LCFG environment mounted via NFS Load minimal config data via DHCP: IP Address, Gateway, LCFG Config URL Load minimal config data via DHCP: IP Address, Gateway, LCFG Config URL Load complete configuration via HTTP Start object “install”: disk partitioning, network,... installation of required packages copy of LCFG configuration reboot Start object “install”: disk partitioning, network,... installation of required packages copy of LCFG configuration reboot

M.Biasotto, CERN, 5 november LCFG: summary  Pros: –In Edinburgh it has been used for years in a complex environment, managing hundreds of nodes –Supports the complete installation and management of all the software (both O.S. and applications) –Extremely flexible and easy to customize  Cons: –Complex: steep learning curve –Prototype: the evolution of this tool is not clear yet –Lack of user-friendly tools for the creation and management of configuration files: errors can be very dangerous!

M.Biasotto, CERN, 5 november Future plans  Future evolution not clearly defined: it will depend also on results of forthcoming tests (1 st Datagrid milestone)  Integration of current prototype with Configuration Management components –Config Cache Manager and API released ad prototypes but not yet integrated with LCFG  Configuration DataBase –complete definition of node profiles –user-friendly tools to access and modify config information  Development of still missing objects –system services (AFS, PAM,...) –fabric software (grid sw, globus, batch systems,...) –application software (CMS, Atlas,...) in collaboration with people from experiments