1. Quattor : Installation and Configuration Management
M. Jouvin (LAL-Orsay)
Grid Administration Training
LAL, Orsay, September 2008, 15-19

2. Agenda Why Quattor ? Architecture Configuration Description
Serveur et client
Installation initiale
Administration
GRIF Example
Quattor usage status
Proposed configuration for tutorial
Useful links
Quattor - M. Jouvin
12/11/2018

3. Why Quattor ? Quattor developed EDG WP4
EDG : need a management tool for large farms
Initial installation AND configuration changes
Existing tools do one or the other
Examples : Kickstart, Imaging, APT…
Based on CERN (difficult) experience with management of a large number of machines
Crucial to have a unique and uniform description of the whole configuration
Be able of doing atomic installation of several packages
Installation must be reproducible
Quick reinstallation of a machine after a failure
Be able to ensure 2 machines are identical (WNs)
Linux (RH, rpm)
No dependency on grid middleware
Quattor - M. Jouvin
12/11/2018

4. Main Characteristics
Machine configuration stored in a versioned database
Reproducible installation : will reuse the same configuration
Active configuration is updated by modifying configuration database
From initial installation to machine death…
Initial installation with standard tool (Kickstart)
Configuration versioning allows an easy rollback
Service configuration reuse
Different types of machines can share a service configuration
Easier maintenance : only one place to modify to update a service configuration
Configuration consistency checked before deployment to lower risk of mistakes and broken configuration
Configuration information checked against a schema
Existence of packages to deploy
Quattor - M. Jouvin
12/11/2018

5. Quattor Workflow
Quattor - M. Jouvin
12/11/2018

6. Quattor (Core Components)…
Quattor = 1 database + 1 deployment server
Configuration database is a SVN repository
Configuration organized in templates
Description of final configuration, not how to do it
Language: PAN
Template inheritance
Hardware and software
Deployment server: dhcpd + tftpd
Server is notified after a configuration database change
Notification of clients to trig a reload of their configuration
Possible to have several servers attached to the same configuration database (multi-site configuration)
No Quattor specific service
Quattor - M. Jouvin
12/11/2018

7. … Quattor (Core Components)
Client : Configuration Cache Manager (CCM)
Receive a configuration change notification (cdp-listend) and download new configuration (command ‘ccm-fetch’)
Profile update may be forced by manually executing ‘ccm-fetch’
Client : Node Configuration Manager (NCM)
Process ncm-cdispd
Implements configuration changes
Based on “components” (plug-ins)
1 component manages 1 “service” (NIS, accounts, Globus…)
A component is a Perl script with a particular API
‘ncm-query’ alllows to display node configuration
Quattor management can be done from any platform (Unix, Windows, Macintosh)
PAN compiler (Java) + client SVN + ant available on all platforms
May use GUI like Eclipse
Quattor - M. Jouvin
12/11/2018

8. Quattor Other Components
Package deployment : SPMA
Implemented as a NCM component
Developed in the framework of Quattor
Atomic modification of installed packages (rpmt)
Relies on an explicit list of package version/arch to install
Allows upgrade and downgrade
Check package dependencies before deployment
APT and YUM are also supported but less integrated
Generally not used at Quattor sites
Initial installation : AII
Initial installation of OS and Quattor client
Configuration and package deployment done via NCM components
Use standard installation tool (Kickstart/Anaconda) + PXE
DHCP, TFTP and KS configured from configuration database
Managed with command ‘aii-shellfe’
Quattor - M. Jouvin
12/11/2018

9. Configuration and Templates
Machine configuration used by various Quattor components is a XML file
1 profile per machine, downloaded with http
Machine profile built from a high-level configuration description : templates
Templates use PAN language (declarative-like)
Templates are compiled to produce the machine profile
Standard and generic templates available to configure OS and gLite
QWG Templates
1profile generally uses many templates
Templates are similar to Russian puppets…
In QWG, several hundreds of templates used by all machines, most of them shared between machines and services
Quattor - M. Jouvin
12/11/2018

10. PAN Language… Declarative language Allow to define functions
Describes final state
Mainly made of value assignments to configuration paths or variables
A value may be based on another configuration path/variable
Non procedural: impossible to define conditional block
Allow to define functions
Written in PAN language
Produce value to be assigned to configuration paths/variable
Validation functions used to check validity of values at the end of compilation
Quattor - M. Jouvin
12/11/2018

11. … PAN Language Hierarchical organization of information
Exemples :
« /hardware/network/interfaces/eth0/ip » = «   »;
Strong typing of values
Configuration schema defines allowed resources and values
Similar to a structure/class definition
1 standard schema proposed
Type conversion functions available
Support for variables
Can be used as target of assignements and as input to compute other configuration values
Heavily used by QWG templates to customize generic templates
Quattor - M. Jouvin
12/11/2018

12. Quattor SCDB Versioned database of PAN templates Based on Subversion
No versioning of machine profiles : always possible to rebuild from templates
Based on Subversion
No Quattor specific components except a few ant tasks
Fine-grained access control based on SVN ACLs
SCDB operations implemented as ‘ant’ tasks
Written in pure Java : usable on all platforms
3 main opérations
compile : compile locally all machine profiles affected by template changes
Dependency management, incremental compilation
deploy : deployment of new configuration (machine profiles)
Profiles compiled on server, then clients are notified
Occurs only if compilation was successful and local copy of repository in synch with the repository itself
update.rep.templates : refresh templates associated with RPM repositories
Quattor - M. Jouvin
12/11/2018

13. NCM Components
Plug-ins of ncm-ncd managing one part of the configuration
Each component manages « /software/components/component- name » by default
Possible to define dependencies between components
E.g. SPMA to install the component
Must be written in Perl
Many components available
core : general components (in QWG standard/components)
chkconfig, named, filecopy…
lcg : configuration of gLite services (in QWG grid/version/components)
Quattor - M. Jouvin
12/11/2018

14. Quattor Client (SPMA)
Composant NCM responsable du déploiement logiciel
Lancé par ncm-ncd
Configuration via les templates : « /software/packages »
Configuré avec les fonctions PAN pkg_add, pkg_repl…
Vérification que les packages requis existent dans un repository
Possible d’utiliser APT ou YUM à la place de SPMA
Utilise rpmt pour faire des installations atomiques et vérifier les dépendances
rpmt basé sur les librairies RPM standards, écrit en Python
Aucune modification si problème de dépendance
Upgrade ou downgrade (rollback)
Sélection explicite de la version à installer (avantage sur APT)
Possibilité de versions différentes suivant les machines
Quattor - M. Jouvin
12/11/2018

15. Quattor AII AII : Automated Installation Infrastructure
PXE (DHCP+TFTP) + Kickstart configured from machine profile using ‘aii-shellfe’ command
DHCP (ISC) : may be shared between Quattor and non Quattor usage
Entries managed by Quattor must be in one dedicated file
TFTP : No exclusive usage by Quattor required
Kickstart : may be customized via AII hooks but normally standard configuration is enough
Re-installation may be triggered by reboot
Always boot PXE
Select between boot or reinstallation with ‘aii-shellfe’
Quattor - M. Jouvin
12/11/2018

16. Administration with Quattor
Template editions (site specific templates)
Most of the templates (>95%) are standard and should not be modified
Compile and deploy new configuration
ant, svn commit, ant deploy
Eclipse (IDE) may be used to edit, compile and deploy
Initial installation
Generate Kickstart+DHCP+TFTP configuration :
aii-shellfe --configure node
Installation at next reboot :
aii-shellfe --install node
Reboot using PXE (normally configured in BIOS as first device)
Quattor - M. Jouvin
12/11/2018

17. MW Update
Download templates for the new version and merge into local configuration
Tool directory-sync ( Install)
‘ant deploy’
Recompile les profils
Notifie les clients qu’un nouveau profil est disponible
Quattor - M. Jouvin
12/11/2018

18. escape("grid01.lal.in2p3.fr"), "sl450-x86_64",
OS Update
Download templates for the new version into local configuration
Update OS version used by a node in template os/version.db (site specific)
escape("grid01.lal.in2p3.fr"), "sl450-x86_64",
‘ant deploy’
Recompile profiles
Notify clients that a new configuration (profile) is available
Quattor - M. Jouvin
12/11/2018

19. Complex Configuration
Resources are organized in « cluster »
Set of machines sharing configuration parameters
Machines in a cluster may run different OS version/arch
Not necessarily connected to a cluster HW
Support for multi-site configurations
E.g. GRIF = 6 sites with specificities
Every cluster may belong to several sites
E.g.: local site + GRIF
1 template may be redefined in smaller scope
Search path defined per cluster
E.g. Standard, GRIF, site local, cluster
GRIF : +95% common templates
10-20 specific templates per cluster or site (except profiles)
Quattor - M. Jouvin
12/11/2018

20. SCDB Example : GRIF Quattor - M. Jouvin 12/11/2018 (15)
cluster.pan.includes=sites/ipno/**/* sites/grif/**/* os/sl305-i386/**/* standard/**/*
(2500)
(4)
Quattor - M. Jouvin
12/11/2018

21. Example of Templates...
All profiles for one specific machine types are identical
WN type defined based on other standard templates
object template profile_ipnls2005;
include machine-types/wn;
# Add repositories
include repository/config;
template machine-types/wn; …
# VO configuration
variable CONFIGURE_VOS = true;
variable CREATE_HOME ?= undef;
variable NODE_VO_PROFILE_ENV = true;
# Include base configuration of a gLite node
include { 'machine-types/base' };
# Include WN components
include { 'glite/wn/service' };
# Add site specific configuration, if any
include { return(WN_CONFIG_SITE) };
Quattor - M. Jouvin
12/11/2018

22. … Exemple de Templates Base configuration of a gLite machine
unique template machine-types/base; …
# profile_base for profile structure
include { 'quattor/profile_base' };
# NCM core components
include { 'components/spma/config' };
include { 'components/grub/config' };
# hardware
include { 'hardware/functions' };
"/hardware" = if ( exists(DB_MACHINE[escape(FULL_HOSTNAME)]) ) {
create(DB_MACHINE[escape(FULL_HOSTNAME)]);
} else {
error(FULL_HOSTNAME + " : hardware not found in machine database"); };
# common site machine configuration
include { 'pro_site_config' };
# File system configuration.
variable FILESYSTEM_LAYOUT_CONFIG_SITE ?= "site/filesystems/glite";
variable FILESYSTEM_CONFIG_SITE ?= "site/filesystems/glite";
# Select OS version based on machine name
include { 'os/version' };
# Load gLite version information
include { 'defaults/glite/version' };
Quattor - M. Jouvin
12/11/2018

23. Site Parameters Site configuration defined with variables
template site/glite/config; …
# MYPROXY CONFIGURATION
variable PX_HOST ?= "myproxy.grif.fr";
variable GRID_TRUSTED_BROKERS ?= list(
"/O=GRID-FR/C=FR/O=CNRS/OU=LAL/CN=grid09.lal.in2p3.fr",
"/O=dutchgrid/O=hosts/OU=nikhef.nl/CN=boswachter.nikhef.nl",
"/O=dutchgrid/O=hosts/OU=nikhef.nl/CN=bosheks.nikhef.nl",
"/O=GRID-FR/C=FR/O=CEA/OU=DAPNIA/CN=node27.datagrid.cea.fr",
"/O=GRID-FR/C=FR/O=CEA/OU=IRFU/CN=node08.datagrid.cea.fr", );
# MON BOX PARAMETERS (R-GMA), Apel
variable MYSQL_PASSWORD ?= "wp6_saclay";
variable MON_HOST ?= "node06.datagrid.cea.fr";
variable APEL_ENABLED ?= true;
variable APELDB_PWD ?= "lhc4G";
# OTHER SERVICE LOCATIONS
variable LFC_HOSTS ?= nlist(
"grid14.lal.in2p3.fr", nlist('alias', 'lfc.grif.fr'),
variable GRIDICE_SERVER_HOST ?= MON_HOST;
# RB / WMS
variable RB_HOST ?= "node04.datagrid.cea.fr";
variable WMS_HOST ?= "wms.grif.fr";
Quattor - M. Jouvin
12/11/2018

24. SCDB : Tracing Changes…
Change log easy to browser with a web-based SVN client (e.g. Trac)
Quattor - M. Jouvin
12/11/2018

25. ... SCDB : Tracing Changes
Quattor - M. Jouvin
12/11/2018

26. Who is Using Quattor ? Usage growth
50+ sites, mainly in Europe
Several group of sites managed with one DB (Ireland, Belgium)
Several quattorized countries
France, Spain, Ireland, Belgium
Wide variety in size, most using QWG templates
CERN : ~5000 nodes
~500 nodes : NIKHEF, CNAF, PIC, GRIF
Many smaller sites, e.g. SN-UCAD (4 machines + 2 VMs)
Main usage is grid management but growing interest for other usages
Internal servers, desktops, virtual machines (Xen)
Quattor - M. Jouvin
12/11/2018

27. Tutorial Proposed Configuration
1 Quattor server
1 CE
2 WNs
1 SE
1 BDII
Start with dedicated HW for each machine and demonstrate use of VMs if enough time
Quattor - M. Jouvin
12/11/2018

28. Documentation and Links
QWG web site :
Documentation of grid templates and their customization
SCDB
Initial installation of SCDB :
Quattor official site :
Being redesigned
Scientific Linux :
Quattor - M. Jouvin
12/11/2018