Scaling Puppet and Foreman for HPC Trey Dockendorf HPC Systems Engineer Ohio Supercomputer Center

Introduction Puppet configuration management Hiera YAML data for Puppet Foreman provisioning HPC Environment using NFS root Deployed to 1000 HPC and Infrastructure systems First used on Owens 824 node cluster

Motivation Requirement of any large HPC center is scaling the provisioning and management of HPC clusters Common provisioning and configuration management between compute and infrastructure Unified management of PXE, DHCP and DNS Audit networking interfaces Support testing configuration changes Unit and system

Foreman Host life cycle management DNS, DHCP, TFTP – both Infrastructure and HPC Tuning: PassengerMaxPoolSize NFS root support required custom provisioning templates Local Boot PXE override to always network boot Workflow change for HPC – no ”Build”, use key-value TUNING – prevent overload of Foreman host

Foreman Key-Value Storage Key-value stored in Foreman as Parameters Change behavior during boot nfsroot_build Change TFTP templates nfsroot_path nfsroot_host nfsroot_kernel_version Hierarchical storage provides inheritance base/owens -> base/owens/compute -> FQDN Managed using web UI and scripts via API host-parameter.py & hostgroup-parameter.py

Foreman NFS Root Provisioning Provisioning handled by read/write host Support by Foreman written from scratch Read-only hosts have specific locations writable defined by /etc/statetab & /etc/rwtab statetab – persists through reboots rwtab – does not persist through reboots Read-only rebuild: nfsroot_build parameter osc-partition service Partition scripts generated by Puppet Defined using a partition schema in Hiera partition-wait script if nfsroot_build=false Statetab/rwtab Statetab used only when absolutely necessary to allow reboots ability to reset systems All puppet managed resources go in rwtab Read-only rebuild set by host-parameter.py Foreman Role allows students ability to rebuild nodes Osc-partition checks key-value of Foreman Partition schema defines contents of partition scripts Partition wait designed to wait for LVM to become available

NFS Root Boot Workflow

Parameter Manipulation Timing Operation Avg Time (sec) StdDev (sec) host get 0.522 0.021 host list 0.467 0.025 host set 0.581 0.024 host set /w TFTP sync 1.321 0.057 host delete 0.433 0.026 host delete w/ TFTP sync 1.148 0.038 hostgroup get 0.510 hostgroup list 0.514 0.034 hostgroup set 0.526 0.030 hostgroup delete 0.489

Scaling Puppet – Standalone Hosts Typically master compiles catalog for agents Scaling achieved by load balancing between masters Subject Alternative Name certificates – any master can be CA Masters synced with mcollective and r10k Environments isolated by r10k control repo and git branching Foreman acts as ENC (External Node Classifier) to Puppet Scaling Number of CPUs available to masters is number of concurrent agents Environments isolated… Modules not in control repo are defined by Puppetfile, mostly community modules ENC supplies important data to Puppet like IP address and hostgroup

Puppet Performance – Standalone Hosts Min Max Mean Std Resource Count 644 9093 1313 806 Compile Times (sec) 14 82 21 9

Scaling Puppet – HPC Systems Scaling achieved by removing master and using masterless Primary bottleneck is performance reading manifests and modules then compiling locally /opt/puppet – manifest and modules synced by mcollective and r10k Read-write hosts still use puppet masters Masterless puppet run via papply Environment isolation defaults to read-write host’s Stateful - use PuppetDB and Foreman ENC Stateless - uses minimal catalog run in two stages at boot Stateless - manages locations in rwtab

NFS Root Boot Workflow

Puppet Performance - Masterless Min (sec) Max (sec) Mean (sec) Std (sec) Early 27 257 115 57 Late 87 414 185 73 Compile 3 19 9 Late contains 71 managed resources Late contains 60 second wait for filesystems Times collected from data of system bring-up after maintenance 60 second wait in order to get GPFS mounted before pbs_mom and NHC run

Cluster Definitions - YAML YAML files define cluster Script used to sync YAML with Foreman Loaded into Puppet as Hiera data Make Puppet aware of cluster nodes and their properties Populates clustershell, pdsh, Torque, conman, powerman, SSH host based auth, etc YAML deployed to root filesystem as Python pickle and Ruby marshall YAML on root filesystem loaded to populate facts Informational such as node location Determine behavior when Puppet runs Ruby and Python based facts

Custom Fact Example Usage

Repositories Foreman Templates Foreman Plugin NFS Root Module https://github.com/treydock/osc-foreman-templates Foreman Plugin https://github.com/treydock/foreman_osc NFS Root Module https://github.com/treydock/puppet-nfsroot Puppet Masterless Module https://github.com/treydock/puppet-puppet_masterless Cluster Facts module https://github.com/treydock/puppet-osc_facts